Data processing method, apparatus, medium, device, and program product

Abstract

A data processing method, apparatus, medium, device, and program product are disclosed. The data processing method includes: obtaining a plurality of first fragment identifiers, wherein the plurality of first fragment identifiers include a plurality of second fragment identifiers, wherein the second fragment identifiers include fragment identifiers of first data, and the second fragment identifiers are obtained based on the data content of the first data; and initiating a plurality of processing tasks, wherein the processing tasks correspond to the first fragment identifiers, and the processing tasks are used to process second data to obtain third data, wherein the second data includes the first data corresponding to the first fragment identifiers. Through the above technical solution, first data with the same fragment identifier can correspond to one processing task. One processing task can be understood as being responsible for processing a small batch of data. The data in each small batch can be processed in parallel, thereby adopting a micro-batch processing approach, reducing data processing time, and improving data processing efficiency.

Description

Technical Field

[0001] This article relates to the field of data processing technology, specifically to a data processing method, apparatus, medium, device, and program product. Background Technology

[0002] Currently, processing large amounts of raw data is time-consuming. Taking the training data of Vision Language Models (VLM) as an example, it is usually necessary to process massive amounts of raw data to obtain the training data for the Vision Language Model, and the data processing process is time-consuming. Summary of the Invention

[0003] This content section is provided to briefly introduce the ideas, which will be described in detail in the examples section later. This content section is not intended to identify key or essential features of the claimed content, nor is it intended to limit the scope of the claimed content.

[0004] Firstly, a data processing method is provided, including: A plurality of first fragment identifiers are obtained, wherein the plurality of first fragment identifiers include a plurality of second fragment identifiers, wherein the second fragment identifiers include fragment identifiers of first data, and the second fragment identifiers are obtained based on the data content of the first data; Multiple processing tasks are initiated, each corresponding to the first shard identifier. The processing tasks are used to process the second data to obtain the third data, whereby the second data includes the first data corresponding to the first shard identifier.

[0005] Secondly, a data processing apparatus is provided, comprising: The first acquisition module is configured to acquire multiple first fragment identifiers, the multiple first fragment identifiers including multiple second fragment identifiers, the second fragment identifiers including fragment identifiers of first data, and the second fragment identifiers being obtained based on the data content of the first data; The first startup module is configured to start multiple processing tasks, each processing task corresponding to the first shard identifier. The processing task is used to process the second data to obtain the third data, the second data including the first data corresponding to the first shard identifier.

[0006] Thirdly, a computer-readable medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processing device, implements the data processing method provided in the first aspect.

[0007] Fourthly, an electronic device is provided, comprising: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the data processing method provided in the first aspect.

[0008] Fifthly, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the data processing method provided in the first aspect.

[0009] Using the method described above, the shard identifier of the first data can be used for data sharding. This means that first data with the same shard identifier belongs to the same data shard. Each processing task corresponds to a first shard identifier and is used to process the first data corresponding to that identifier. Thus, each set of first data with the same shard identifier can correspond to one processing task, which can be understood as being responsible for processing a small batch of data. Through multiple processing tasks, each small batch of data can be processed in parallel, thereby employing micro-batch processing to reduce data processing time and improve data processing efficiency.

[0010] Other features and advantages will be described in detail in the following examples section. Attached Figure Description

[0011] The above and other features, advantages, and aspects of this document will become more apparent when viewed in conjunction with the accompanying drawings and the following examples. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 This is an example of a data processing method in related technologies.

[0012] Figure 2 This is a schematic diagram illustrating an exemplary application scenario.

[0013] Figure 3 This is a schematic diagram illustrating an application processing method.

[0014] Figure 4 This is a flowchart illustrating an application processing method.

[0015] Figure 5 This is a flowchart illustrating an exemplary data processing method.

[0016] Figure 6 This is a block diagram illustrating an exemplary data processing apparatus.

[0017] Figure 7 A schematic diagram of an electronic device suitable for implementing the above method is shown. Detailed Implementation

[0018] The following description will be given in more detail with reference to the accompanying drawings. While certain scenarios are shown in the drawings, it should be understood that this document can be implemented in various forms and should not be construed as limited to the scenarios described herein. Rather, these scenarios are provided to provide a more thorough and complete understanding of this document. It should be understood that the accompanying drawings and the scenarios depicted are for illustrative purposes only and are not intended to limit the scope of this document.

[0019] It should be understood that the steps described in the method may be performed in different orders and / or in parallel. Furthermore, the method may include additional steps and / or omit the steps shown. The scope of this document is not limited in this respect.

[0020] The term "comprising" and its variations can be open-ended, meaning "including but not limited to". The term "based on" can mean "at least partially based on". The term "one case" means "at least one case"; the term "another case" means "at least one additional case"; the term "some cases" means "at least some cases". Definitions of other terms will be given in the following description.

[0021] It should be noted that the concepts of "first" and "second" are only used to distinguish different devices, modules or units, and are not used to limit the order of the functions performed by these devices, modules or units or their interdependencies.

[0022] It should be noted that the modifiers “one” and “multiple” can be illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as “one or more”.

[0023] The names of messages or information exchanged between multiple devices are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0024] It is understandable that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of relevant regulations.

[0025] First, let's introduce the relevant concepts.

[0026] Distributed computing engine: Typically used for batch processing and data transformation tasks of large-scale data. It manages the tasks, generates execution plans, and schedules each computing node through management nodes. The computing nodes are responsible for performing data computation, data reading and writing, and other operations.

[0027] Resilient Distributed Dataset (RDD): A data abstraction model in a distributed computing engine that supports distributed storage and parallel computing. It achieves efficient processing of massive amounts of data by dividing the data into multiple partitions and distributing them to multiple computing nodes to perform computing tasks.

[0028] A partition is a data unit in a resilient distributed dataset. A resilient distributed dataset consists of multiple partitions, and a partition can contain multiple data entries. For example, in model training, a single data entry can be considered a training data entry, or a record. These partitions can be processed in parallel on different computing nodes, and a single computing node can be responsible for one or more partitions. Partitions can be used as a data redistribution mechanism in distributed computing. By adjusting the data partitioning method, data can be redistributed across multiple computing nodes to optimize parallel computing performance and resource utilization efficiency.

[0029] Columnar storage format: A storage method that stores data in columns. Parquet is a commonly used columnar storage format.

[0030] The Hadoop Distributed File System (HDFS) is used to store and manage massive amounts of data. It can support files in formats such as Parquet and provides reliable data storage capabilities for distributed computing.

[0031] Combination Figure 1 This section introduces the data processing methods used in related technologies. The original dataset refers to the original RDD dataset, which includes the initial partitions of the data to be processed, such as partition 1, partition 2, partition 3 to partition N. That is, the data to be processed is divided into N partitions, where N is a positive integer greater than 1.

[0032] Related technologies employ online streaming processing for data processing. Online streaming processing can be understood as continuously and in real-time processing of data streams. For example, data in partition 1 is processed and then written to the storage system, data in partition 2 is processed and then written to the storage system, and so on. However, online streaming processing is susceptible to the limitations of message queue resources, resulting in lower write throughput when applied to large-scale data processing.

[0033] Some related technologies employ offline batch processing for data processing. Offline batch processing can be understood as writing the entire dataset at once. For example, after processing the data in partitions 1 to N, the processed data from each partition is written to the storage system all at once. However, offline batch processing has some drawbacks for complex computational tasks, such as high task time consumption and the need to retry the entire dataset if a write failure occurs, resulting in high retry costs.

[0034] This paper presents a data processing method that obtains the fragment identifiers of the data to be processed. Different data sets may share the same fragment identifier; for example, data set 'a' and data set 'b' may have the same fragment identifier. Based on these fragment identifiers, multiple processing tasks are created for data processing. Data sets with the same fragment identifier can correspond to one processing task. This can be understood as data sets with the same fragment identifier belonging to the same data fragment. Each data fragment corresponds to a small batch of data processing, and these small batches can be processed in parallel, thus employing micro-batch processing to reduce data processing time.

[0035] Furthermore, by dividing the data into multiple processing tasks and processing each small batch of data in parallel, for processing tasks that fail to write data, only the small batch of data can be retried, reducing the cost of retrying. In addition, the processing progress of small batches of data can be obtained separately and reported to the client. Technical personnel do not have to wait for the entire dataset to be completed to receive progress feedback, allowing them to know the processing progress of small batches of data.

[0036] Figure 2 This is a schematic diagram illustrating an exemplary application scenario, such as... Figure 2 As shown, the application scenario of the data processing method may include a management node 21, a computing node 22, a storage system 23, and a client 24. The management node 21, computing node 22, and storage system 23 can form a distributed cluster. There can be multiple computing nodes 22, as illustrated in the diagram. The management node 21 and computing node 22 can be implemented in hardware or software.

[0037] The management node 21 can assign tasks to each computing node 22 and obtain the task execution progress of the computing node 22. For example, the computing node 22 can respond to the scheduling of the management node 21 and perform operations such as repartitioning, data reading, and data writing mentioned below.

[0038] Storage system 23 can be a distributed file system, for example, capable of hosting files in formats such as Parquet, as well as lake tables of a data lake. A data lake is a storage architecture that centrally stores massive amounts of raw data and various types of data, supporting direct storage of structured, semi-structured, and unstructured data in raw formats. Management node 21 and compute node 22 can store data to storage system 23 via a network.

[0039] The management node 21 can obtain the processing progress of small batches of data and send the processing progress to the client 24, which can then display the processing progress of the small batches of data. The client 24 can be, for example, a personal computer, a laptop, or other devices such as a mobile phone.

[0040] The following is combined with Figure 3 and Figure 4 This section introduces examples of the application processing methods.

[0041] Figure 3 This is a schematic diagram illustrating an application processing method. Figure 4 This is a flowchart illustrating an application processing method, such as... Figure 4 As shown, the application processing method may include steps 41 to 47, which can be applied to management node 21.

[0042] Step 41: Obtain the total number of entries.

[0043] Reference Figure 3 As shown, to distinguish them from the new partitions after repartitioning mentioned below, the partitions of the original RDD dataset are referred to as original partitions, including original partition 1 to original partition N. The total number of rows includes the number of rows to be processed, which can be taken as the total number of rows.

[0044] Step 42: Based on the total number of records, the preset maximum number of partitions, and the preset maximum number of records, obtain the number of partitions.

[0045] The preset maximum number of records indicates the maximum number of records to be processed in a new partition. If the total number of records is greater than the preset maximum number of records, it means that the data to be processed can be divided into multiple new partitions. In this case, the number of partitions is calculated. This number of partitions refers to the number of new partitions after repartitioning. For example, the number of new partitions is M, where M is a positive integer greater than 1.

[0046] Optionally, the number of partitions can be calculated by determining the ratio of the total number of records to the preset maximum number of records, rounding the ratio up, and taking the smaller of the rounded-up value and the preset maximum number of partitions as the number of partitions. For example, if the preset maximum number of partitions is 100, and the rounded-up value is 105, then the number of partitions is 100; if the rounded-up value is 95, then the number of partitions is 95.

[0047] Step 43: Obtain multiple first fragment identifiers.

[0048] The management node 21 can distribute the number of partitions to the compute node 22 and instruct the compute node 22 to determine the rules for the shard identifiers of the data to be processed. Optionally, the shard identifier can be calculated based on all the data content of the data to be processed, for example, by calculating a hash value based on all the data content of the data to be processed, or by calculating a hash value based on the primary key content of the data to be processed and then performing a modulo operation between the hash value and the number of partitions to obtain the shard identifier. The shard identifier of the data to be processed can be a value between 0 and the number of partitions minus 1.

[0049] The computing node 22 can determine the fragment identifiers of each piece of data to be processed in the original partition it is responsible for, and mark the fragment identifiers of the data to be processed. The second fragment identifier includes the fragment identifier of the data to be processed, and the second fragment identifier corresponds to the data to be processed. For example, the second fragment identifier of data to be processed a is 00, the second fragment identifier of data to be processed b is 00, and the second fragment identifier of data to be processed c is 01. There are data to be processed with the same fragment identifier, for example, data to be processed a and data to be processed b have the same fragment identifier. Multiple first fragment identifiers include multiple second fragment identifiers. Multiple first fragment identifiers can be understood as a list of all fragment identifiers, for example, multiple first fragment identifiers include fragment identifier 00, fragment identifier 01 to fragment identifier M-1. Furthermore, referring to the exemplary method of determining fragment identifiers described above, assuming that the result of the modulo operation does not have 00, then there is no data to be processed with a fragment identifier of 00.

[0050] Step 44: Obtain new partition information.

[0051] The new partition information is obtained based on the number of partitions and the fragment identifiers of each piece of data to be processed. Specifically, the management node 21 can use the number of partitions as the number of new partitions after repartitioning and the fragment identifiers of the data to be processed as the partition keys to instruct each computing node 22 to perform the repartitioning operation.

[0052] Reference Figure 3The original partition contains data to be processed with different shard identifiers. The purpose of the repartitioning operation is to make a new partition contain data to be processed with the same shard identifier. The content in parentheses above the new partition represents the shard identifier. For example, new partition 0 contains data to be processed with shard identifier 00, new partition 1 contains data to be processed with shard identifier 01, new partition 2 contains data to be processed with shard identifier 02, and so on for other new partitions.

[0053] Step 45: Generate a write task based on the new partition information.

[0054] Write tasks can be used to write unprocessed data from a new partition to a corresponding set of files. (See reference...) Figure 3 For example, the write task corresponding to new partition 0 is used to write the data to be processed in new partition 0 into the file set (00) in the form of a file. The write task corresponding to new partition 1 is used to write the data to be processed in new partition 1 into the file set (01) in the form of a file. The write tasks of other new partitions are similar and will not be described in detail.

[0055] That is, each first segment identifier can correspond to a file set. File set (00) is used to store the data to be processed with segment identifier 00 in file form, and file set (01) is used to store the data to be processed with segment identifier 01 in file form. File set (02) and file set (03) are similar. The ellipsis after file set (03) indicates the file sets corresponding to other first segment identifiers that are omitted.

[0056] Optionally, the file set can be a Parquet directory on a distributed file system, and the files written to the file set can be Parquet files. Taking file set (00) as an example, Parquet files can be written to file set (00). The number of Parquet files can be one or more. The Parquet files include the data to be processed in the new partition 0 with the shard identifier 00.

[0057] Step 46: Submit multiple processing tasks to the thread pool and run multiple first threads to start multiple processing tasks.

[0058] The processing task corresponds to the first fragment identifier. (See reference...) Figure 3 For example, the processing task corresponding to fragment identifier 00 is represented as processing task (00)), the processing task corresponding to fragment identifier 01 is represented as processing task (01) and the processing task corresponding to fragment identifier 02 is represented as processing task (02). Similarly, the ellipsis after processing task (03) indicates the omitted processing tasks corresponding to other first fragment identifiers.

[0059] For example, processing task (00) is used to process the data to be processed with shard identifier 00, processing task (01) is used to process the data to be processed with shard identifier 01, and processing task (02) is used to process the data to be processed with shard identifier 02. Other processing tasks are similar. Therefore, data to be processed with the same shard identifier can correspond to one processing task. A processing task can be regarded as being responsible for processing small batches of data with the same shard identifier. The data in each small batch can be processed in parallel, which can reduce the data processing time compared to writing the entire dataset at once.

[0060] Step 47: Submit the write task to the thread pool and run the second thread to start the write task.

[0061] The total number of threads in a thread pool can be greater than the number of tasks to be processed. First, submit multiple processing tasks to the thread pool so that multiple processing tasks can occupy thread resources. After multiple processing tasks have occupied thread resources, there are still unoccupied threads in the thread pool. Then, submit write tasks to the thread pool so that the write tasks can be started using the remaining unoccupied threads.

[0062] This ensures that each processing task has thread resources, thereby reducing data processing time through micro-batch processing. Furthermore, after a processing task starts, it can wait for new files to be added to the corresponding file set during execution. Taking processing task (01) as an example, after starting, processing task (01) can wait for new files to be added to the file set (01). By maintaining a list of read files, processing task (01) can discover new files. After reading a new file, it can convert it into an elastic distributed dataset, call processing functions to process the data to be processed in the new file, and write the processed data to the storage system, such as to a data lake. Thus, while reducing time consumption through micro-batch processing, processing and writing tasks can be parallelized, further reducing the data size written to the storage system in a single operation and alleviating data disk pressure.

[0063] Alternatively, the data processing method may include, for example, data ETL (Extraction Transformation Loading).

[0064] Assuming that a write task is submitted to the thread pool first, after the thread in the thread pool is occupied by the write task, the processing task may not be able to get thread resources. Taking the file set (01) as an example, if the write task is executed first and the processing task (01) waits for thread resources, a large number of files may have been written to the file set (01). When the processing task (01) starts, it needs to read a large amount of data and write the data to disk, which may increase the pressure of writing data to disk.

[0065] Optionally, taking writing processed data to the data lake as an example, this can be done offline to the lake tables. Offline writing to lake tables typically involves pre-sorting by partition, primary key, or sort key before writing, and then compressing the data to improve subsequent query efficiency and storage compactness. Lake table compaction is an optimization method for data lake storage structures. It merges and reorganizes scattered data files, optimizing storage layout, improving data access performance, and reducing system management costs. By using micro-batch processing in parallel with write tasks, the size of pre-sorted and compressed data during a single write to the lake table can be reduced, alleviating the pressure on the lake table to disk. Although micro-batch writing may generate more fragments or small files, these can be merged, rearranged, and compressed by subsequent lake table compaction tasks.

[0066] like Figure 3 As shown, assuming the number of processing tasks is 4, and the total number of threads in the thread pool is 5, processing tasks (00), (01), (02), and (03) can be submitted to the thread pool first. Each of these four processing tasks occupies one thread, and the remaining thread can be allocated to the write task. For example, the write task for the first preset number of new partitions can occupy one thread resource, as shown in the example. Figure 3 For example, write tasks for two new partitions occupy one thread. The write tasks for new partition 0 and new partition 1 (indicated by rounded boxes) occupy one thread, the write tasks for new partition 2 and new partition 3 occupy one thread, and the write tasks for new partition 4 and new partition 5 occupy one thread. The remaining thread can be allocated to the write tasks for new partition 0 and new partition 1.

[0067] After a processing task is completed, thread resources can be released and allocated to write tasks that are in a waiting state. For example, if a processing task (00) does not read any new files from the file set (00) within a preset time period and does not perform data processing or write operations to the storage system, the processing task (00) can be considered complete and thread resources can be released. If a write task has written all the data to be processed in the new partition it is responsible for to the file set, it can release thread resources.

[0068] Reference Figure 3 For example, the thread resources released by processing task (00) are allocated to write tasks in new partitions 2 and 3. The solid rounded-corner boxes indicate that the write and processing tasks are in progress, while the dashed rounded-corner boxes indicate that the write tasks in new partitions 4 and 5 have not yet been allocated thread resources and are in a waiting state. The dashed rounded-corner box indicates that processing task (00) has been completed and its thread resources have been released. The first thread mentioned above can be understood as the thread resources occupied by the processing task, and the second thread can be understood as the thread resources occupied by the write task.

[0069] The thread pool can be managed on the management node 21, which is responsible for scheduling various tasks. The computing node 22 performs specific operations, such as reading, processing, and writing.

[0070] As described above, data to be processed with the same shard identifier can correspond to a processing task. A processing task can be understood as being responsible for processing a small batch of data. The data in each small batch can be processed in parallel, thereby adopting a micro-batch processing approach to reduce data processing time and improve task throughput.

[0071] Furthermore, starting the processing task first and then the write task allows the processing task to continuously read newly added files from the corresponding file set. After reading the new files, the processing function can be called to process the data to be processed in the new files, realizing a micro-batch processing approach where processing and write tasks run in parallel. Since each processing task is responsible for a small batch of data, and the processing and write tasks run in parallel, data processing time can be reduced. Moreover, when writing the processed data to the storage system, the size of the data written to the storage system at one time can be further reduced. Taking writing to a data lake as an example, the size of the data that needs to be pre-sorted and compressed before writing to the lake table is reduced, alleviating the pressure on the lake table to be written to disk.

[0072] For failed processing tasks, the processing task can be retried individually. For example, if processing task (01) fails, processing task (01) can be retried without affecting other processing tasks. Therefore, while reducing data processing time, the retry cost is lower than retrying the writing of the entire dataset.

[0073] It should be noted that, for Figure 4 The execution order of the steps shown is not restricted. Figure 4 The example shown is for illustrative purposes only; for instance, step 46 may also be performed before step 45.

[0074] In addition, writing the processed data to the data lake is an example of how it can also be written to the storage system through other storage architectures, without any restrictions.

[0075] In one embodiment, the application processing method may include: obtaining the fragment processing progress based on first metadata for at least one first fragment identifier.

[0076] For example, for each first shard identifier, the shard processing progress corresponding to that first shard identifier can be statistically analyzed.

[0077] Taking shard identifier 01 as an example, the data to be processed with shard identifier 01 can be written to file set (01) in the form of a Parquet file. The end of the Parquet file can contain file metadata, which includes the number of data entries to be processed for shard identifier 01 in the Parquet file. When the processing task (01) reads a newly added Parquet file from file set (01), it can obtain the number of data entries to be processed in the Parquet file from the file metadata at the end of the Parquet file. Thus, the processing task (01) can accumulate the number of data entries to be processed in each Parquet file it reads and dynamically update the accumulated data. This accumulated data can represent the processing progress of the data to be processed for shard identifier 00, i.e., the shard processing progress. The shard processing progress corresponding to other shard identifiers is similar. Through the file metadata, the number of data in the data shards can be counted. While reducing the time consumption through micro-batch processing, the processing progress of each micro-batch can also be counted without triggering a full data read to count the number of data in the data shards.

[0078] In this way, the processing task can obtain the processing progress of the corresponding shard identifier by reading the file metadata of the file, thereby enabling the dynamic processing progress of each small batch of data to be statistically analyzed. The processing progress of each first shard identifier is sent to the client 24, which can display the dynamic processing progress of each small batch in real time. Technicians do not have to wait for the entire dataset to be completed to receive progress feedback, and it is also convenient for technicians to observe which one or more small batches of data have processing anomalies.

[0079] In addition, during task execution, logs can be recorded for file waiting, new file discovery, and read exceptions, making it easier to locate the execution status of individual data shards. Management node 21 can also collect statistics on the data size, processing time, and exception information of each data shard.

[0080] Figure 5 This is a flowchart illustrating an exemplary data processing method, which can be applied to management node 21, such as... Figure 5 As shown, the data processing method includes steps 51 and 52.

[0081] Step 51: Obtain multiple first fragment identifiers.

[0082] The multiple first fragment identifiers include multiple second fragment identifiers, and the second fragment identifiers include the fragment identifiers of the first data. The second fragment identifiers are obtained based on the data content of the first data.

[0083] The first fragment identifier and the second fragment identifier have been described in the above embodiments. The first data can refer to the data to be processed in the above embodiments, that is, a piece of data to be processed.

[0084] Step 52: Start multiple processing tasks.

[0085] Each processing task corresponds to a first shard identifier, and can be viewed as a one-to-one correspondence between multiple processing tasks and multiple first shard identifiers. The processing task is used to process the second data to obtain the third data, which includes the first data corresponding to the first shard identifier.

[0086] Multiple first fragment identifiers can refer to fragment identifier 00, fragment identifier 01, fragment identifier 02 and fragment identifier 03 in the above embodiment, and multiple processing tasks can refer to processing task (00), processing task (01), processing task (02) and processing task (03).

[0087] Taking task (00) as an example, the second data can refer to the data to be processed with shard identifier 00, and the third data may include, for example, the data obtained by performing ETL on the data to be processed with shard identifier 00. Optionally, the third data can also be written to a storage system, such as a data lake.

[0088] Through the above technical solution, the fragment identifier of the first data can be used for data fragmentation. This can be understood as first data with the same fragment identifier belonging to the same data fragment. The processing task corresponds to the first fragment identifier and is used to process the first data corresponding to that first fragment identifier. In this way, the portion of first data with the same fragment identifier can correspond to one processing task, and one processing task can be understood as being responsible for processing a small batch of data. Through multiple processing tasks, each small batch of data can be processed in parallel, thereby adopting a micro-batch processing approach, reducing data processing time, and improving data processing efficiency.

[0089] In one embodiment, step 52 may include: Run multiple first threads to start multiple processing tasks, with each first thread corresponding to a processing task; The processing task is used to read the newly added first file from the first file set, process the second data in the first file to obtain the third data, and the first file set corresponds to the first fragment identifier.

[0090] For example, multiple processing tasks can be submitted to a thread pool to run multiple first threads, and the multiple first threads can correspond one-to-one with the multiple processing tasks. As described in the above embodiment, multiple first file sets can refer to file set (00), file set (01), file set (02), and file set (03), and the multiple first file sets can correspond one-to-one with multiple first fragment identifiers. The newly added first file refers to the newly added file described above, and the first file is, for example, a Parquet file. For example, the first file in file set (00) includes the data to be processed with fragment identifier 00.

[0091] The above scheme can create a corresponding first file set for each first segment identifier. The processing task can read the newly added first file from the corresponding first file set to realize micro-batch processing with the first segment identifier as the granularity. After reading the newly added first file, the processing function can be called to process the data.

[0092] In one embodiment, the data processing method may include: Generate a write task, which is used to write the first file to the first set of files; Run the second thread to start the write task. The first and second threads belong to the thread pool. The second thread includes threads that are not occupied when processing tasks. The total number of threads in the thread pool is greater than the first number, which includes the number of tasks to be processed.

[0093] For example, a write task can be submitted to a thread pool to run a second thread. Taking the write task corresponding to shard identifier 00 and the first file being a Parquet file as an example, this write task is used to write the data to be processed for shard identifier 00 into the file set (00) as a Parquet file.

[0094] The first file can also be other file formats used for columnar storage formats; the Parquet file is an example.

[0095] The step of initiating multiple processing tasks can be executed before the step of submitting write tasks to the thread pool. This allows multiple processing tasks to utilize thread resources. After a processing task starts, it can wait to be written to the newly added first file in the corresponding first file set, realizing a micro-batch processing approach that allows processing and writing tasks to run in parallel. This reduces data processing time through micro-batch processing and further reduces the size of a single write operation when writing the processed third data to the storage system, alleviating the pressure of data persistence to disk.

[0096] In one embodiment, generating a write task may include: Obtain the first partition information, which indicates multiple first partitions. Each first partition corresponds to a first fragment identifier and includes the second data. Based on the information of the first partition, a write task is generated. The write task is used to write the second data in the first partition into the first file set through the first file.

[0097] The first partition information can refer to the new partition information in the above embodiment. The first partition refers to the new partition mentioned above, and multiple first partitions include, for example, the following: Figure 3 The new partitions shown are 0 to M-1. Multiple first partitions can be one-to-one corresponded to multiple first shard identifiers.

[0098] For example, a second thread is used to initiate a first preset number of write tasks, each corresponding to a first partition. This example can be referenced to the explanation above regarding the write task for two new partitions occupying one thread. Multiple write tasks can be performed in parallel, thereby improving write efficiency. The first preset number of first partitions can be either consecutively numbered new partitions or non-consecutively numbered new partitions; there is no restriction.

[0099] Using the above scheme, multiple initial data entries can be in RDD form, divided into multiple original partitions. To facilitate the construction of the first file set corresponding to each first shard identifier, a repartitioning operation can be performed to obtain the first partition information. The purpose of the repartitioning operation is to ensure that a first partition includes first data entries with the same shard identifier, thus facilitating the writing of first data entries with the same shard identifier into the same first file set.

[0100] In one embodiment, obtaining the first partition information may include: The number of partitions is obtained based on the total number of records, the preset maximum number of partitions, and the preset maximum number of records. The total number of records includes the number of records in the first data section, and the preset maximum number of records includes the maximum number of records in the first partition. Obtain the information of the first partition, which is based on the number of partitions and the second fragment identifier.

[0101] This implementation method refers to the description of step 42 above.

[0102] Using the above scheme, the management node 21 can determine the number of partitions and instruct the computing node 22 to perform repartitioning operations according to the number of partitions and the shard identifier of each first data, so as to reasonably configure the number of first partitions and ensure that a first partition includes first data with the same shard identifier.

[0103] In one embodiment, generating a write task includes: Based on the second partition information, a write task is generated. The second partition information indicates multiple second partitions. Each second partition includes first data with different fragment identifiers. The write task is used to write the second data in the second partition to a set of first files through the first file.

[0104] The second partition can refer to the original partition in the above embodiment. Multiple second partitions include, for example, the following: Figure 3 The original partitions 1 to N are shown. An original partition may contain first data with different fragment identifiers.

[0105] For example, a second thread is used to initiate a second preset number of write tasks, which correspond to the second partition.

[0106] For example, original partition 1 includes first data a and first data b with shard identifier 00, and first data c with shard identifier 01. Original partition 2 includes first data d with shard identifier 00. For example, the second preset quantity is 2. Original partition 1 and original partition 2 correspond to one second thread. When the second thread runs, it can write the first data a and first data b in original partition 1 to file set (00) in file form, write the first data c in original partition 1 to file set (01) in file form, and write the first data d in original partition 2 to file set (00) in file form. Multiple write tasks can be performed in parallel, thereby improving write efficiency.

[0107] A file written to the file set can include the first data from one partition or combine the first data from multiple partitions, without restriction.

[0108] This embodiment can be used as a second implementation method for generating write tasks. If no repartitioning operation is performed, the first data with the same shard identifier can be found from each original partition, and the first data with the same shard identifier can be aggregated into the same file set, that is, the first file set corresponding to each first shard identifier can be constructed.

[0109] In one embodiment, the data processing method may include: For at least one first fragment identifier, the fragment processing progress is obtained based on the first metadata. The first metadata includes the file metadata of the first file, the file metadata includes the number of fourth data entries, the fourth data includes the second data in the first file, and the fragment processing progress is used to characterize the processing progress of the second data.

[0110] The method for tracking the processing progress of each data shard can be found in the description above and will not be repeated here. Taking shard identifier 01 as an example, the fourth data can be found in the pending data of shard identifier 01 in the Parquet file mentioned above.

[0111] Using the above scheme, the dynamic processing progress of each small batch of data can be statistically analyzed. While reducing the time consumption through micro-batch processing, the fragment processing progress can be obtained through file metadata, without triggering a full data read to statistically analyze the number of fragments.

[0112] In one embodiment, the data processing method may include: In response to a processing task failure, the failed processing task will be retried.

[0113] The above approach allows for individual retries of failed processing tasks, reducing data processing time. Compared to offline batch processing methods that retry the writing of the entire dataset, this approach has a lower cost for retrying failed tasks.

[0114] Based on the same inventive concept, a data processing device is also provided. Figure 6 This is a block diagram of an exemplary data processing apparatus, such as... Figure 6 As shown, the data processing device 60 may include: The first acquisition module 61 is configured to acquire a plurality of first fragment identifiers, the plurality of first fragment identifiers including a plurality of second fragment identifiers, the second fragment identifiers including fragment identifiers of first data, and the second fragment identifiers being obtained based on the data content of the first data; The first startup module 62 is configured to start multiple processing tasks, each processing task corresponding to the first shard identifier. The processing task is used to process the second data to obtain the third data, the second data including the first data corresponding to the first shard identifier.

[0115] Optionally, the first startup module 62 is configured as follows: Run multiple first threads to initiate the multiple processing tasks, wherein each first thread corresponds to a processing task; The processing task is used to read a newly added first file from the first file set, process the second data in the first file, and obtain the third data. The first file set corresponds to the first fragment identifier.

[0116] Optionally, the data processing device 60 includes: The generation module is configured to generate a write task, which is used to write the first file into the first file set. The second startup module is configured to run a second thread to start the write task. The first thread and the second thread belong to a thread pool. The second thread includes threads that are not occupied by the processing task. The total number of threads in the thread pool is greater than a first number, which includes the number of processing tasks.

[0117] Optionally, the generation module includes: The submodule is configured to obtain first partition information, which indicates multiple first partitions, each of which corresponds to a first shard identifier, and the first partition includes the second data. The first generation submodule is configured to generate the write task based on the first partition information. The write task is used to write the second data in the first partition into the first file set through the first file.

[0118] Optionally, the first obtaining submodule is configured as follows: The number of partitions is obtained based on the total number of records, the preset maximum number of partitions, and the preset maximum number of records. The total number of records includes the number of records of the first data, and the preset maximum number of records includes the maximum number of records of the first partition. Obtain the first partition information, which is based on the number of partitions and the second fragment identifier.

[0119] Optionally, the generation module includes: The second generation submodule is configured to generate the write task based on the second partition information, wherein the second partition information indicates multiple second partitions, and the second partitions include the first data with different fragment identifiers. The write task is used to write the second data in the second partitions into the first file set through the first file.

[0120] Optionally, the data processing device 60 includes: The second obtaining module is configured to obtain the fragment processing progress based on first metadata for at least one first fragment identifier. The first metadata includes file metadata of the first file, the file metadata includes the number of fourth data entries, the fourth data includes the second data in the first file, and the fragment processing progress is used to characterize the processing progress of the second data.

[0121] Optionally, the data processing device 60 includes: The retry module is configured to retry the failed processing task in response to the failure of the processing task.

[0122] Using a data processing device 60, the fragment identifier of the first data can be used for data fragmentation. This means that first data with the same fragment identifier belongs to the same data fragment. Each processing task corresponds to a first fragment identifier and is used to process the first data corresponding to that first fragment identifier. Thus, each portion of first data with the same fragment identifier can correspond to one processing task, and each processing task can be understood as being responsible for processing a small batch of data. Through multiple processing tasks, each small batch of data can be processed in parallel, thereby adopting a micro-batch processing approach, reducing data processing time, and improving data processing efficiency.

[0123] The effects that each module in the data processing device 60 can achieve can be found in the effect descriptions of the corresponding steps in the method.

[0124] The following is for reference. Figure 7The diagram illustrates a structural schematic of an electronic device 600 suitable for implementing the above-described method. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Personal Computers), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs (Televisions), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not be construed as limiting its functionality or scope of use.

[0125] like Figure 7 As shown, electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. The random access memory 603 also stores various programs and data required for the operation of electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0126] Typically, the following devices can be connected to the input / output interface 605: input devices 606 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 608 including, for example, magnetic tape, hard disk, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0127] In particular, depending on certain circumstances, the processes described in the flowchart above can be implemented as computer software programs. For example, a computer program product is provided, comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. This computer program can be downloaded and installed from a network via communication device 609, or installed from storage device 608, or installed from read-only memory 602. When the computer program is executed by processing device 601, it performs the functions defined in the above-described methods.

[0128] It should be noted that the aforementioned computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In one case, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In another case, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0129] In some scenarios, management nodes can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communications of any form or medium (e.g., communication networks). Examples of communication networks include Local Area Networks (LANs), Wide Area Networks (WANs), Internets (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0130] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0131] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: obtain a plurality of first fragment identifiers, the plurality of first fragment identifiers including a plurality of second fragment identifiers, the second fragment identifiers including fragment identifiers of first data, the second fragment identifiers being obtained based on the data content of the first data; Multiple processing tasks are initiated, each corresponding to the first shard identifier. The processing tasks are used to process the second data to obtain the third data, whereby the second data includes the first data corresponding to the first shard identifier.

[0132] Computer program code for performing the above operations can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages, as well as conventional procedural programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0133] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products under various scenarios. Each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that, in some alternative cases, the functions indicated in the blocks may occur in a different order than those indicated in the figures. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0134] The modules mentioned above can be implemented in software or hardware. In some cases, the name of a module does not necessarily limit the module itself; for example, the first acquisition module can also be described as "the module that acquires multiple first fragment identifiers".

[0135] The functions described above can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Parts (ASSPs), Systems on Chips (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0136] In this context, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] The above description is merely illustrative and explains the technical principles employed. Those skilled in the art should understand that the scope of this document is not limited to the specific combinations of the above-described technical features, but should also cover any combination of the above-described technical features or their equivalents without departing from the above concept.

[0138] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, while some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of this paper. Certain features described in the context of a single example can also be implemented in combination in a single example. Conversely, various features described in the context of a single example can also be implemented individually or in any suitable sub-combination in multiple examples.

[0139] Although this document has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims. Regarding the aforementioned apparatus, the specific manner in which the various modules perform their operations has already been described in detail in the section concerning the method, and will not be elaborated upon here.

Claims

1. A data processing method, comprising: A plurality of first fragment identifiers are obtained, wherein the plurality of first fragment identifiers include a plurality of second fragment identifiers, wherein the second fragment identifiers include fragment identifiers of first data, and the second fragment identifiers are obtained based on the data content of the first data; Multiple processing tasks are initiated, each corresponding to the first shard identifier. The processing tasks are used to process the second data to obtain the third data, whereby the second data includes the first data corresponding to the first shard identifier.

2. The data processing method according to claim 1, wherein initiating multiple processing tasks includes: Run multiple first threads to initiate the multiple processing tasks, wherein each first thread corresponds to a processing task; The processing task is used to read a newly added first file from the first file set, process the second data in the first file, and obtain the third data. The first file set corresponds to the first fragment identifier.

3. The data processing method according to claim 2, wherein the method comprises: Generate a write task, the write task being used to write the first file into the first file set; Run a second thread to start the write task. The first thread and the second thread belong to a thread pool. The second thread includes threads that are not occupied by the processing task. The total number of threads in the thread pool is greater than a first number, which includes the number of processing tasks.

4. The data processing method according to claim 3, wherein generating the write task includes: Obtain first partition information, which indicates multiple first partitions, each first partition corresponding to a first shard identifier, and the first partition including the second data; Based on the first partition information, the write task is generated, and the write task is used to write the second data in the first partition into the first file set through the first file.

5. The data processing method according to claim 4, wherein obtaining the first partition information includes: The number of partitions is obtained based on the total number of records, the preset maximum number of partitions, and the preset maximum number of records. The total number of records includes the number of records of the first data, and the preset maximum number of records includes the maximum number of records of the first partition. Obtain the first partition information, which is based on the number of partitions and the second fragment identifier.

6. The data processing method according to claim 3, wherein generating the write task includes: The write task is generated based on the second partition information, which indicates multiple second partitions. Each second partition includes the first data with different shard identifiers. The write task is used to write the second data in the second partition to the first file set through the first file.

7. The data processing method according to claim 2, wherein the method comprises: For at least one of the first fragment identifiers, a fragment processing progress is obtained based on the first metadata, wherein the first metadata includes the file metadata of the first file, the file metadata includes the number of fourth data entries, the fourth data includes the second data in the first file, and the fragment processing progress is used to characterize the processing progress of the second data.

8. The data processing method according to claim 1, wherein the method comprises: In response to the failure of the processing task, the failed processing task is retried.

9. A data processing apparatus, comprising: The first acquisition module is configured to acquire multiple first fragment identifiers, the multiple first fragment identifiers including multiple second fragment identifiers, the second fragment identifiers including fragment identifiers of first data, and the second fragment identifiers being obtained based on the data content of the first data; The first startup module is configured to start multiple processing tasks, each processing task corresponding to the first shard identifier. The processing task is used to process the second data to obtain the third data, the second data including the first data corresponding to the first shard identifier.

10. A computer-readable medium having a computer program stored thereon, wherein, When executed by a processing device, the computer program implements the method described in any one of claims 1-8.

11. An electronic device, comprising: A storage device on which computer programs are stored; A processing apparatus for executing the computer program in the storage device to implement the method of any one of claims 1-8.

12. A computer program product comprising a computer program, wherein, When executed by a processor, the computer program implements the method described in any one of claims 1-8.

Images