Data processing method and device, electronic equipment and storage medium

Abstract

The application discloses a data processing method and device, electronic equipment and a storage medium. The method can be used for cloud data calculation. The method comprises the following steps: based on a data redistribution task distributed by a driving node, performing data redistribution processing, and storing to-be-processed data of a target data partition corresponding to the data redistribution task in a local. Based on a data reduction task distributed by the driving node, reading the redistribution data corresponding to the target data partition from the local, performing data reduction processing, and obtaining a data reduction processing result. The method improves the execution efficiency of data redistribution and data reduction by sequentially writing data in a data partition during redistribution and reading data from the local for processing during reduction processing.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to data processing methods, apparatus, electronic devices and storage media. Background Technology

[0002] In distributed computing, each computing node in each stage only processes a portion of the task's data. If the next stage needs to rely on all the computation results from the previous stages, a shuffling operation is required to reorganize and classify all the computation results from the previous stages. In existing technologies, the shuffling operation relies on third-party services for data management, or the currently running redistribution task performs random writes, or the data reduction task performs random reads, thus reducing the efficiency of data redistribution and data reduction. Summary of the Invention

[0003] This application provides a data processing method, apparatus, electronic device, and storage medium that improve the efficiency of data redistribution and data reduction.

[0004] On the one hand, this application provides a data processing method, the method comprising:

[0005] The system receives a data reallocation task assigned by the driver node; the data reallocation task corresponds to a target data partition, and the data reallocation task is used to perform data reallocation processing on multiple unprocessed data in the target data partition; the target data partition is at least one of multiple data partitions.

[0006] Data reallocation is performed based on the data characteristics of the multiple data to be processed, and the reallocation partition identifier corresponding to each of the multiple data to be processed is determined; the data partition corresponding to the reallocation partition identifier is one of the multiple data partitions.

[0007] Based on the redistribution partition identifiers corresponding to each of the multiple data to be processed, the redistribution data corresponding to the target data partition is stored locally;

[0008] Receives a data reduction task assigned by the driver node; the data reduction task corresponds to the target data partition.

[0009] The redistributed data in the target data partition is obtained locally, and the redistributed data is subjected to data reduction processing to obtain the data reduction processing result corresponding to the target data partition.

[0010] On the other hand, this application provides a data processing method, the method comprising:

[0011] A data redistribution task is generated; the data redistribution task corresponds to a target data partition, and the data redistribution task is used to perform data redistribution processing on multiple data items to be processed in the target data partition; the target data partition is at least one of multiple data partitions.

[0012] The data redistribution task is sent to each data processing node so that each data processing node performs data redistribution based on the data characteristics of the multiple data to be processed, determines the redistribution partition identifier corresponding to each of the multiple data to be processed, and stores the redistributed data corresponding to the target data partition locally based on the redistribution partition identifier corresponding to each of the multiple data to be processed; the data partition corresponding to the redistribution partition identifier is one of the multiple data partitions.

[0013] A data reduction task is generated; the data reduction task corresponds to a target data partition, and the data reduction task is used to perform data reduction processing on multiple redistributed data in the target data partition; the target data partition is at least one of multiple data partitions;

[0014] The data reduction task corresponding to each data processing node is sent to each data processing node. The target data partition corresponding to the data reduction task running in the data processing node is the same as the target data partition corresponding to the data redistribution task running in the data processing node, so that each data processing node obtains the redistributed data corresponding to the target data partition from the local machine and performs data reduction processing on the redistributed data to obtain the data reduction processing result corresponding to the target data partition.

[0015] On the other hand, a data processing apparatus is provided, the apparatus comprising:

[0016] The first task receiving module is used to receive a data redistribution task allocated by the driver node; the data redistribution task corresponds to a target data partition, and the data redistribution task is used to perform data redistribution processing on multiple unprocessed data in the target data partition; the target data partition is at least one of multiple data partitions.

[0017] The data redistribution module is used to redistribute data based on the data characteristics of the multiple data to be processed, and to determine the redistribution partition identifier corresponding to each of the multiple data to be processed; the data partition corresponding to the redistribution partition identifier is one of the multiple data partitions.

[0018] The data redistribution module is used to store the redistribution data corresponding to the target data partition locally based on the redistribution partition identifier corresponding to each of the multiple data to be processed.

[0019] The second task receiving module is used to receive the data reduction task allocated by the driving node; the data reduction task corresponds to the target data partition.

[0020] The data reduction processing module is used to obtain the redistributed data in the target data partition from the local machine, perform data reduction processing on the redistributed data, and obtain the data reduction processing result corresponding to the target data partition.

[0021] On the other hand, a data processing apparatus is provided, the apparatus comprising:

[0022] The first task generation module is used to generate a data redistribution task; the data redistribution task corresponds to a target data partition, and the data redistribution task is used to perform data redistribution processing on multiple unprocessed data in the target data partition; the target data partition is at least one of multiple data partitions.

[0023] The first task sending module is used to send the data redistribution task to each data processing node, so that each data processing node performs data redistribution based on the data characteristics of the multiple data to be processed, determines the redistribution partition identifier corresponding to each of the multiple data to be processed; and stores the redistributed data corresponding to the target data partition locally based on the redistribution partition identifier corresponding to each of the multiple data to be processed; the data partition corresponding to the redistribution partition identifier is one of the multiple data partitions.

[0024] The second task generation module is used to generate a data reduction task; the data reduction task corresponds to a target data partition, and the data reduction task is used to perform data reduction processing on multiple redistributed data in the target data partition; the target data partition is at least one of multiple data partitions.

[0025] The second task sending module is used to send the data reduction task corresponding to each data processing node to each data processing node. The target data partition corresponding to the data reduction task running in the data processing node is the same as the target data partition corresponding to the data redistribution task running in the data processing node, so that each data processing node obtains the redistributed data corresponding to the target data partition from the local machine, performs data reduction processing on the redistributed data, and obtains the data reduction processing result corresponding to the target data partition.

[0026] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement a data processing method as described above.

[0027] On the other hand, a computer-readable storage medium is provided, the storage medium including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement a data processing method as described above.

[0028] This application provides a data processing method, apparatus, electronic device, and storage medium. The data processing method includes: performing data redistribution processing based on a data redistribution task allocated by a driver node, storing the data to be processed in the target data partition corresponding to the data redistribution task locally; and reading the redistributed data corresponding to the target data partition from the local storage based on a data reduction task allocated by the driver node, performing data reduction processing, and obtaining a data reduction processing result. This method improves the execution efficiency of data redistribution and data reduction by sequentially writing data to the data partition during redistribution and reading and processing data locally during reduction. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram illustrating an application scenario of a data processing method provided in an embodiment of this application;

[0031] Figure 2 An interactive flowchart of a data processing method provided in an embodiment of this application;

[0032] Figure 3 This is a schematic diagram illustrating the changes in data partitions after data redistribution in a data processing method provided in this application embodiment;

[0033] Figure 4 This is a schematic diagram of the data reduction task corresponding to each data processing node in a data processing method provided in an embodiment of this application;

[0034] Figure 5 A flowchart illustrating the storage of local data in a data processing method provided in this application embodiment;

[0035] Figure 6 A flowchart illustrating the sending of first external data in a data processing method provided in this application embodiment;

[0036] Figure 7A flowchart illustrating the storage of second external data in a data processing method provided in this application embodiment;

[0037] Figure 8 This is a schematic diagram illustrating the sending of task execution completion prompts between reassigned subtasks in a data processing method provided in an embodiment of this application.

[0038] Figure 9 A flowchart of a data processing method provided in this application embodiment on the data processing node side;

[0039] Figure 10 A flowchart illustrating a data processing method on the driver node side, as provided in an embodiment of this application;

[0040] Figure 11 A schematic diagram illustrating the data storage of local data, first external data, and second external data during data writing in a data processing method provided in this application embodiment;

[0041] Figure 12 This is a schematic diagram illustrating the data processing method performed during a shuffling operation in a distributed computing system, as provided in an embodiment of this application.

[0042] Figure 13 This is a schematic diagram of the structure of a data processing device on the data processing node side, provided in an embodiment of this application.

[0043] Figure 14 This is a schematic diagram of the structure of a data processing device on the driving node side, provided in an embodiment of this application.

[0044] Figure 15 This is a schematic diagram of the hardware structure of a device for implementing the method provided in the embodiments of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0046] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. Furthermore, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein.

[0047] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0048] Please see Figure 1 The illustration shows an application scenario diagram of a data processing method provided in this application embodiment. The application scenario includes a driver node 110 and a data processing node 120. The driver node 110 assigns a data redistribution task to the data processing node 120. The data processing node 120 runs the data redistribution task to redistribute the data to be processed and write it to the corresponding target data partition. The redistributed data in the target data partition is stored locally on the data processing node 120. Based on the correspondence between the data redistribution task and the target data partition, the driver node 110 determines the data reduction task corresponding to each data processing node 120. The data processing node 120 runs the data reduction task and reads data from the local data for data reduction processing.

[0049] In this embodiment, the server includes a driver node 110 and a data processing node 120. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited herein.

[0050] Please see Figure 2 It demonstrates a data processing method, the method comprising:

[0051] S210. The driving node generates a data redistribution task; the data redistribution task corresponds to the target data partition, and the data redistribution task is used to redistribute multiple unprocessed data in the target data partition; the target data partition is at least one of multiple data partitions;

[0052] In some embodiments, a data redistribution task may include multiple redistribution subtasks, the number of which corresponds to the number of data partitions included in the target data partition. That is, there is a one-to-one correspondence between data partitions and redistribution subtasks. The redistribution subtasks can be map tasks performed during the shuffle operation of the distributed computing engine; each time a map task is run, data redistribution can be performed on the data to be processed. Figure 3 The diagram shows the changes in data partitions after data redistribution. After data redistribution, the data partitions of the data to be processed will change. For example, if there were originally n data to be processed, corresponding to 3 data partitions, after redistribution, these n data to be processed may correspond to 5 data partitions.

[0053] S220. The driving node sends a data reallocation task to each data processing node;

[0054] In some embodiments, data redistribution tasks can be randomly assigned to each data processing node. The data processing node corresponding to each data redistribution task can be determined based on the remainder of the ratio between the task sequence number identifier of the data redistribution task and the number of data processing nodes.

[0055] S230. The data processing node receives the data redistribution task assigned by the driver node;

[0056] In some embodiments, the target data partition corresponding to the data redistribution task received by the data processing node is consistent with the data partition to be processed by the data processing node. The data processing node can run the data redistribution task, so that each redistribution subtask in the data redistribution task redistributes the data to be processed in the corresponding data partition.

[0057] S240. The data processing node redistributes data based on the data characteristics of multiple data to be processed, and determines the redistribution partition identifier corresponding to each of the multiple data to be processed; the data partition corresponding to the redistribution partition identifier is one of the multiple data partitions.

[0058] In some embodiments, the data processing node can classify multiple data to be processed based on the data characteristics of multiple data to be processed, dividing the data to be processed that were originally in the same data partition into at least one category of data to be processed. Different data categories correspond to different data partitions, and the redistribution partition identifier corresponding to the data to be processed can be determined based on the data category corresponding to the data to be processed.

[0059] S250. The data processing node stores the redistributed data corresponding to the target data partition locally based on the redistribution partition identifier corresponding to each of the multiple data to be processed;

[0060] In some embodiments, reallocated data of the same category are stored in the same data partition. If the data to be processed in the original data partition is itself data in that data partition, it can be stored directly locally. If the data to be processed in the original data partition is not itself data in that data partition, it can be sent to other data processing nodes. Furthermore, it can receive data to be processed corresponding to its local data partition sent by other data processing nodes.

[0061] S260. The driving node generates a data reduction task; the target data partition corresponding to the data reduction task running in the data processing node is the same as the target data partition corresponding to the data redistribution task running in the data processing node;

[0062] In some embodiments, the data reduction task is used to perform operations such as aggregation or sorting on the data in the corresponding data partition. Based on the target data partition corresponding to the data redistribution task running in each data processing node, the data reduction task corresponding to each data processing node is determined, such that the target data partition corresponding to the data reduction task is consistent with the target data partition corresponding to the data redistribution task.

[0063] A data reduction task can include multiple reduction processing subtasks. The number of reduction processing subtasks in a data reduction task is the same as the number of data partitions included in the target data partition. In other words, there is a one-to-one correspondence between data partitions and reduction processing subtasks.

[0064] Please see Figure 4 ,like Figure 4 The diagram illustrates the data reduction tasks corresponding to each data processing node. Based on the correspondence between data processing nodes and target data partitions, the driving node assigns the data reduction tasks corresponding to each target data partition to the corresponding data processing node.

[0065] The data reduction task can be the reduction task (reducetask) when the distributed computing engine performs a shuffle operation. Each time the reduceetask is run, it can read data from the local machine and perform data reduction processing such as data aggregation or sorting.

[0066] S270. The driver node sends a data reduction task corresponding to the target data partition of each data processing node to each data processing node;

[0067] In some embodiments, the target data partition corresponding to the data reduction task sent by the driver node to each data processing node is consistent with the target data partition corresponding to the data redistribution task running in the data processing node.

[0068] S280. The data processing node receives the data reduction task assigned by the driver node; the data reduction task corresponds to the target data partition.

[0069] In some embodiments, the data processing node can run a data reduction task, such that each reduction processing subtask in the data reduction task performs data reduction processing on the data to be processed in the corresponding data partition.

[0070] S290. The data processing node obtains the redistributed data from the target data partition from the local machine, performs data reduction processing on the redistributed data, and obtains the data reduction processing result corresponding to the target data partition.

[0071] In some embodiments, when a data processing node runs a data reduction task, each reduction processing subtask in the data reduction task can directly obtain the redistributed data from the local data processing node to which it is assigned when processing the data to be processed in the corresponding data partition.

[0072] In some embodiments, see Figure 5 The reallocation partition identifier includes the local partition identifier, which is the partition identifier corresponding to the target data partition on the local side.

[0073] Based on the reallocation partition identifiers corresponding to each of the multiple data items to be processed, the reallocation data corresponding to the target data partition is stored locally, including:

[0074] S510. Determine local data from multiple pending data sets; the reallocation partition identifier corresponding to the local data is the local partition identifier;

[0075] S520. Write local data to the target data partition;

[0076] S530. Store the reallocated data in the target data partition locally.

[0077] In some embodiments, local data refers to the data whose reassignment partition identifier is determined during the data redistribution task run on the local data processing node and written to the target data partition corresponding to the local data processing node. When the data processing node runs the data redistribution task to redistribute multiple pieces of data to be processed, the partition identifiers corresponding to the data partitions containing the multiple pieces of data to be processed can be updated to obtain the redistribution partition identifier. If the redistribution partition identifier is the same as the partition identifier of the target data partition corresponding to the data processing node, the data to be processed corresponding to the redistribution partition identifier can be determined to be local data, and the redistribution partition identifier is the local partition identifier. Local data can be directly written to the target data partition, and the redistributed data in the target data partition is stored locally.

[0078] When a data redistribution task includes multiple redistribution subtasks, the data to be processed in the data partition corresponding to each redistribution subtask can be traversed. The redistribution partition identifier corresponding to the currently traversed data is determined. If the redistribution partition identifier is the same as the partition identifier of the data partition corresponding to the redistribution subtask being traversed, the currently traversed data is designated as local data, and the redistribution partition identifier is designated as the local partition identifier. The currently traversed data is written to the cache of the data partition corresponding to the local partition identifier. When the amount of data to be processed stored in the cache of this data partition reaches a preset data volume threshold, the redistributed data in this data partition is sequentially stored in the local disk of the data processing node.

[0079] Based on the redistribution partition identifier, local data stored locally on the data processing node is selected from the data to be processed. The local data in the data to be processed is directly written to the target data partition and stored locally on the data processing node, so that the local data can be quickly identified and directly stored, thereby improving the execution efficiency of the data redistribution task.

[0080] In some embodiments, see Figure 6 The reallocation partition identifier includes the external partition identifier, and the local partition identifier is the partition identifier corresponding to the external target data partition;

[0081] Data reallocation is performed based on the data characteristics of multiple data sets to be processed. After determining the reallocation partition identifier corresponding to each of the multiple data sets to be processed, the method also includes:

[0082] S610. Determine the first external data from multiple data to be processed; the reallocation partition identifier corresponding to the first external data is the external partition identifier;

[0083] S620. Determine the first data processing node corresponding to the first external data;

[0084] S630. Send the first external data to the first data processing node so that the first data processing node writes the first external data to the data partition corresponding to the external partition identifier.

[0085] In some embodiments, the first external data refers to the data whose reassignment partition identifier is determined during the data redistribution task run on the local data processing node, and which is written to the target data partition corresponding to another data processing node besides the local data processing node. When the data processing node runs the data redistribution task to redistribute multiple pieces of data to be processed, the partition identifiers corresponding to the data partitions where the multiple pieces of data to be processed are located can be updated to obtain the redistribution partition identifier. If the redistribution partition identifier is different from the partition identifier of the target data partition corresponding to the data processing node, the data to be processed corresponding to the redistribution partition identifier can be determined as the first external data, and the redistribution partition identifier is the external partition identifier. Based on the external partition identifier, the first data processing node corresponding to the first external data can be determined, and the first external data can be sent to the first data processing node. In the first data processing node, the first external data can be written to the data partition corresponding to the external partition identifier.

[0086] When a data redistribution task includes multiple redistribution subtasks, the data to be processed in the data partition corresponding to each redistribution subtask can be traversed to determine the redistribution partition identifier corresponding to the currently traversed data. If the redistribution partition identifier is different from the partition identifier of the data partition corresponding to the redistribution subtask being traversed, the currently traversed data to be processed is identified as the first external data, and the redistribution partition identifier is identified as the external partition identifier. After the first external data is sent to the first data processing node, the first data processing node can write the first external data into the cache of the data partition corresponding to the external partition identifier. When the amount of data to be processed stored in the cache of this data partition reaches a preset data volume threshold, the redistributed data in this data partition is stored sequentially into the local disk of the first data processing node.

[0087] Based on the redistribution partition identifier, the first external data is selected from the data to be processed and sent to the corresponding first data processing node for data writing. This allows non-local data to be quickly identified and sent to the corresponding data processing node for storage, thereby improving the accuracy of the data redistribution task.

[0088] In some embodiments, see Figure 7 The redistributed data includes local data and second external data; the second external data is the data sent by the external second data processing node, and the redistributed partition identifier corresponding to the second external data is the local partition identifier;

[0089] Based on the reallocation partition identifiers corresponding to each of the multiple data items to be processed, the reallocation data corresponding to the target data partition is stored locally, including:

[0090] S710. Receive second external data sent by the second data processing node, wherein the target data partition corresponding to the data redistribution task running in the second data processing node is different from that corresponding to the data redistribution task running in the local data processing node;

[0091] S720. Store local data and second external data locally.

[0092] In some embodiments, the second external data is the external data of the second data processing node determined during the process of the second data processing node running the data redistribution task. The data redistribution task running in the second data processing node is different from the data redistribution task running in the local data processing node, and the corresponding target data partition is also different.

[0093] The second data processing node sends the second external data to the local data processing node based on the reallocation partition identifier corresponding to the second external data. The local data processing node receives the second external data sent by the second data processing node, and the reallocation partition identifier corresponding to the second external data is the local partition identifier corresponding to the local data processing node.

[0094] Both local data and second external data are written to the target data partition corresponding to the local data processing node, and the local data and second external data are stored locally as the redistributed data of the target data partition.

[0095] After data redistribution, the corresponding data processing nodes can be determined based on the partition identifiers and the number of data processing nodes corresponding to the redistributed data. The remainder of the ratio between the partition identifiers and the number of data processing nodes determines the corresponding data processing node.

[0096] When a data redistribution task includes multiple redistribution subtasks, the redistribution subtasks corresponding to the redistributed data can be determined based on the partition identifier corresponding to the redistributed data and the number of redistribution subtasks running in the data processing node. The redistribution subtask corresponding to the redistributed data can be determined by taking the remainder of the ratio between the partition identifier corresponding to the redistributed data and the number of redistribution subtasks running in the data processing node.

[0097] Based on the redistribution partition identifier, the system receives the second external data sent by the second data processing node, writes the second external data into the target data partition, and stores it locally on the data processing node. This allows the data processing node to store all the data to be processed in the same data partition locally, thereby improving the comprehensiveness of the data redistribution task.

[0098] In some embodiments, after redistributing data based on the data characteristics of multiple data sets to be processed and determining the redistribution partition identifier corresponding to each of the multiple data sets to be processed, the method further includes:

[0099] Upon receiving a notification that the associated data redistribution task has finished executing, stop writing data to the target data partition; the associated data redistribution task is a data redistribution task running on data processing nodes other than the local data processing node.

[0100] In some embodiments, when a redistribution subtask in the local data processing node has traversed all the data to be processed in its corresponding data partition, it generates a notification message indicating that the redistribution task has finished executing, and sends this notification message to all other redistribution subtasks. Similarly, after completing the traversal of the data to be processed in its corresponding data partition, each redistribution subtask included in the associated data redistribution task also generates a notification message indicating that the redistribution task has finished executing. The local data processing node can receive these notification messages from each of the redistribution subtasks included in the associated data redistribution task. See also... Figure 8 ,like Figure 8 The diagram illustrates the sending of task completion notifications between reassigned subtasks. Specifically, after completing its traversal, reassigned subtask 1, running on data processing node 1, sends notifications to reassigned subtasks 2, 3, and others.

[0101] After the local data processing node receives the prompts from all the relocation subtasks in the associated data relocation task, it closes the data write stream of the target data partition and stops writing data to the target data partition. Before the local data processing node receives the prompts from all the relocation subtasks in the associated data relocation task, it keeps the data write stream of the target data partition open to facilitate writing the second external data sent by the second data processing node.

[0102] Upon receiving a notification that the associated data redistribution task has ended, stopping the writing of data to the target data partition on the local data processing node can prevent the omission of pending data that can be written to the target data partition in the associated data redistribution task, thereby improving the accuracy and comprehensiveness of data redistribution processing.

[0103] Please see Figure 9It demonstrates a data processing method applied to the data processing node side, the method comprising:

[0104] S910. Receive a data reallocation task allocated by the driver node; the data reallocation task corresponds to the target data partition, and the data reallocation task is used to perform data reallocation processing on multiple unprocessed data in the target data partition; the target data partition is at least one of multiple data partitions;

[0105] S920. Based on the data characteristics of multiple data to be processed, data reallocation is performed, and the reallocation partition identifier corresponding to each of the multiple data to be processed is determined; the data partition corresponding to the reallocation partition identifier is one of the multiple data partitions.

[0106] S930. Based on the reallocation partition identifiers corresponding to each of the multiple data to be processed, store the reallocation data corresponding to the target data partition locally;

[0107] S940. Receives a data reduction task assigned by the driver node; the data reduction task corresponds to the target data partition.

[0108] S950. Obtain the redistributed data from the target data partition from the local machine, perform data reduction processing on the redistributed data, and obtain the data reduction processing result corresponding to the target data partition.

[0109] In some embodiments, each data processing node may run a data redistribution task and a data reduction task, and the target data partition corresponding to the data redistribution task and the target data partition corresponding to the data reduction task running on each data processing node are the same.

[0110] Running a data redistribution task in the data processing node, based on the data characteristics of multiple data sets to be processed, redistributes the data and determines the redistribution partition identifier for each set of data. Please refer to [link to relevant documentation]. Figure 10 ,like Figure 10 The diagram illustrates the data storage of local data, the first external data, and the second external data during data writing. The reallocated partition identifier includes both a local partition identifier and an external partition identifier.

[0111] If the reallocated partition identifier is the same as the partition identifier of the target data partition corresponding to the data processing node, the data to be processed corresponding to the reallocated partition identifier can be determined to be local data, and the reallocated partition identifier is a local partition identifier. Local data can be directly written to the local target data partition. If the reallocated partition identifier is different from the partition identifier of the target data partition corresponding to the data processing node, the data to be processed corresponding to the reallocated partition identifier can be determined to be first external data, and the reallocated partition identifier is an external partition identifier. The first data processing node corresponding to the external partition identifier is determined, and the first external data is sent to the first data processing node for data writing.

[0112] The system receives second external data sent by a second data processing node, the reallocation partition identifier corresponding to which matches the partition identifier of the target data partition corresponding to the local data processing node. Based on the reallocation partition identifier corresponding to the second external data, the system writes the second external data to the local target data partition. Here, both the first and second data processing nodes are data processing nodes other than the local data processing node; the first and second data processing nodes can be the same data processing node or different data processing nodes.

[0113] After the data redistribution tasks corresponding to each data processing node have been completed, the data processing node can report the correspondence between the target data partition and the data processing node to the driver node. The driver node can then allocate data reduction tasks based on this correspondence.

[0114] The data processing node receives and runs the data reduction task assigned by the driver node. After requesting the data of the data partition to be processed by the data reduction task from the driver node, it can be determined that the data of the data partition to be processed by the data reduction task is on the local data processing node. The redistributed data in the target data partition can be directly obtained from the local data processing node, and the redistributed data is processed by data reduction to obtain the data reduction result corresponding to the target data partition.

[0115] To directly obtain the redistributed data from the target data partition locally on the data processing node, the operating system's file system caching and memory mapping techniques can be used to read the data and obtain the redistributed data from the target data partition.

[0116] By executing sequential writing of each data partition through the correspondence between data partitions and data processing nodes, and reading the redistributed data of each data partition through the correspondence between data partitions and data processing nodes, the steps of data query and data retrieval can be omitted, as well as the complex data index relationships between redistributed data. This improves the execution efficiency of data redistribution and data reduction processing, and avoids the data processing process from relying on third-party remote services, thereby reducing the cost of operation and maintenance and deployment.

[0117] Please see Figure 11 It demonstrates a data processing method applied to the driver node side, which includes:

[0118] S1110. Generate a data redistribution task;

[0119] S1120. Send a data redistribution task to each data processing node so that each data processing node redistributes data based on the data characteristics of multiple data to be processed, determines the redistribution partition identifier corresponding to each of the multiple data to be processed; and stores the redistributed data corresponding to the target data partition locally based on the redistribution partition identifier corresponding to each of the multiple data to be processed; the data partition corresponding to the redistribution partition identifier is one of the multiple data partitions.

[0120] S1130. Generate a data reduction task;

[0121] S1140. Send the data reduction task corresponding to each data processing node to each data processing node. The target data partition corresponding to the data reduction task running in the data processing node is the same as the target data partition corresponding to the data redistribution task running in the data processing node, so that each data processing node can obtain the redistributed data corresponding to the target data partition from the local machine and perform data reduction processing on the redistributed data to obtain the data reduction processing result corresponding to the target data partition.

[0122] In some embodiments, the driver node can generate data redistribution tasks and data reduction tasks, and assign these tasks to each data processing node. Each data redistribution task corresponds to a target data partition. After a data processing node completes its data redistribution task, it can send the correspondence between the target data partition and the data processing node to the driver node.

[0123] Based on the correspondence between target data partitions and data processing nodes, the driving node can assign the data reduction tasks corresponding to each target data partition to the corresponding data processing node, so that the target data partitions corresponding to the data reduction tasks running in the data processing node are the same as the target data partitions corresponding to the data redistribution tasks running in the data processing node.

[0124] Data redistribution tasks are generated and allocated based on the correspondence between data partitions and data processing nodes, enabling data redistribution tasks to write sequentially based on the corresponding data partitions. Subsequently, data reduction tasks are generated and allocated based on the correspondence between data partitions and data processing nodes, allowing data reduction tasks to read data directly from the local machine, thereby improving the execution efficiency of data redistribution and data reduction.

[0125] The data processing method provided in this application can be applied to the shuffling operation of a distributed computing system. During the shuffle operation, map tasks and reducer tasks can be executed using the aforementioned data processing method. During the shuffle operation, the driving node generates data redistribution subtasks equal to the number of data partitions, and assigns these subtasks to data processing nodes, with each subtask constituting a map task. These at least one data redistribution subtasks are the corresponding data redistribution tasks for that data processing node.

[0126] The local data processing node receives the data redistribution task and the target data partition corresponding to the data redistribution task. During the execution of each data redistribution subtask, the data to be processed in the data partition corresponding to the data redistribution subtask is traversed and processed. Based on the data characteristics of the currently traversed data to be processed, the data to be processed is redistributed, and the redistribution partition identifier corresponding to the currently traversed data to be processed is determined. The above data traversal process is repeated until all the data to be processed in the data partition has been traversed.

[0127] The reallocation partition identifier corresponding to the currently traversed data to be processed may be either a local partition identifier or an external partition identifier. The local partition identifier is the partition identifier of the target data partition corresponding to the local data processing node. If the reallocation partition identifier is a local partition identifier, the currently traversed data to be processed is written as local data into the cache corresponding to the data partition. When the amount of data in the cache reaches a preset data volume threshold, the cached data is stored in the local disk corresponding to the local data processing node. If the reallocation partition identifier is an external partition identifier, the currently traversed data to be processed is treated as first external data. The first data processing node corresponding to the first external data is determined, and the first external data is sent to the first data processing node, where it writes and stores the data in its corresponding local disk.

[0128] During the process of storing the data to be processed locally, the local data processing node can also receive second external data sent by the second data processing node. The reallocation partition identifier corresponding to the second external data is the local partition identifier. The local data processing node writes the second external data into the cache corresponding to the data partition, and when the amount of data in the cache reaches a preset data amount threshold, it stores the cached data into the local disk corresponding to the local data processing node.

[0129] Please see Figure 12 ,like Figure 12 The diagram illustrates data processing during a shuffling operation in a distributed computing system. When running a data redistribution task, data processing node 1 stores local data, sends the first external data corresponding to data processing node 2 to data processing node 2, and receives the second external data sent by data processing node 2. Similarly, it sends the first external data corresponding to data processing node 3 to data processing node 3 and receives the second external data sent by data processing node 3. The same logic applies to the sending and receiving of the first and second external data by data processing nodes 2 and 3 during the data redistribution task.

[0130] After each data reallocation subtask in the local data processing node completes its traversal, it generates a traversal completion message and sends this message to all other data reallocation subtasks except the one that has completed its traversal. Upon receiving traversal completion messages from all other data reallocation subtasks, any data reallocation subtask can stop writing data to the corresponding data partition.

[0131] After the data redistribution task is completed, the driver node can generate a reduction processing subtask for each data partition. Based on the correspondence between data partitions and data processing nodes, the reduction processing subtasks are assigned to the corresponding data processing nodes, ensuring that each data processing node receives the corresponding data reduction task. When the data reduction task in the data processing node performs data reduction processing, it can directly read the redistributed data from the local data processing node and perform data reduction processing to obtain the data reduction result.

[0132] This application provides a data processing method, which includes: performing data redistribution processing based on a data redistribution task allocated by a driver node, storing the data to be processed in the target data partition corresponding to the data redistribution task locally; and reading the redistributed data corresponding to the target data partition from the local storage based on a data reduction task allocated by the driver node, performing data reduction processing, and obtaining the data reduction processing result. This method improves the execution efficiency of data redistribution and data reduction by sequentially writing data to the data partition during redistribution and reading and processing data locally during reduction, while avoiding dependence on third-party remote services and reducing operation and deployment costs.

[0133] This application also provides a data processing apparatus, which is applied to a data processing node. Please refer to [link to relevant documentation]. Figure 13 The device includes:

[0134] The first task receiving module 1310 is used to receive a data redistribution task allocated by the driver node; the data redistribution task corresponds to the target data partition, and the data redistribution task is used to perform data redistribution processing on multiple unprocessed data in the target data partition; the target data partition is at least one of multiple data partitions.

[0135] The data redistribution module 1320 is used to redistribute data based on the data characteristics of multiple data to be processed, and to determine the redistribution partition identifier corresponding to each of the multiple data to be processed; the data partition corresponding to the redistribution partition identifier is one of the multiple data partitions.

[0136] The reallocation data storage module 1330 is used to store the reallocation data corresponding to the target data partition locally based on the reallocation partition identifier corresponding to each of the multiple data to be processed.

[0137] The second task receiving module 1340 is used to receive the data reduction task allocated by the driver node; the data reduction task corresponds to the target data partition.

[0138] The data reduction processing module 1350 is used to obtain the redistributed data from the target data partition from the local machine, perform data reduction processing on the redistributed data, and obtain the data reduction processing result corresponding to the target data partition.

[0139] In some embodiments, the reallocation partition identifier includes a local partition identifier, which is the partition identifier corresponding to the target data partition on the local side.

[0140] The data storage reallocation module includes:

[0141] The local data determination unit is used to determine local data from multiple data to be processed; the reallocation partition identifier corresponding to the local data is the local partition identifier.

[0142] The local data writing unit is used to write local data to the target data partition;

[0143] The first redistribution data storage unit is used to store the redistribution data from the target data partition locally.

[0144] In some embodiments, the reallocation partition identifier includes the external partition identifier, and the local partition identifier is the partition identifier corresponding to the external target data partition.

[0145] The data storage reallocation module also includes:

[0146] The first external data determination unit is used to determine the first external data from multiple data to be processed; the reallocation partition identifier corresponding to the first external data is the external partition identifier.

[0147] The first data processing node determination unit is used to determine the first data processing node corresponding to the first external data.

[0148] The first external data sending unit is used to send the first external data to the first data processing node, so that the first data processing node writes the first external data into the data partition corresponding to the external partition identifier.

[0149] In some embodiments, the redistribution data includes local data and second external data; the second external data is data sent by an external second data processing node, and the redistribution partition identifier corresponding to the second external data is the local partition identifier;

[0150] The data storage reallocation module also includes:

[0151] The second external data receiving unit is used to receive the second external data sent by the second data processing node. The target data partition corresponding to the data redistribution task running in the second data processing node is different from that of the data redistribution task running in the local data processing node.

[0152] The first redistribution data storage unit is used to store local data and second external data locally.

[0153] In some embodiments, the device further includes:

[0154] The write operation stop module is used to stop the data write operation on the target data partition when a prompt message indicating that the associated data redistribution task has been completed is received; the associated data redistribution task is a data redistribution task running on data processing nodes other than the local data processing node.

[0155] This application also provides a data processing apparatus, which is applied to the driving node side. Please refer to [link to relevant documentation]. Figure 14 The device includes:

[0156] The first task generation module 1410 is used to generate a data redistribution task; the data redistribution task corresponds to a target data partition, and the data redistribution task is used to redistribute multiple data to be processed in the target data partition; the target data partition is at least one of multiple data partitions.

[0157] The first task sending module 1420 is used to send a data redistribution task to each data processing node, so that each data processing node redistributes data based on the data characteristics of multiple data to be processed, determines the redistribution partition identifier corresponding to each of the multiple data to be processed, and stores the redistribution data corresponding to the target data partition locally based on the redistribution partition identifier corresponding to each of the multiple data to be processed; the data partition corresponding to the redistribution partition identifier is one of the multiple data partitions.

[0158] The second task generation module 1430 is used to generate a data reduction task; the data reduction task corresponds to the target data partition, and the data reduction task is used to perform data reduction processing on multiple redistributed data in the target data partition; the target data partition is at least one of multiple data partitions.

[0159] The second task sending module 1440 is used to send the data reduction task corresponding to each data processing node to each data processing node. The target data partition corresponding to the data reduction task running in the data processing node is the same as the target data partition corresponding to the data redistribution task running in the data processing node, so that each data processing node can obtain the redistributed data corresponding to the target data partition from the local machine, and perform data reduction processing on the redistributed data to obtain the data reduction processing result corresponding to the target data partition.

[0160] The apparatus provided in the above embodiments can execute the methods provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in the above embodiments can be found in a data processing method provided in any embodiment of this application.

[0161] This embodiment also provides a computer-readable storage medium storing computer-executable instructions, which are loaded by a processor and executed by the data processing method described above in this embodiment.

[0162] This embodiment also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations of the data processing described above.

[0163] This embodiment also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program adapted to be loaded by the processor and executed as described above in this embodiment of a data processing method.

[0164] The device may be a computer terminal, a mobile terminal, or a server, and may also participate in constituting the apparatus or system provided in the embodiments of this application. For example... Figure 15 As shown, server 15 may include one or more processors 1502 (shown as 1502a, 1502b, ..., 1502n in the figure) (processor 1502 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPLD, etc.), a memory 1504 for storing data, and a transmission device 1506 for communication functions. In addition, it may also include: input / output interfaces (I / O interfaces) and network interfaces. Those skilled in the art will understand that... Figure 15 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 15 may also include... Figure 15 The more or fewer components shown, or having the same Figure 15 The different configurations shown.

[0165] It should be noted that the aforementioned one or more processors 1502 and / or other data processing circuitry are generally referred to herein as "data processing circuitry". This data processing circuitry may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the server 15.

[0166] The memory 1504 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method described in the embodiments of this application. The processor 1502 executes various functional applications and data processing by running the software programs and modules stored in the memory 1504, thereby realizing the above-described method for generating temporal behavior capture boxes based on self-attention networks. The memory 1504 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1504 may further include memory remotely located relative to the processor 1502, and these remote memories can be connected to the server 15 via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0167] The transmission device 1506 is used to receive or send external data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 15. In one example, the transmission device 1506 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 1506 may be a radio frequency (RF) module for wireless communication with the Internet.

[0168] This specification provides the operational steps of the methods described in the embodiments or flowcharts, but more or fewer operational steps may be included based on conventional or non-inventive labor. The steps and order listed in the embodiments are merely one possible execution order among many steps and do not represent the only execution order. In actual system or interrupt product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0169] The structure shown in this embodiment is only a partial structure related to the solution of this application and does not constitute a limitation on the device to which the solution of this application is applied. Specific devices may include more or fewer components than shown, or combinations of certain components, or arrangements of different components. It should be understood that the methods, apparatuses, etc., disclosed in this embodiment can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or unit modules through some interfaces.

[0170] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0171] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0172] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, The method includes: The system receives a data reallocation task assigned by the driver node; the data reallocation task corresponds to a target data partition, and the data reallocation task is used to perform data reallocation processing on multiple unprocessed data in the target data partition; the target data partition is at least one of multiple data partitions. Data reallocation is performed based on the data characteristics of the multiple data to be processed, and the reallocation partition identifier corresponding to each of the multiple data to be processed is determined; the data partition corresponding to the reallocation partition identifier is one of the multiple data partitions. Based on the redistribution partition identifiers corresponding to each of the multiple data to be processed, the redistribution data corresponding to the target data partition is stored locally; the redistribution data corresponding to the other data partitions besides the target data partition is sent to other data processing nodes; and the redistribution data corresponding to the target data partition sent by other data processing nodes is received. Receives a data reduction task assigned by the driver node; the data reduction task corresponds to the target data partition. The redistributed data in the target data partition is obtained locally, and the redistributed data is subjected to data reduction processing to obtain the data reduction processing result corresponding to the target data partition.

2. The data processing method according to claim 1, characterized in that, The reallocation partition identifier includes a local partition identifier, which is the partition identifier corresponding to the target data partition in the local area. The step of storing the reassigned data corresponding to the target data partition locally based on the reassignment partition identifier corresponding to each of the multiple data to be processed includes: Local data is determined from the multiple unprocessed data sets; the reallocation partition identifier corresponding to the local data is the local partition identifier. Write the local data into the target data partition; The redistributed data in the target data partition is stored locally.

3. The data processing method according to claim 2, characterized in that, The reallocation partition identifier includes an external partition identifier, which is the partition identifier corresponding to the external target data partition. After redistributing data based on the data characteristics of the multiple datasets to be processed, and determining the redistribution partition identifier corresponding to each of the multiple datasets to be processed, the method further includes: A first external data is determined from the plurality of data to be processed; the reallocation partition identifier corresponding to the first external data is the external partition identifier; Determine the first data processing node corresponding to the first external data; The first external data is sent to the first data processing node so that the first data processing node writes the first external data into the data partition corresponding to the external partition identifier.

4. The data processing method according to claim 2, characterized in that, The redistribution data includes the local data and the second external data; the second external data is data sent by an external second data processing node, and the redistribution partition identifier corresponding to the second external data is the local partition identifier; The step of storing the reassigned data corresponding to the target data partition locally based on the reassignment partition identifier corresponding to each of the multiple data to be processed includes: The system receives second external data sent by the second data processing node, wherein the target data partition corresponding to the data redistribution task running in the second data processing node is different from that corresponding to the data redistribution task running in the local data processing node. The local data and the second external data are stored locally.

5. The data processing method according to claim 1, characterized in that, After redistributing data based on the data characteristics of the multiple datasets to be processed, and determining the redistribution partition identifier corresponding to each of the multiple datasets to be processed, the method further includes: Upon receiving a notification that the associated data redistribution task has been completed, the data writing operation on the target data partition is stopped; the associated data redistribution task is a data redistribution task running on data processing nodes other than the local data processing node.

6. A data processing method, characterized in that, The method includes: A data redistribution task is generated; the data redistribution task corresponds to a target data partition, and the data redistribution task is used to perform data redistribution processing on multiple data items to be processed in the target data partition; the target data partition is at least one of multiple data partitions. The data redistribution task is sent to each data processing node, enabling each node to redistribute data based on the data characteristics of the plurality of data to be processed, determine the redistribution partition identifier corresponding to each of the plurality of data to be processed; and, based on the redistribution partition identifier corresponding to each of the plurality of data to be processed, store the redistribution data corresponding to the target data partition locally; the data partition corresponding to the redistribution partition identifier is one of the plurality of data partitions; the redistribution data corresponding to the other data partitions in the plurality of data partitions besides the target data partition are sent to other data processing nodes; and the redistribution data corresponding to the target data partition sent by other data processing nodes is received. A data reduction task is generated; the data reduction task corresponds to a target data partition, and the data reduction task is used to perform data reduction processing on multiple redistributed data in the target data partition; the target data partition is at least one of multiple data partitions; The data reduction task corresponding to each data processing node is sent to each data processing node. The target data partition corresponding to the data reduction task running in the data processing node is the same as the target data partition corresponding to the data redistribution task running in the data processing node, so that each data processing node obtains the redistributed data corresponding to the target data partition from the local machine and performs data reduction processing on the redistributed data to obtain the data reduction processing result corresponding to the target data partition.

7. A data processing apparatus, characterized in that, The device includes: The first task receiving module is used to receive a data redistribution task allocated by the driver node; the data redistribution task corresponds to a target data partition, and the data redistribution task is used to perform data redistribution processing on multiple unprocessed data in the target data partition; the target data partition is at least one of multiple data partitions. The data redistribution module is used to redistribute data based on the data characteristics of the multiple data to be processed, and to determine the redistribution partition identifier corresponding to each of the multiple data to be processed; the data partition corresponding to the redistribution partition identifier is one of the multiple data partitions. The data redistribution storage module is used to store the redistribution data corresponding to the target data partition locally based on the redistribution partition identifier corresponding to each of the multiple data to be processed; send the redistribution data corresponding to the other data partitions among the multiple data partitions except the target data partition to other data processing nodes; and receive the redistribution data corresponding to the target data partition sent by other data processing nodes. The second task receiving module is used to receive the data reduction task allocated by the driving node; the data reduction task corresponds to the target data partition. The data reduction processing module is used to obtain the redistributed data in the target data partition from the local machine, perform data reduction processing on the redistributed data, and obtain the data reduction processing result corresponding to the target data partition.

8. The apparatus according to claim 7, characterized in that, The reallocation partition identifier includes a local partition identifier, which is the partition identifier corresponding to the target data partition in the local area. The reallocation data storage module includes: The local data determination unit is used to determine local data from multiple data to be processed; the reallocation partition identifier corresponding to the local data is the local partition identifier. The local data writing unit is used to write local data to the target data partition; The first redistribution data storage unit is used to store the redistribution data from the target data partition locally.

9. The apparatus according to claim 8, characterized in that, The reallocation partition identifier includes an external partition identifier, which is the partition identifier corresponding to the external target data partition. The reallocation data storage module further includes: The first external data determination unit is used to determine the first external data from multiple data to be processed; the reallocation partition identifier corresponding to the first external data is the external partition identifier. The first data processing node determination unit is used to determine the first data processing node corresponding to the first external data. The first external data sending unit is used to send the first external data to the first data processing node, so that the first data processing node writes the first external data into the data partition corresponding to the external partition identifier.

10. The apparatus according to claim 8, characterized in that, The redistribution data includes local data and second external data; the second external data is data sent by an external second data processing node, and the redistribution partition identifier corresponding to the second external data is the local partition identifier; The reallocation data storage module further includes: The second external data receiving unit is used to receive the second external data sent by the second data processing node. The target data partition corresponding to the data redistribution task running in the second data processing node is different from that of the data redistribution task running in the local data processing node. The first redistribution data storage unit is used to store local data and second external data locally.

11. The apparatus according to claim 7, characterized in that, The device further includes: The write operation stop module is used to stop the data write operation on the target data partition when a prompt message indicating that the associated data redistribution task has been completed is received; the associated data redistribution task is a data redistribution task running on data processing nodes other than the local data processing node.

12. A data processing apparatus, characterized in that, The device includes: The first task generation module is used to generate a data redistribution task; the data redistribution task corresponds to a target data partition, and the data redistribution task is used to perform data redistribution processing on multiple unprocessed data in the target data partition; the target data partition is at least one of multiple data partitions. The first task sending module is configured to send the data redistribution task to each data processing node, so that each data processing node performs data redistribution based on the data characteristics of the plurality of data to be processed, determines the redistribution partition identifier corresponding to each of the plurality of data to be processed; and stores the redistribution data corresponding to the target data partition locally based on the redistribution partition identifier corresponding to each of the plurality of data to be processed; the data partition corresponding to the redistribution partition identifier is one of the plurality of data partitions; sends the redistribution data corresponding to the other data partitions in the plurality of data partitions besides the target data partition to other data processing nodes; and receives the redistribution data corresponding to the target data partition sent by other data processing nodes. The second task generation module is used to generate a data reduction task; the data reduction task corresponds to a target data partition, and the data reduction task is used to perform data reduction processing on multiple redistributed data in the target data partition; the target data partition is at least one of multiple data partitions. The second task sending module is used to send the data reduction task corresponding to each data processing node to each data processing node. The target data partition corresponding to the data reduction task running in the data processing node is the same as the target data partition corresponding to the data redistribution task running in the data processing node, so that each data processing node obtains the redistributed data corresponding to the target data partition from the local machine, performs data reduction processing on the redistributed data, and obtains the data reduction processing result corresponding to the target data partition.

13. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement a data processing method as described in any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, The storage medium includes a memory that stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by a processor to implement a data processing method as described in any one of claims 1-6.

15. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a data processing method as described in any one of claims 1-6.

Images