Data processing system and method, and related device

Abstract

A data processing system and method, and a related device, which relate to the technical field of data processing. The data processing system comprises a first computing node, a second computing node and a plurality of storage nodes. The first computing node is used for determining a partition to which data corresponding to each map task belongs, and a storage node which stores data of each partition, and storing first metadata and writing into the plurality of storage nodes the data corresponding to each map task, wherein the first metadata comprises partition information of partitions to which the data belongs, and information about the storage node which stores the data of each partition. The plurality of storage nodes are used for aggregating data which belongs to the same partition. The second computing node is used for acquiring, from the plurality of storage nodes on the basis of acquired first metadata, data of partitions corresponding to a reduce operation during the process of the first computing node writing the data corresponding to each map task into the plurality of storage nodes. In this way, the utilization rate of network resources of a plurality of storage nodes can be increased.

Description

Data processing system, method and related device

[0001] The present application claims priority from the Chinese patent application No. 202411402041.3 filed on September 30, 2024, and entitled "Data processing system, method and related device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of data processing, and in particular to a data processing system, method and related device. BACKGROUND

[0003] Remote shuffle service (RSS) is an architecture design of distributed computing, which aims to provide shuffle service by using separate storage nodes, so as to separate shuffle operation from computing task, that is, to decouple computing and storage. Shuffle operation refers to partitioning and sorting data in storage nodes during computing, and providing subsequent reduce operation on the data after shuffle operation by computing nodes.

[0004] At present, under the RSS architecture, the computing nodes usually first send data to the storage nodes for partitioning and saving, and then perform reduce operation on the data belonging to the same partition by the computing nodes after all data is saved in the storage nodes. However, this way of processing data not only has low efficiency, but also causes serious waste of network resources of the storage nodes. SUMMARY

[0005] The present application provides a data processing system to improve the efficiency of processing data and alleviate the problem of network resource waste. In addition, the present application also provides a corresponding data processing method, a computing device, a computer readable storage medium and a computer program product.

[0006] In a first aspect, the present application provides a data processing system, comprising a plurality of computing nodes and a plurality of storage nodes, the computing nodes can be nodes with computing capability such as servers, and the storage nodes can be nodes with storage capability such as storage servers, the plurality of computing nodes comprise a first computing node for executing Map tasks and a second computing node for executing reduce operations. The first computing node refers to one or more computing nodes, each of which executes at least one Map task, and similarly, the second computing node can also be one or more computing nodes. The first computing node and the second computing node can be the same computing node or different computing nodes, and no limitation is made in this regard. The first computing node is configured to determine a partition to which data corresponding to each of the plurality of Map tasks belongs, and in an actual application scenario, the data corresponding to each of the plurality of Map tasks can comprise data belonging to a plurality of partitions. The first computing node is further configured to determine a storage node for storing data of each of the plurality of partitions, and store first metadata, the first metadata comprising partition information of a partition to which data corresponding to each of the plurality of Map tasks belongs (which can be used to describe which partition the data belongs to) and information of a storage node for storing data of each of the plurality of partitions (which can be used to describe in which storage node the data of the partition is stored). The first computing node is further configured to write data corresponding to each of the plurality of Map tasks into the plurality of storage nodes. The plurality of storage nodes are configured to aggregate data belonging to a same partition in data corresponding to each of the plurality of Map tasks, and after the data belonging to the same partition is aggregated, the data can be written into a storage area corresponding to the partition in the storage node. In an actual application, when storing data corresponding to the Map tasks, the storage node also stores second metadata, the second metadata being used to describe a storage location of the data in the storage node. The second computing node is configured to acquire the first metadata during the process in which the first computing node writes data corresponding to each of the plurality of Map tasks into the plurality of storage nodes, and acquire data of a partition corresponding to a reduce operation from the plurality of storage nodes according to the first metadata.

[0007] Since the first computing node generates the first metadata corresponding to the partition information of the data of the Map task and the storage node where the partition is located before storing the data of the Map task to the storage node (i.e. between performing the shuffle write), the second computing node performing the Reduce operation can obtain the first metadata in the process of the first computing node writing the data to the plurality of storage nodes (i.e. the process of performing the shuffle write), and read the data of the partition corresponding to the reduce operation which has been saved from the corresponding storage node according to the first metadata (i.e. performing the shuffle read operation), so that the second computing node performs the corresponding reduce operation on the read partition data. In this way, in the process of the first computing node writing the data to the plurality of storage nodes, the second computing node performing the reduce operation can perform the process of reading the data of the partition in parallel, i.e. the second computing node can start the reduce operation before the data of the Map task is completely written to the storage node, which not only improves the efficiency of data processing, but also can utilize the uplink bandwidth resources and the downlink bandwidth resources of the plurality of storage nodes in parallel, and improves the utilization rate of the network resources of the plurality of storage nodes.

[0008] In a possible implementation, the first computing node is further configured to instruct the second computing node to obtain the data of the partition corresponding to the reduce operation from the plurality of storage nodes after the first computing node saves the first metadata corresponding to all the Map tasks in the process of the first computing node writing the data corresponding to each Map task to the plurality of storage nodes. In this way, the first computing node can store the first metadata corresponding to all the Map tasks, i.e. can save the complete first metadata. In this way, the second computing node can access the data of the corresponding partition based on the complete first metadata when performing the reduce task, so as to improve the reliability of the second computing node performing the reduce task.

[0009] In a possible implementation, when storing the first metadata, the first computing node stores the first metadata corresponding to the target data of the target Map task in the plurality of Map tasks, for example, the first computing node specifically stores the first metadata corresponding to the target data after writing the target data of the target Map task in the plurality of Map tasks into the sending buffer area in the first computing node; then, when the first computing node instructs the second computing node to obtain the data of the partition corresponding to the reduce operation from the plurality of storage nodes after the first computing node stores the first metadata corresponding to all the Map tasks, the first computing node can specifically increase the number of the sending buffer area in the first computing node, write all the remaining data of the data corresponding to the plurality of Map tasks that are not sent to the plurality of storage nodes for storage into the sending buffer area, and store the first metadata corresponding to the remaining data, so that the first computing node instructs the second computing node to obtain the data of the partition corresponding to the reduce operation from the plurality of storage nodes. In this way, the first computing node can realize the process of writing all the data corresponding to the plurality of Map tasks into the sending buffer area by increasing the number of the sending buffer area, which enables the first computing node to store the first metadata corresponding to all the data corresponding to the plurality of Map tasks, that is, to store the complete first metadata. At this time, the first computing node triggers the second computing node to execute the reduce task, which enables the second computing node to access the data of the corresponding partition based on the complete first metadata, and ensures the reliability of the second computing node executing the reduce task.

[0010] In a possible implementation, when increasing the number of the sending buffer area in the first computing node, the first computing node can specifically estimate a first time length, which is a time length for transmitting the data corresponding to the Map task that has been stored in the plurality of storage nodes to the second computing node. In addition, the first computing node is further configured to obtain a second time length, which is a remaining time length for the second computing node to store all the data corresponding to the plurality of Map tasks into the plurality of storage nodes. Therefore, the first computing node is further configured to increase the number of the sending buffer area in the first computing node when the first time length is greater than or equal to the second time length. In this way, the first computing node can increase the number of the sending buffer area by estimating the time length for transmitting the remaining data to the plurality of storage nodes for storage and the time length for the second computing node to read the stored data from the storage nodes, so as to automatically trigger the second computing node to execute the reduce task in advance in the case that all the data corresponding to the plurality of Map tasks are cached into the sending buffer area. In this way, the first computing node not only can improve the overall efficiency of the second computing node for reducing the data, but also can utilize the uplink bandwidth resources and the downlink bandwidth resources of the plurality of storage nodes as much as possible.

[0011] In a possible implementation, when estimating the first time length, the first computing node is specifically configured to: determine the quantity of data corresponding to the plurality of Map tasks that has been saved to the plurality of storage nodes; and estimate the first time length according to the quantity, and the bandwidth of the first computing node writing data to the plurality of storage nodes.

[0012] In a possible implementation, the first computing node is further configured to: obtain the available resources of the second computing node; and in response to the indication that the second computing node obtains the data of the partition corresponding to the reduce operation from the plurality of storage nodes, the first computing node is specifically configured to: in a case where the available resources of the second computing node satisfy the task execution condition, and the first time length is greater than or equal to the second time length, instruct the second computing node to obtain the data of the partition corresponding to the reduce operation from the plurality of storage nodes. In this way, the second computing node is instructed to start executing the reduce task in combination with the available resources of the second computing node, so that the reduce task can be started to be executed in a case where the available resources of the second computing node are sufficient, thereby avoiding the insufficient resources of the second computing node from affecting the execution of the reduce task, and improving the rationality and reliability of task scheduling.

[0013] In a possible implementation, when obtaining the data of the partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata, the second computing node is specifically configured to: read second metadata of the data of the partition corresponding to the reduce operation from the storage node indicated by the first metadata, the second metadata being used to indicate the storage location of the data of the partition corresponding to the reduce operation in the storage node; and read the data of the partition corresponding to the reduce operation from the storage node indicated by the first metadata according to the second metadata. In this way, the second computing node can first determine the storage node where the data of each partition corresponding to the reduce operation is located according to the first metadata, so that the second computing node can access all the data belonging to the same partition according to the second metadata on the storage node, thereby ensuring that the second computing node can read all the data, and ensuring the reliability of the second computing node executing the reduce task.

[0014] In a possible implementation, when the second computing node obtains the data of the partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata, the second computing node is specifically configured to: read part of the data of the first partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata during the process in which the first computing node writes the data corresponding to each Map task to the plurality of storage nodes; and after the first computing node writes the data corresponding to all Map tasks to the plurality of storage nodes, the second computing node is further configured to read the remaining part of the data of the first partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata. In this way, the second computing node can perform the process of reading the data of the first partition in parallel during the process in which the first computing node writes the data to the plurality of storage nodes, which can realize parallel use of the uplink bandwidth resources and the downlink bandwidth resources of the plurality of storage nodes, improve the utilization of the network resources of the plurality of storage nodes, and improve the overall efficiency of the reduce processing of the second computing node on the data of the first partition.

[0015] In a possible implementation, when the second computing node obtains the data of the partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata, the second computing node is specifically configured to: read part of the data of the first partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata during the process in which the first computing node writes the data corresponding to each Map task to the plurality of storage nodes; and after the first computing node writes the data corresponding to all Map tasks to the plurality of storage nodes, the second computing node is further configured to read the remaining part of the data of the first partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata. In this way, the second computing node can perform the process of reading the data of the first partition in parallel during the process in which the first computing node writes the data to the plurality of storage nodes, which can realize parallel use of the uplink bandwidth resources and the downlink bandwidth resources of the plurality of storage nodes, improve the utilization of the network resources of the plurality of storage nodes, and improve the overall efficiency of the reduce processing of the second computing node on the data of the first partition.

[0016] In a possible implementation, the data stored in the plurality of storage nodes comprises compressed data, the compressed data being obtained by compressing the data corresponding to the plurality of Map tasks. In this way, the first computing node writes the data corresponding to each Map task into the sending buffer after compression, which not only reduces the storage space required by the data in each Map task in the sending buffer, but also reduces the amount of data sent by the first computing node to the plurality of computing nodes, reduces the transmission delay of the data, and reduces the storage space required by the data when stored in the plurality of storage nodes.

[0017] In a second aspect, the present application provides a data processing method, the method being applied to a data processing system, the data processing system comprising a plurality of computing nodes and a plurality of storage nodes, the plurality of computing nodes comprising a first computing node performing a Map task and a second computing node performing a reduce operation, each first computing node performing at least one Map task; the method comprising: determining, by the first computing node, a partition to which data corresponding to each Map task in the plurality of Map tasks belongs, and determining a storage node for storing data of each partition, and storing first metadata, the first metadata comprising partition information of the partition to which the data corresponding to each Map task belongs and information of the storage node for storing the data of each partition; writing, by the first computing node, the data corresponding to each Map task into the plurality of storage nodes; aggregating, by the plurality of storage nodes, data belonging to the same partition in the data corresponding to each Map task; during the process of writing, by the first computing node, the data corresponding to each Map task into the plurality of storage nodes, obtaining, by the second computing node, the first metadata, and obtaining, by the second computing node, data of a partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata.

[0018] In a possible implementation, the method further comprises: after the first computing node saves the first metadata corresponding to all Map tasks, instructing, by the first computing node, the second computing node to obtain the data of the partition corresponding to the reduce operation from the plurality of storage nodes during the process of writing, by the first computing node, the data corresponding to each Map task into the plurality of storage nodes.

[0019] In a possible implementation, the first computing node stores the first metadata, including: after writing the target data corresponding to a target Map task in the plurality of Map tasks into the sending buffer area in the first computing node, the first computing node stores the first metadata corresponding to the target data; after the first computing node saves the first metadata corresponding to all the Map tasks, the first computing node instructs the second computing node to obtain the data corresponding to the partition of the reduce operation from the plurality of storage nodes, including: the first computing node increases the number of the sending buffer areas in the first computing node; the first computing node writes all the remaining data corresponding to the data of the plurality of Map tasks and not sent to the plurality of storage nodes for storage into the sending buffer area, and saves the first metadata corresponding to the remaining data; and the first computing node instructs the second computing node to obtain the data corresponding to the partition of the reduce operation from the plurality of storage nodes.

[0020] In a possible implementation, the first computing node increases the number of the sending buffer areas in the first computing node, including: the first computing node estimates a first time length, the first time length being a time length for transmitting the data corresponding to the Map tasks that have been saved in the plurality of storage nodes to the second computing node; the first computing node obtains a second time length, the second time length being a remaining time length for the second computing node to save the data corresponding to all the Map tasks to the plurality of storage nodes; and when the first time length is greater than or equal to the second time length, the first computing node increases the number of the sending buffer areas in the first computing node.

[0021] In a possible implementation, the first computing node stores the first metadata, including: after writing the target data corresponding to a target Map task in the plurality of Map tasks into the sending buffer area in the first computing node, the first computing node stores the first metadata corresponding to the target data; and the first computing node instructs the second computing node to obtain the data corresponding to the partition of the reduce operation from the plurality of storage nodes, including: when the first time length is greater than or equal to the second time length, the first computing node increases the number of the sending buffer areas; the first computing node writes all the remaining data corresponding to the data of the plurality of Map tasks and not sent to the plurality of storage nodes for storage into the sending buffer area, and saves the first metadata corresponding to the remaining data; and the first computing node instructs the second computing node to obtain the data corresponding to the partition of the reduce operation from the plurality of storage nodes.

[0022] In a possible implementation, the first computing node estimates the first time length, including: the first computing node determines the number of the data corresponding to the plurality of Map tasks and already saved to the plurality of storage nodes; and the first computing node estimates the first time length according to the number and the bandwidth of the second computing node reading the data from the plurality of storage nodes.

[0023] In a possible implementation, the method further includes: the first computing node obtaining available resources of the second computing node; and the first computing node instructing the second computing node to obtain data of a partition corresponding to the reduce operation from the plurality of storage nodes, including: in a case where the available resources of the second computing node satisfy the task execution condition and the first time length is greater than or equal to the second time length, the first computing node instructing the second computing node to obtain the data of the partition corresponding to the reduce operation from the plurality of storage nodes.

[0024] In a possible implementation, the second computing node obtains the data of the partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata, including: the second computing node reading second metadata of the data of the partition corresponding to the reduce operation from the storage node indicated by the first metadata, the second metadata being used to indicate a storage location of the data of the partition corresponding to the reduce operation in the storage node; and the second computing node reading the data of the partition corresponding to the reduce operation from the storage node indicated by the first metadata according to the second metadata.

[0025] In a possible implementation, the second computing node obtains the data of the partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata, including: in a process in which the first computing node writes the data corresponding to each Map task to the plurality of storage nodes, the second computing node reads part of the data of a first partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata; and the method further includes: after the first computing node writes the data corresponding to all Map tasks to the plurality of storage nodes, the second computing node reads the remaining part of the data of the first partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata.

[0026] In a possible implementation, the second computing node obtains the data of the partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata, including: in a process in which the first computing node writes the data corresponding to each Map task to the plurality of storage nodes, the second computing node reads part of the data of a first partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata; after reading the part of the data of the first partition corresponding to the reduce operation, the second computing node reads part of the data of a second partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata; and the method further includes: after the first computing node writes the data corresponding to all Map tasks to the plurality of storage nodes, the second computing node reads the remaining part of the data of the first partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata, and reads the remaining part of the data of the second partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata.

[0027] In a possible implementation, the data stored in the plurality of storage nodes comprises compressed data, the compressed data being obtained by compressing the data corresponding to the plurality of Map tasks.

[0028] The data processing method provided in the second aspect corresponds to the data processing system provided in the first aspect, and therefore the technical effects of the second aspect and any of the implementation manners of the second aspect can be understood with reference to the descriptions of the corresponding implementation manners of the first aspect, which will not be repeated here.

[0029] In the third aspect, the present application provides a computing device, which comprises a processor and a memory. The processor and the memory are in communication with each other. The processor is configured to execute instructions stored in the memory, so that the computing device performs the data processing method performed by the first computing node in the second aspect or any of the implementation manners of the second aspect. It should be noted that the memory can be integrated into the processor or independent of the processor. The computing device can further comprise a bus. The processor is connected to the memory through the bus. The memory can comprise a readable memory and a random access memory.

[0030] In the fourth aspect, the present application provides a computer readable storage medium, which stores instructions. When the instructions are executed on a computing device, the computing device performs the operation steps of the data processing method in the second aspect or any of the implementation manners of the second aspect.

[0031] In the fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device, cause the computing device to perform the operation steps of the data processing method in the second aspect or any of the implementation manners of the second aspect.

[0032] On the basis of the implementation manners of the above aspects, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0033] FIG. 1 is a structural schematic diagram of an exemplary data processing system provided by the present application;

[0034] FIG. 2a is a flowchart of a process in which the computing node 102 performs the reduce task after all data is written into the storage cluster 200;

[0035] FIG. 2b is a flowchart of a process in which the computing node 102 performs the reduce task during the writing of data into the storage cluster 200;

[0036] FIG. 3 is a flowchart of a data processing method provided by the present application;

[0037] FIG. 4 is a flow diagram illustrating execution of a Map task and a reduce task in the data processing system 10;

[0038] FIG. 5 is a diagram illustrating overall latency reduction by serial execution of a reduce task and parallel execution of a reduce task in the computing node 101;

[0039] FIG. 6 is a diagram illustrating a hardware structure of a computing device provided in the present application. DETAILED DESCRIPTION

[0040] The terms "first", "second", and the like in the description and claims of the present application and in the above drawings are used only to distinguish similar objects and do not necessarily have a specific order or sequence. It should be understood that the terms as used in the description of the embodiments of the present application can be interchanged, as appropriate, and that the use of such terms in the description of the embodiments of the present application is merely intended to distinguish between similar objects for the purpose of description in the embodiments of the present application.

[0041] The technical solutions in the present application will be described below in conjunction with the accompanying drawings provided in the present application.

[0042] Referring to FIG. 1, a structural diagram of an exemplary data processing system 10 is shown. As shown in FIG. 1, the data processing system 10 includes a plurality of computing nodes and a plurality of storage nodes. For ease of understanding and description, FIG. 1 illustrates an example in which the data processing system 10 includes two computing nodes (computing node 101 and computing node 102) and two storage nodes (storage node 201 and storage node 202). The computing nodes 101 and 102 and the storage nodes 201 and 202 can be connected by a network, such as a wired network.

[0043] The computing node (e.g., the computing node 101) refers to a node with data computing capability, which can be implemented by a computing device including a processor, such as a server and the like. The processor may, for example, be any one of a central processing unit (CPU), an accelerator, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system on chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, a data processing unit (DPU), or any combination thereof. The accelerator may, for example, be a graphics processing unit (GPU), a neural-network processing unit (NPU), or a tensor processing unit (TPU), and the like. In actual application scenarios, multiple computing nodes in the data processing system 10 can form a computing cluster, such as the computing cluster 100 shown in FIG. 1.

[0044] A storage node (such as the storage node 201) refers to a node having a data storage capability, for example, can be implemented by a device including a storage medium, such as a storage server, etc. The storage medium may, for example, be a solid state disk (SSD), a shingled magneting recording (SMR) hard disk, a hard disk drive (HDD), a dynamic random access memory (DRAM), a dual in-line memory module, or a dual in-line memory module (DIMM), etc., or can be another type of storage medium. In an actual application scenario, multiple storage nodes in the data processing system 10 can constitute a storage cluster, such as the storage cluster 200 shown in FIG. 1.

[0045] In a big data scenario, the computing node 101 can generate raw data (taking a single computing node generating raw data as an example), and the quantity of the generated raw data is usually large, while the storage space in the computing node 101 is usually limited and difficult to store all the raw data. Therefore, the computing node 101 can first write the data into the storage cluster 200 for storage, so as to subsequently gradually process the data stored in the storage cluster 200.

[0046] In a specific implementation, when the data processing system 10 adopts an RSS (Remote Shuffle Service) architecture, a plurality of Map tasks can be generated on the computing node 101, each of which is configured to convert the original data into a key-value pair. The value in the key-value pair can be the original data itself, and the key can be the offset or line number of the original data. Then, the computing node 101 can write the data (key-value pair) contained in the plurality of Map tasks to one or more storage nodes in the storage cluster 200, each of which can also be referred to as an RSS node, and the storage cluster 200 formed by the plurality of storage nodes can be referred to as an RSS cluster. The storage cluster 200 can provide a shuffle service for splitting and aggregating the data (key-value pair) contained in the plurality of Map tasks written to the storage cluster 200 to obtain a plurality of partitions. When aggregating the received data, the storage cluster 200 specifically writes the data belonging to the same partition to the storage area corresponding to the partition, and the data in different partitions is saved in different storage areas in the storage cluster 200. The storage area corresponding to each partition can be pre-configured. When all the data in the same partition is aggregated, the metadata corresponding to the data in the partition is saved in the computing node 101, as shown in FIG. 2a, which can describe the storage location of the data in each partition in the storage node. After the storage cluster 200 successfully stores all the data, the computing node 102 can execute a plurality of reduce tasks, each of which is configured to perform a reduce operation on the data belonging to one partition stored in the storage cluster 200, such as summing or averaging the data in one partition. In the process of performing the corresponding reduce task on the data in each partition, the computing node 102 first reads the metadata saved in the computing node, so as to read the data belonging to the same partition from the storage cluster 200 according to the metadata and perform the reduce operation thereon, as shown in FIG. 2a. In this way, after the computing node 102 performs the corresponding reduce task on the data in each partition, the reduce processing on the data in all the partitions is implemented.

[0047] Since the computing node 102 performs reduce processing on the data in a granularity of partitions, and the metadata of the data in each partition stored in the computing node 101 needs to be stored in the computing node 101 after the data belonging to the same partition is aggregated by the storage cluster 200, the computing node 102 usually needs to wait for all the data to be written into the storage cluster 200 before the data belonging to each partition is aggregated, so that the computing node 102 can determine the storage locations of the data belonging to the same partition in the storage nodes by using the metadata of the data in each partition stored in the computing node 101, read the data in the partition, and perform reduce operation on the data in the partition. In this way, the computing node 102 does not read the data in the storage cluster 200 during the whole process of the computing node 101 sending the data to the storage cluster 200, which causes the uplink bandwidth resources of the storage cluster 200 to be idle for a long time, resulting in waste of network resources. Moreover, the computing node 102 needs to wait for all the data in the partitions to be stored in the storage cluster 200 before starting to perform the reduce task, which also reduces the overall efficiency of the data processing system 10 in processing the original data.

[0048] Based on this, in the data processing system 10 shown in FIG. 1, the computing node 101 determines the partition to which the data corresponding to each Map task belongs, and determines the storage node for storing the data of each partition (i.e., determines in which storage node or storage nodes the data of each partition is stored). Moreover, the computing node 101 also stores the first metadata, as shown in FIG. 2b. The first metadata includes the partition information (such as partition identifier, etc.) of the partition to which the data corresponding to each Map task belongs, and the information of the storage node for storing the data of each partition. That is, the first metadata can describe the partition to which the data corresponding to each Map task belongs, and the storage node in which the data of each partition is stored. The computing node 101 also writes the data corresponding to each Map task to the storage cluster 200. Accordingly, the storage nodes in the storage cluster 200 aggregate the data belonging to the same partition in the data corresponding to each Map task, which can specifically be writing the data belonging to the same partition in the data corresponding to different Map tasks to the storage area designated for the partition in the storage node. In actual application, the storage node stores the second metadata of the data in the multiple partitions aggregated by the storage node, and the second metadata is used to describe the storage location of the data in the storage node, as shown in FIG. 2b. Assuming that the computing node 102 is responsible for executing the reduce task, specifically, performing the reduce operation on the data belonging to the same partition, then, in the process of the computing node 101 writing the data corresponding to each Map task to the storage cluster 200, the computing node 102 can read the partition data corresponding to the reduce operation from the storage cluster 200 according to the first metadata saved by the computing node 101. In specific implementation, the computing node 101 can access the corresponding storage node in the storage cluster 200 according to the first metadata, and access the data belonging to the same partition in the corresponding storage location according to the second metadata saved by the storage node, so that the computing node 102 can subsequently perform the corresponding reduce operation on the accessed data belonging to the same partition, to complete the reduce task corresponding to the partition, as shown in FIG. 2b.

[0049] Since the first metadata of the partition level (i.e., the metadata used to describe the storage node where the data in the partition is located) is stored in the computing node 101, when the computing node 101 writes the data corresponding to the Map task to the storage cluster 200, the computing node 102 can determine the storage node where the data of the partition corresponding to the reduce operation is located according to the first metadata, and further read the data from the storage node. In this way, when the computing node 101 writes the data to the storage cluster 200, the computing node 102 can perform the process of reading the data of the partition corresponding to the reduce operation, so as to subsequently perform the reduce operation on the read data in the partition, which can realize parallel use of the uplink bandwidth resource and the downlink bandwidth resource of the storage cluster 200, thereby improving the utilization of the network resource of the storage cluster 200.

[0050] Meanwhile, since the data corresponding to the multiple Map tasks (i.e., the data in the multiple partitions) are all successfully written to the storage cluster 200, the computing node 102 has read out the part of the data required for performing the reduce operation from the storage cluster 200, without the waiting process of transmitting the part of the data from the storage cluster 200 to the computing node 102, which can improve the efficiency of the computing node 102 in obtaining all the data required for performing the reduce operation, thereby improving the overall efficiency of the computing node 102 in performing the reduce processing on the data (i.e., improving the efficiency of the computing node 102 in performing the reduce task).

[0051] It is worth noting that the above is described by taking the example of the computing node 101 writing the data to the storage cluster 200 and the computing node 102 reading the data corresponding to the reduce operation from the storage cluster 200 to perform the reduce operation, and the actual application is not limited thereto.

[0052] In addition, the data processing system 10 shown in FIG. 1 is only used as an example for illustration, and is not used for limitation. For example, in other possible embodiments, the data processing system can include a larger number of computing nodes or storage nodes, at this time, the computing node writing the data to the storage cluster 200 can be multiple computing nodes, or the computing node reading the data of the partition corresponding to the reduce operation from the storage cluster 200 can be multiple computing nodes. Or, in other possible data processing systems, other types of nodes can also be included, such as a management node which can collect the resource usage of the multiple computing nodes, so as to select the computing node with sufficient resources to perform the corresponding reduce operation on the data of each partition based on the resource usage of the multiple computing nodes.

[0053] For ease of understanding, the embodiments of the data processing method provided in the present application are described below with reference to the accompanying drawings.

[0054] Referring to FIG. 3, FIG. 3 is a flowchart of a data processing method provided in an embodiment of the present application. The method can be applied to the data processing system 10 shown in FIG. 1, or can be applied to other applicable data processing systems. For ease of illustration, an exemplary description is given in this embodiment by taking the data processing system 10 shown in FIG. 1 as an example, and taking the computing node 101 responsible for data writing and the computing node 102 responsible for performing reduce operation.

[0055] The data processing method shown in FIG. 3 can specifically include the following steps.

[0056] S301: The computing node 101 determines the partition to which the data corresponding to each Map task belongs, and determines the storage node for storing the data of each partition, and stores first metadata including the partition information of the partition to which the data corresponding to each Map task belongs and the information of the storage node for storing the data of each partition.

[0057] S302: The computing node 101 writes the data corresponding to each Map task to the plurality of storage nodes in the storage cluster 200.

[0058] In an actual application scenario, the computing node 101 can generate raw data, and the raw data generated by the computing node 101 can need to be performed reduce operation. For example, the computing node 101 can generate a plurality of transaction records based on the transaction behaviors of a plurality of users, and the data processing system 10 can calculate the sum of the transaction amounts of each user based on the transaction amounts in the transaction records. However, the storage space of the computing node 101 is limited, and it is unable to save all the raw data generated thereby. Therefore, the computing node 101 can first save the raw data to the storage cluster 200, specifically, to the storage node 201 and / or the storage node 202 in the storage cluster 200, so as to subsequently process the saved raw data step by step.

[0059] In a possible implementation, the computing node 101 can divide the original data into multiple data shards, and generate a Map task for each data shard, each of which is used to convert the original data in the data shard into key-value pairs. The value in each key-value pair is an original data, and the key can be an offset or a row number, etc. as shown in FIG. 4. Then, the computing node 101 can execute the Map tasks to convert the original data in each data shard into the corresponding key-value pairs. The computing node 101 can be configured with a send buffer, and after the execution of each Map task is completed, the computing node 101 can write the data (key-value pairs) in each Map task into the send buffer. In actual application scenarios, the number of send buffers created by the computing node 101 in the initial state can be limited to a specified number, for example, the computing node 101 can only create 2 send buffers. Therefore, the computing node 101 can first write the data included in part of the Map tasks into the send buffer. Then, the send thread on the computing node 101 can send the data in the send buffer to the storage nodes 201 and 202. The send thread can send the data included in each Map task to the storage nodes 201 and 202 based on a preset strategy, which can be a load balancing strategy or other types of strategies, and the present disclosure does not limit this. When the data in the send buffer is completely written to the storage cluster 200, the computing node 101 can continue to write the data included in other Map tasks into the send buffer, so that the send thread continues to execute the process of writing the data in the send buffer to the storage cluster 200.

[0060] Further, before writing the data included in the Map task into the send buffer, the computing node 101 can first compress the data to obtain compressed data. Then, as shown in FIG. 4, the computing node 101 writes the compressed data into the send buffer, which can reduce the storage space required by the data in each Map task in the send buffer, and also can reduce the amount of data sent by the computing node 101 to the storage cluster 200, reduce the transmission delay of data between the storage cluster 200 and each computing node, and reduce the storage space required by the data stored in the storage cluster 200.

[0061] Generally, a single Map task generated by the computing node 101 can include data in multiple partitions; or, a part of Map tasks can only include data in a single partition, and no limitation is made in this regard. In this embodiment, the computing node 101 can determine the partitions to which the data in each Map task belongs before sending the data, for example, data having the same key can belong to the same partition, or the key can be subjected to a hash operation, and data having the same hash value of the key belong to the same partition, and the number of data in different partitions can be different. Then, taking a target Map task in the multiple Map tasks as an example, after the computing node 101 writes the target data corresponding to the target Map task into the sending buffer area, the computing node 101 can generate first metadata corresponding to the target Map task. The first metadata corresponding to the Map task includes the partition information of the partition to which the target data corresponding to the target Map task belongs and the information of the storage node for storing the data in each partition. The partition information can be, for example, the identifier of the partition, such as the number of the partition. The information of the storage node can be, for example, the identifier of the storage node. In this way, the first metadata corresponding to each Map task can describe the partition to which each piece of data corresponding to the Map task belongs and describe the storage node in which the data belonging to each partition in the Map task is located (that is, describe in which storage node(s) in the storage cluster 200 the data in each partition is stored). For example, it is assumed that one of the Map tasks includes data A belonging to partition 1 and data B belonging to partition 2, and the computing node 101 writes the data A belonging to partition 1 in the Map task into the storage node 201 and writes the data B belonging to partition 2 into the storage node 202, so that the computing node 101 can generate, for the first metadata corresponding to the Map task, the partition information indicating that the data A belongs to partition 1 and the data B belongs to partition 2, and the information indicating that the data in partition 1 is located in the storage node 201 and the data in partition 2 is located in the storage node 202. Further, the first metadata can also be used to describe the number of data included in each partition (or the size of the partition).

[0062] In actual application scenarios, the computing node 101 can send the data belonging to the same partition in the Map task to a single storage node for storage. Alternatively, the computing node 101 can send the data belonging to the same partition in the Map task to different storage nodes for storage based on a load balancing strategy or other types of strategies, for example, the data in the same partition can be divided into multiple groups of data, and the multiple groups of data can be sent to multiple storage nodes for storage, and each storage node stores one group of data belonging to the partition. At this time, the first metadata of each partition can describe that the data in the partition is located in multiple storage nodes, for example, the first metadata can include the identifier of the partition and the identifier of the multiple storage nodes.

[0063] In this embodiment, after the computing node 101 writes the data in the Map task into the sending buffer, the computing node 101 can save the first metadata of the partition to which the data in the Map task belongs in the computing node 101, as shown in FIG. 4. In this case, the computing node 101 can merge the first metadata when saving the first metadata corresponding to the plurality of Map tasks. For example, it is assumed that the Map task 1 includes data belonging to the partition 1 (to be stored in the storage node 201) and data belonging to the partition 2, and the Map task 2 includes data belonging to the partition 1 (to be stored in the storage node 202) and data belonging to the partition 3. Then, the computing node 101 can collect the first metadata corresponding to the partition 1 (describing storage of the data in the storage node 201) corresponding to the data in the Map task 1, and the first metadata corresponding to the partition 1 (describing storage of the data in the storage node 202) corresponding to the data in the Map task 2. The computing node 101 can collect and save the two pieces of first metadata, so that the first metadata recorded on the computing node 101 can describe that the storage nodes in which the data in the partition 1 is located include the storage node 201 and the storage node 202.

[0064] S303: The storage cluster 200 aggregates the data belonging to the same partition in the data corresponding to each Map task.

[0065] It is assumed that the storage cluster 200 saves data by using the storage node 201 and the storage node 202. When receiving the data sent by the computing node 101, the network card in the storage node 201 and the network card in the storage node 202 can respectively buffer the data received by themselves into a receive buffer, as shown in FIG. 4. Then, the storage node 201 and the storage node 202 aggregate the buffered data respectively, and write the data in each partition obtained by aggregation into a corresponding storage area in the storage medium (such as an SSD, etc.). In the process of aggregating the data belonging to the same partition, the storage cluster 200 can determine the partition to which the data belongs according to the key of the received data, such as by performing a hash operation on the key and determining the partition to which the data belongs according to the hash value of the key. After determining the partition to which the data belongs, the storage cluster 200 can save the data into the storage area corresponding to the partition, so as to realize aggregation of the data in the same partition. Based on a similar process, the storage cluster 200 can aggregate the data in each partition.

[0066] Exemplarily, the storage node 201 and the storage node 202 can store data in each partition in a block unit. In the process of storing data in each partition, the storage node 201 and the storage node 202 can generate second metadata for data in each partition, the second metadata being used to describe storage locations of the data in the storage node 201 and / or the storage node 202.

[0067] In the process that the computing node 101 sends data to the storage cluster 200 for storage, the network cards in the storage node 201 and the storage node 202 can receive data by using the downlink ports, so that the downlink bandwidth resources of the storage node 201 and the storage node 202 are in a used state.

[0068] S304: In the process that the computing node 101 writes data corresponding to the plurality of Map tasks to the storage cluster 200, the computing node 101 acquires the first metadata and acquires data of the partition corresponding to the reduce operation from the storage cluster 200 according to the first metadata.

[0069] S305: After the data corresponding to the plurality of Map tasks is all stored to the storage cluster 200, the computing node 102 reads the remaining data of the partition corresponding to the reduce operation from the storage cluster 200.

[0070] In this embodiment, the computing node 101 can generate a plurality of reduce tasks for data of the plurality of partitions, each reduce task being used to be responsible for performing a reduce operation on data belonging to one partition, such as accumulating, multiplying, averaging, or finding maximum / minimum value of data in the same partition. Then, in the process that the computing node 101 writes data to the storage cluster 200, the computing node 101 can schedule the generated reduce task to the computing node 102, as shown in FIG. 4, and trigger the computing node 102 to execute the reduce task (it is assumed in this embodiment that the reduce task is executed by the computing node 102). In an actual application scenario, a driver process can be run in the computing node 101, the driver process can generate a plurality of reduce tasks according to the first metadata, and schedule each reduce task to the computing node 102 for execution.

[0071] In a possible implementation, the computing node 101 can determine a timing of instructing the computing node 102 to execute the reduce task according to a progress of writing data corresponding to the plurality of Map tasks to the storage cluster 200.

[0072] In a specific implementation, in the process of writing the data corresponding to the plurality of Map tasks by the computing node 101 (through the sending thread) to the storage cluster 200, the computing node 101 can determine that the computing node 102 starts to execute the reduce task after saving the first metadata corresponding to all the Map tasks.

[0073] For example, the computing node 101 can estimate a first time length for pulling the saved data corresponding to the Map tasks from the storage cluster 200 (i.e., the time length for transmitting the saved data in the storage cluster 200 from the storage cluster 200 to the computing node 102). For example, the computing node 101 can obtain a first quantity of data that has been sent by the computing node 101 to the storage cluster 200 in real time, such as by the computing node 101 notifying the computing node 101 in real time, and the computing node 101 can also obtain the uplink bandwidth supported by the storage cluster 200, so that the computing node 101 can calculate a first ratio between the first quantity and the uplink bandwidth to obtain a first time length required for transmitting the data based on the uplink bandwidth.

[0074] Further, the computing node 101 can obtain a second time length, which is a remaining time length for saving the data corresponding to all the Map tasks to the storage cluster 200, i.e., a time length required for transmitting the remaining data that has not been transmitted from the computing node 101 to the storage cluster 200. For example, the computing node 101 can calculate a second ratio between a second quantity of data included in the Map tasks that have not been written to the sending buffer (and the quantity that has not been sent in the sending buffer) and the downlink bandwidth supported by the storage cluster 200 to obtain a second time length required for transmitting the data based on the downlink bandwidth. The downlink bandwidth supported by the storage cluster 200 is usually the bandwidth for writing data from the computing node 101 to the storage cluster 200. Alternatively, the computing node 101 can count the second quantity of data that has not been sent and calculate a second ratio between the second quantity and the downlink bandwidth between the storage cluster 200 to obtain the second time length, and notify the computing node 101, etc.

[0075] Then, the computing node 101 can compare the first time length with the second time length. When the first time length is less than the second time length, it indicates that the computing node 101 has more time to write data into the storage cluster 200 (longer than the time length of pulling stored data from the storage cluster 200). In this case, the computing node 101 can wait for the computing node 101 to continue writing data into the storage cluster 200. When the first time length is greater than or equal to the second time length, it indicates that the time length of the computing node 101 to write data into the storage cluster 200 is similar to the time length of pulling stored data from the storage cluster 200. In this case, the computing node 101 can assign the reduce task to the computing node 102 to trigger the computing node 102 to execute the reduce task and start to pull data from the storage cluster 200, which belongs to the partition corresponding to the reduce task.

[0076] It should be noted that the above-mentioned computing node 101 determines the timing of instructing the computing node 102 to execute the reduce task only as an implementation example. In actual application, the computing node 101 can also determine when to instruct the computing node 102 to execute the reduce task based on other manners.

[0077] For example, the computing node 101 can monitor the number of Map tasks whose data is not written into the sending buffer area in the computing node 101, and when the number of Map tasks whose data is not written into the sending buffer area reaches a specified number, or the number of Map tasks whose data is not written into the sending buffer area is less than the number of Map tasks whose data is written into the sending buffer area, the computing node 101 can determine to start instructing the computing node 102 to execute the reduce task.

[0078] For another example, during the process that the computing node 101 writes data corresponding to multiple Map tasks into the storage cluster 200, the computing node 102 can also start to launch a reduce task, which can be assigned to the computing node 102 by the computing node 101. Thus, in the case that the computing node 101 does not store complete first metadata, the computing node 102 can also obtain the saved first metadata from the computing node 101, and read data of the partition corresponding to the execution of the reduce operation from the storage cluster 200 according to the currently obtained first metadata. In this way, during the process that the computing node 101 writes data into the storage cluster 200, the computing node 102 can simultaneously start the process of reading data from the storage cluster 200, so as to further improve the utilization rate of network bandwidth resources of the storage cluster 200.

[0079] It can be understood that when it is determined that the first duration is greater than or equal to the second duration, the data in the computing node 101 can still exist in the data in the Map task that is not written into the sending buffer area. Based on this, the computing node 101 can increase the number of sending buffer areas in the computing node 101 when it is determined that the first duration is greater than or equal to the second duration, such as increasing the corresponding number of sending buffer areas according to the number of Map tasks whose data is not written into the sending buffer area. In this way, the computing node 101 can cache the remaining data in the increased sending buffer area after increasing the sending buffer area, and the remaining data is the data in the Map task whose data is not written into the sending buffer area, that is, the data in the plurality of partitions that is not written into the sending buffer area. Moreover, after successfully writing the remaining data into the newly added sending buffer area, the first metadata corresponding to the remaining data is saved to the computing node 101. Wherein, the first metadata corresponding to the remaining data includes the partition information of the partition to which the remaining data belongs and the information of the storage node for storing the remaining data. Thus, the computing node 101 instructs the computing node 102 to start executing the reduce task in the case that the first metadata corresponding to the data in all Map tasks is saved. Wherein, the first metadata corresponding to the data in all Map tasks includes the first metadata corresponding to the remaining data and the first metadata that has been saved before the computing node 101 increases the number of sending buffer areas. In this way, the computing node 101 can save the complete first metadata (that is, the first metadata corresponding to the data in all Map tasks) by increasing the number of sending buffer areas, which enables the computing node 102 to determine the complete information of the storage node in the storage cluster 200 where the data in the same partition is located based on the complete first metadata when starting to execute the reduce task, so as to read the data required for executing the reduce task from the storage node.

[0080] Exemplarily, the computing node 101 can be configured with a driver module 1011 and a scheduling management module 1012. Wherein, the driver module 1011 can be responsible for estimating the first duration and the second duration, and notifying the scheduling management module 1012 after determining that the first duration is greater than or equal to the second duration. The scheduling management module 1012 can be responsible for instructing the computing node 101 to increase the number of sending buffer areas, as shown in FIG. 4. Thus, the driver module 1011 instructs the computing node 102 to start executing the reduce task in the case that the first metadata corresponding to all data in the plurality of partitions has been saved.

[0081] The driving module 1011 and the scheduling management module 1012 can be implemented by software or hardware. When implemented by software, the driving module 1011 and the scheduling management module 1012 can be a thread or hardware in the computing node 101. When implemented by hardware, the driving module 1011 and the scheduling management module 1012 can be an ASIC or other hardware in the computing node 101.

[0082] In this embodiment, the execution of the reduce task by the computing node 102 is taken as an example for description. In actual application, multiple computing nodes included in the computing cluster 100 can execute multiple reduce tasks, and different computing nodes can be responsible for executing different reduce tasks.

[0083] As an implementation example, the computing node 101 can obtain available resources of each computing node in the computing cluster 100. The available resources of the computing node can include, for example, the number of available processors in the computing node, the size of the available storage area in the memory / hard disk, and the size of the available network bandwidth, etc. Then, the computing node 101 can determine one or more computing nodes whose available resources satisfy the task execution condition when the first time length is greater than or equal to the second time length. The determined computing node is the computing node that executes the reduce task and pulls data from the storage cluster 200. The available resources satisfying the task execution condition means that the available resources of the computing node can support the resource consumption required by the computing node to execute the reduce task. For ease of description, the execution of the reduce task by the computing node 102 is taken as an example for description. In actual application, the reduce task can also be executed by the computing node 101 or other computing nodes in the computing cluster 100.

[0084] In other implementation manners, the data processing system 10 can further include a management node (not shown in FIG. 1). The management node can collect the available resources of each computing node, and determine one or more computing nodes (available resources satisfying the task execution condition) that can execute the reduce task according to the available resources of each computing node. Then, the management node can provide the identifier of the one or more computing nodes that can execute the reduce task to the computing node 101, so that the computing node 101 can select at least one computing node from the one or more computing nodes to execute the reduce task according to the received identifier. For example, the computing node 101 can randomly select some computing nodes to execute the reduce task, or can select the computing node closest to the storage cluster 200 to execute the reduce task, etc., which is not limited.

[0085] After determining that the computing node 102 is the computing node to execute the reduce task and determining the timing to start executing the reduce task, the computing node 101 can assign the reduce task to the computing node 102 to instruct the computing node 102 to start reading the data stored in the storage cluster 200 that is required to execute the reduce task. In the embodiment, it is assumed that the reduce task assigned to the computing node 102 is to process the data in the partition 1, which is one of the plurality of partitions. The process of the computing node 102 reading the data from the storage cluster 200 is described as follows.

[0086] In a possible implementation, the computing node 102 can send an access request to the computing node 101, which can carry the identification of the partition 1, so that the computing node 101 can query the locally stored first metadata based on the access request and feed back the first metadata about the partition 1 to the computing node 102, as shown in FIG. 4.

[0087] Then, the computing node 102 can access the one or more storage nodes in the storage cluster 200 indicated by the first metadata about the partition 1 according to the first metadata about the partition, to obtain the second metadata stored in the storage nodes and belonging to the data in the partition 1, so that the computing node 102 can access the data belonging to the partition 1 stored in the storage nodes according to the second metadata. When the data in the partition 1 is stored in a plurality of storage nodes, the computing node 102 can access each of the plurality of storage nodes to read the data belonging to the partition 1 in each of the plurality of storage nodes. In the process of pulling the data from the storage cluster 200, the computing node 102 can first pull the data to the receiving buffer area of the computing node 102, as shown in FIG. 4, so as to subsequently perform the corresponding reduce operation on the data in the receiving buffer area by the computing node 102. In this way, before the data belonging to the partition 1 is completely written into the storage cluster 200, the computing node 102 can start reading the data belonging to the partition 1 from the storage cluster 200 according to the first metadata and the second metadata.

[0088] It is worth noting that, since the data in the computing node 101 is not stored in the storage cluster 200, the data can include the data belonging to the partition 1 and other partitions, and thus the data belonging to the partition 1 currently accessed by the computing node 102 from the storage cluster 200 is not all the data in the partition 1. However, the computing node 102 can perform the process of reading the data from the storage cluster 200 in parallel with the process of writing the data into the storage cluster 200 by the computing node 101, which makes the uplink bandwidth resources and the downlink bandwidth resources of the storage cluster 200 be used, so as to alleviate the problem of waste of network resources of the storage cluster 200.

[0089] In actual application, the computing node 102 can execute multiple reduce tasks generated by the computing node 101. In the process that the computing node 101 writes data into the storage cluster 200, the computing node 102 can read only the data required for executing one reduce task, or can read the data required for executing multiple reduce tasks respectively.

[0090] In the first implementation, the computing node 102 can execute the reduce tasks in series. Taking the case that the instructing node 202 schedules n reduce tasks to the computing node 102 as an example, n is an integer greater than 1. When the computing node 102 executes the first reduce task, the computing node 102 can read the data belonging to partition 1 required for executing the first reduce task in the process that the computing node 101 writes data into the storage cluster 200. Since the computing node 102 reads data, all the data belonging to partition 1 can not have been written into the storage cluster 200. Therefore, the data read by the computing node 102 is part of the data belonging to partition 1 that has been saved in the storage cluster 200 at the first time. The first time can be the time when the computing node 102 accesses the storage cluster 200, or can be any time before accessing the storage cluster 200, which is not limited. In the process that the computing node 102 accesses the storage cluster 200, the uplink bandwidth resource and the downlink bandwidth resource of the storage cluster 200 are used, so as to alleviate the problem of waste of network resources of the storage cluster 200.

[0091] Then, the computing node 102 can wait for the computing node 101 to write all the data in multiple partitions into the storage cluster 200, and after the storage cluster 200 successfully saves all the data in multiple partitions, the computing node 102 can continue to read the remaining part of the data belonging to partition 1 that has not been saved before the first time from the storage cluster 200. In this way, the computing node 102 can execute the corresponding reduce operation according to all the data belonging to partition 1 read, so as to complete the first reduce task. In this way, in the whole process of executing the first reduce task, the computing node 102 can obtain part of the data from the storage cluster 200 in advance, which can reduce the overall time delay of the computing node 102 to obtain data, thereby improving the overall efficiency of the computing node 102 to execute the first reduce task. As shown in FIG. 5, the overall time delay of the computing node 102 to execute the first reduce task can be reduced by T0.

[0092] After the first reduce task is completed, the computing node 102 can continue to execute the second reduce task, the nth reduce task in turn. At this time, the computing node 102 saves the time length T0 of executing the first reduce task, that is, the total time length of saving of executing the plurality of reduce tasks. Since the data in the plurality of partitions has been written into the storage cluster 200, the computing node 100 can directly read all the data belonging to the same partition required by the remaining reduce tasks from the storage cluster 200 when executing the remaining reduce tasks.

[0093] In an actual application scenario, when the data processing system 10 is applied to the RSS architecture, since the computing node in the RSS architecture usually executes the reduce task in series, by adding a plug-in to the computing node in the RSS architecture, the computing node can read the data required for executing the reduce task from the storage cluster 200 in advance, so that the code in the computing node in the RSS architecture can be modified without invasion, and the implementation difficulty of the scheme is reduced.

[0094] In the second implementation manner, the computing node 102 can execute the plurality of reduce tasks in parallel. Taking the case that the computing node 102 can execute 2 reduce tasks in parallel as an example. Assuming that the computing node 101 schedules n reduce tasks to the computing node 102, and n is an integer greater than 2. Then, when the computing node 102 executes the first reduce task, the computing node 102 can read the part of the data belonging to the partition 1 (that is, the data required for executing the first reduce task) saved in the storage cluster 200 at the first time in the process that the computing node 101 writes the data into the storage cluster 200. The first time can be, for example, the time when the computing node 102 accesses the storage cluster 200 for the first reduce task, or can be any time before the time.

[0095] Then, the computing node 102 can pause the execution of the first reduce task, start the execution of the second reduce task, and read the partial data belonging to the partition 2 that has been saved by the storage cluster 200 at the second time (i.e., the data required for the execution of the second reduce task) in the time period between the first time and the second time. The second time may, for example, be the time at which the computing node 102 accesses the storage cluster 200 for the second reduce task, or may, for example, be any time before the time. In this way, the computing node 102 can read partial data in advance from the storage cluster 200 for multiple reduce tasks in the process of accessing the storage cluster 200, which makes the uplink bandwidth resources and the downlink bandwidth resources of the storage cluster 200 both be used, so as to alleviate the problem of waste of network resources of the storage cluster 200.

[0096] Suppose that the computing node 102 has read all the data belonging to the partition 2 that has been saved by the storage cluster 200 before the second time when the storage cluster 200 has stored all the data in multiple partitions. Then, the computing node 102 can continue to read the remaining partial data belonging to the partition 1 for the first reduce task, i.e., read the data belonging to the partition 1 that has not been saved by the storage cluster 200 before the first time. Then, after reading all the data required for the execution of the first reduce task (i.e., all the data in the partition 1), the computing node 102 can continue to read the remaining partial data belonging to the partition 2 for the second reduce task, i.e., read the data belonging to the partition 2 that has not been saved by the storage cluster 200 before the second time.

[0097] In actual application, when the number of reduce tasks executed in parallel by the computing node 102 exceeds two, the computing node 102 can read the data belonging to the partition 3 for the third reduce task in the above manner, and so on, which will not be described herein.

[0098] After all the data in the multiple partitions is written into the storage cluster 200, for the reduce task that has read part of the data in advance, the computing node 102 can continue to read the remaining data from the storage cluster 200 for the reduce task to meet the requirement for data when the reduce task is executed. In this way, during the whole process of executing the reduce task, the computing node 102 can obtain part of the data from the storage cluster 200 in advance, which can reduce the overall time delay of the computing node 102 obtaining data, thereby improving the overall efficiency of the computing node 102 executing the reduce task. For the reduce task that has not read part of the data in advance, the computing node 102 can directly read all the data required for executing the reduce task from the storage cluster 200 when the reduce task is executed. As shown in FIG. 5, the overall time delay of the computing node 102 executing the first reduce task and the second reduce task can be reduced by a time length T1. At this time, the time length T1 saved by the computing node 102 is the total time length saved for executing the multiple reduce tasks.

[0099] Further, during the process of executing the reduce task, if the data read from the storage cluster 200 is compressed data (obtained by the computing node 101 compressing the data in the Map task), the computing node 102 can first decompress the read compressed data, and then execute the corresponding reduce operation based on the compressed data.

[0100] It should be noted that, in the embodiment, the computing node 101 writes the data in the multiple partitions to the multiple storage nodes, and the computing node 102 pulls the data from the multiple storage nodes, which is used for illustration only and is not used for limitation.

[0101] For example, in other embodiments, the computing node 101 can perform the operation of pulling data from the multiple storage nodes in parallel during the process of writing data to the multiple storage nodes.

[0102] For another example, in other embodiments, the multiple computing nodes in the data processing system 10 can all write data into the storage cluster 200, and each computing node can write different data in the same partition into the storage cluster 200.

[0103] For another example, in other embodiments, the computing node 101 can instruct the multiple computing nodes in the data processing system 10 to execute reduce tasks at the same time, and different computing nodes execute different reduce tasks.

[0104] It is worth noting that other reasonable combinations of steps that can be thought of by those skilled in the art based on the above description are also within the scope of protection of the present application. Secondly, those skilled in the art should also be familiar with the fact that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present application.

[0105] The data processing system and the data processing method provided by the embodiments of the present application are introduced above in combination with FIG. 1 to FIG. 5. Next, the structure of the computing device provided by the embodiments of the present application is introduced in combination with the accompanying drawings.

[0106] FIG. 6 is a schematic diagram of the hardware structure of a computing device 600 provided by the present application, which can implement the computing node 101 or the computing node 102 in the embodiment shown in FIG. 3, for example.

[0107] As shown in FIG. 6, the computing device 600 includes a processor 601, a memory 602, and a communication interface 603. The processor 601, the memory 602, and the communication interface 603 communicate through a bus 604, and can also communicate through wireless transmission or other means. The memory 602 is configured to store instructions, and the processor 601 is configured to execute the instructions stored in the memory 602. Further, the computing device 600 can also include a memory unit 605, which can also be connected to the processor 601, the storage medium 602, and the communication interface 603 through the bus 604. The memory 602 stores program codes, and the processor 601 can execute the operations of the computing node 101 in the embodiment shown in FIG. 3 or the operations of the computing node 102 in the embodiment shown in FIG. 3 by using the program codes stored in the memory 602.

[0108] It should be understood that in the present embodiment, the processor 601 can be a CPU, and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, or the like. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0109] The memory 602 can include read-only memory and random access memory, and provide instructions and data for the processor 601. The memory 602 can also include non-volatile random access memory.

[0110] The memory 602 can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. The nonvolatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), or a flash memory, among others. The volatile memory can be a random access memory (RAM), which is used as the external cache. By way of example, and not limitation, many forms of RAM are available, for example, static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0111] The communication interface 603 is configured to communicate with other devices connected to the computing device 600. The bus 604 can include, in addition to the data bus, a power bus, a control bus, and a state signal bus, among others. However, for the sake of clarity, all of the buses are labeled as the bus 604 in the figure.

[0112] It should be understood that the computing device 600 of the embodiments of the present application can be used to perform the method performed by the computing node 101 in the method shown in FIG. 3, and to perform the method performed by the computing node 102 in the method shown in FIG. 3, and thus the above and other operations and / or functions implemented by the computing device 600 are respectively for implementing the flow of the corresponding method in FIG. 3, and for the sake of brevity, will not be described here.

[0113] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium or a data center containing one or more available media that the computing device can store. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), among others. The computer readable storage medium includes instructions that instruct the computing device to perform the above data processing method.

[0114] The embodiments of the present application also provide a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in the embodiments of the present application are generated.

[0115] The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website site, computer or data center to another website site, computer or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.).

[0116] The computer program product can be a software installation package, which can be downloaded and executed on a computing device when any method of the foregoing data processing method is needed.

[0117] The above embodiments can be implemented all or partially by software, hardware, firmware or any combination thereof. When implemented by software, the above embodiments can be implemented all or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.).

[0118] The terminology used in the above embodiments is for the purpose of describing specific embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, “one or more” refers to one, two, or more; the character “ / ” generally indicates that the preceding and following objects are in an “or” relationship. In the embodiments of this application, “simultaneously” means within the same time period, including situations where they are at the same moment.

[0119] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0120] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing system, characterized by The data processing system comprises a plurality of computing nodes and a plurality of storage nodes, wherein the plurality of computing nodes comprises a first computing node for executing a Map task and a second computing node for executing a reduce operation, and each first computing node executes at least one Map task; The first computing node is configured to determine a partition to which data corresponding to each Map task belongs, determine a storage node for storing data of each partition, and store first metadata, wherein the first metadata comprises partition information of the partition to which the data corresponding to each Map task belongs and information of the storage node for storing data of each partition; The data corresponding to each Map task is written to the plurality of storage nodes; The plurality of storage nodes are configured to aggregate data belonging to the same partition in the data corresponding to each Map task; The second computing node is configured to acquire the first metadata during the process in which the first computing node writes the data corresponding to each Map task to the plurality of storage nodes, and acquire data of a partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata.

2. The system of claim 1, wherein, The first computing node is further configured to: After the first computing node saves the first metadata corresponding to all Map tasks, instruct the second computing node to acquire data of a partition corresponding to the reduce operation from the plurality of storage nodes during the process in which the first computing node writes the data corresponding to each Map task to the plurality of storage nodes.

3. The system of claim 2, wherein, When storing the first metadata, the first computing node is specifically configured to: After writing target data corresponding to a target Map task in the plurality of Map tasks to a sending buffer area in the first computing node, store first metadata corresponding to the target data; When the first computing node saves the first metadata corresponding to all Map tasks, the first computing node is specifically configured to: Increase the number of sending buffer areas in the first computing node; Write all remaining data in the data corresponding to the plurality of Map tasks that have not been sent to the plurality of storage nodes to the sending buffer areas, and save first metadata corresponding to the remaining data; Instruct the second computing node to acquire data of a partition corresponding to the reduce operation from the plurality of storage nodes.

4. The system of claim 3, wherein, When increasing the number of sending buffer areas in the first computing node, the first computing node is specifically configured to: Estimate a first time length, wherein the first time length is a time length for transmitting data corresponding to Map tasks that have been saved in the plurality of storage nodes to the second computing node; Acquire a second time length, wherein the second time length is a remaining time length for saving data corresponding to all Map tasks to the plurality of storage nodes by the second computing node; When the first time length is greater than or equal to the second time length, increase the number of sending buffer areas in the first computing node.

5. The system of claim 4, wherein, The first computing node is specifically configured to: determine the quantity of data saved to the plurality of storage nodes from the data corresponding to the plurality of Map tasks; estimate the first time length according to the quantity and the bandwidth of the second computing node reading data from the plurality of storage nodes.

6. The system of claim 4 or 5, wherein, The first computing node is further configured to: obtain available resources of the second computing node; The first computing node is specifically configured to: instruct the second computing node to obtain the data corresponding to the partition of the reduce operation from the plurality of storage nodes, in a case that the available resources of the second computing node satisfy a task execution condition and the first time length is greater than or equal to the second time length.

7. The system according to any one of claims 1 to 6, characterized in that, The second computing node is specifically configured to: read, from the storage nodes indicated by the first metadata, second metadata of the data corresponding to the partition of the reduce operation, the second metadata being used to indicate storage locations of the data corresponding to the partition of the reduce operation in the storage nodes; read, according to the second metadata, the data corresponding to the partition of the reduce operation from the storage nodes indicated by the first metadata.

8. The system of any one of claims 1 to 7, wherein, The second computing node is specifically configured to: read, according to the first metadata, part of the data corresponding to the first partition of the reduce operation from the plurality of storage nodes in a process that the first computing node writes the data corresponding to each Map task to the plurality of storage nodes; The second computing node is further configured to: read, according to the first metadata, the remaining part of the data corresponding to the first partition of the reduce operation from the plurality of storage nodes after the first computing node writes the data corresponding to all Map tasks to the plurality of storage nodes.

9. The system according to any one of claims 1 to 7, characterized in that, The second computing node is specifically configured to: read, according to the first metadata, part of the data corresponding to the first partition of the reduce operation from the plurality of storage nodes in a process that the first computing node writes the data corresponding to each Map task to the plurality of storage nodes; read, according to the first metadata, part of the data corresponding to the second partition of the reduce operation from the plurality of storage nodes after reading the part of the data corresponding to the first partition of the reduce operation; The second computing node is further configured to: reading, from the plurality of storage nodes, remaining part data of a first partition corresponding to the reduce operation according to the first metadata, and reading, from the plurality of storage nodes, remaining part data of a second partition corresponding to the reduce operation according to the first metadata after the first computing node writes data corresponding to all Map tasks to the plurality of storage nodes.

10. The system of any one of claims 1 to 9, wherein, The data stored in the plurality of storage nodes comprises compressed data obtained by compressing data corresponding to the plurality of Map tasks.

11. A data processing method, characterized by, The method is applied to a data processing system comprising a plurality of computing nodes and a plurality of storage nodes, wherein the plurality of computing nodes comprises a first computing node for executing a Map task and a second computing node for executing a reduce operation, and each first computing node executes at least one Map task. The method comprises: The first computing node determines a partition to which data corresponding to each Map task in the plurality of Map tasks belongs, and determines a storage node for storing data of each partition, and stores first metadata comprising partition information of the partition to which the data corresponding to each Map task belongs and information of the storage node for storing data of each partition; The first computing node writes the data corresponding to each Map task to the plurality of storage nodes; The plurality of storage nodes aggregate data belonging to the same partition in the data corresponding to each Map task; In the process that the first computing node writes the data corresponding to each Map task to the plurality of storage nodes, the second computing node acquires the first metadata, and acquires, from the plurality of storage nodes, data of the partition corresponding to the reduce operation according to the first metadata.

12. The method of claim 11, wherein, The method further comprises: In the process that the first computing node writes the data corresponding to each Map task to the plurality of storage nodes, after the first computing node saves the first metadata corresponding to all Map tasks, the first computing node instructs the second computing node to acquire, from the plurality of storage nodes, data of the partition corresponding to the reduce operation.

13. The method of claim 12, wherein, The first metadata comprises: The first computing node stores first metadata corresponding to target data corresponding to a target Map task in the plurality of Map tasks after the first computing node writes the target data to a sending buffer area in the first computing node; The first computing node instructs the second computing node to acquire, from the plurality of storage nodes, data of the partition corresponding to the reduce operation after the first computing node saves the first metadata corresponding to all Map tasks, comprises: The first computing node increases the number of sending buffer areas in the first computing node; The first computing node writes all remaining data in the data corresponding to the plurality of Map tasks which are not sent to the plurality of storage nodes for storage to the sending buffer area, and saves first metadata corresponding to the remaining data; The first computing node instructs the second computing node to obtain data of a partition corresponding to the reduce operation from the plurality of storage nodes.

14. The method of claim 13, wherein, The first computing node increases the number of sending buffer areas in the first computing node, including: The first computing node estimates a first time length, the first time length being a time length for data corresponding to Map tasks that have been saved in the plurality of storage nodes to be transmitted to the second computing node; The first computing node obtains a second time length, the second time length being a remaining time length for the second computing node to save data corresponding to all Map tasks to the plurality of storage nodes; When the first time length is greater than or equal to the second time length, the first computing node increases the number of sending buffer areas in the first computing node.

15. The method of claim 14, wherein, The first computing node estimates a first time length, including: The first computing node determines a number of data corresponding to the plurality of Map tasks that have been saved to the plurality of storage nodes; The first computing node estimates the first time length according to the number and a bandwidth at which the second computing node reads data from the plurality of storage nodes.

16. The method according to claim 14 or 15, characterized in that, The method further includes: The first computing node obtains available resources of the second computing node; The first computing node instructs the second computing node to obtain data of a partition corresponding to the reduce operation from the plurality of storage nodes, including: In a case where the available resources of the second computing node satisfy a task execution condition and the first time length is greater than or equal to the second time length, the first computing node instructs the second computing node to obtain data of a partition corresponding to the reduce operation from the plurality of storage nodes.

17. The method according to any one of claims 11 to 16, characterized in that, The second computing node obtains data of a partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata, including: The second computing node reads second metadata of data of a partition corresponding to the reduce operation from the storage node indicated by the first metadata, the second metadata being used to indicate a storage location of the data of the partition corresponding to the reduce operation in the storage node; The second computing node reads data of a partition corresponding to the reduce operation from the storage node indicated by the first metadata according to the second metadata.

18. The method according to any one of claims 11 to 17, characterized in that, The second computing node obtains data of a partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata, including: In a process in which the first computing node writes data corresponding to each Map task to the plurality of storage nodes, the second computing node reads part of data of a first partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata; The method further includes: After the first computing node writes data corresponding to all Map tasks to the plurality of storage nodes, the second computing node reads remaining part of data of the first partition corresponding to the reduce operation from the plurality of storage nodes according to the first metadata.

19. The method according to any one of claims 11 to 17, characterized in that, The second computing node obtains, according to the first metadata, data of the partition corresponding to the reduce operation from the plurality of storage nodes, comprising: In the process that the first computing node writes the data corresponding to each Map task to the plurality of storage nodes, the second computing node reads, according to the first metadata, part of the data of the first partition corresponding to the reduce operation from the plurality of storage nodes; After reading the part of the data of the first partition corresponding to the reduce operation, the second computing node reads, according to the first metadata, part of the data of the second partition corresponding to the reduce operation from the plurality of storage nodes; The method further comprises: After the first computing node writes the data corresponding to all Map tasks to the plurality of storage nodes, the second computing node reads, according to the first metadata, the remaining part of the data of the first partition corresponding to the reduce operation from the plurality of storage nodes, and reads the remaining part of the data of the second partition corresponding to the reduce operation from the plurality of storage nodes.

20. The method according to any one of claims 11 to 19, characterized in that, The data stored in the plurality of storage nodes comprises compressed data, and the compressed data is obtained by compressing the data corresponding to the plurality of Map tasks.

21. A computer-readable storage medium, characterized in that, comprising instructions that, when executed on at least one computing device, cause the at least one computing device to perform the steps of the method of any one of claims 11 to 20.

22. A computer program product comprising instructions, wherein: comprising instructions that, when executed on at least one computing device, cause the at least one computing device to perform the steps of the method of any one of claims 11 to 20.

Images