Data processing system and method and related equipment

Abstract

The invention discloses a data processing system and method and related equipment, and relates to the technical field of data processing. The data processing system comprises a first computing node, a second computing node and a plurality of storage nodes. The first computing node is used for determining a partition to which data corresponding to each Map (mapping) task belongs, storing a storage node of the data of each partition, storing first metadata and writing the data corresponding to each Map task into a plurality of storage nodes, the first metadata comprises partition information of partitions to which the data belong and information of storage nodes for storing the data of each partition. And the plurality of storage nodes are used for aggregating the data belonging to the same partition. And the second computing node is used for acquiring data of the partition corresponding to the protocol operation from the plurality of storage nodes according to the acquired first metadata in the process that the first computing node writes the data corresponding to each Map task into the plurality of storage nodes. Therefore, the utilization rate of the network resources of the plurality of storage nodes can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing system, method and related equipment. Background Art

[0002] Remote shuffle service (RSS) is a distributed computing architecture design that aims to use a separate storage node to provide shuffle service, to separate the shuffle operation from the computing task, that is, to decouple computing and storage. The shuffle operation refers to partitioning and sorting data in the storage node during the computing process, and providing computing nodes with the ability to reduce the shuffled data.

[0003] Currently, under the RSS architecture, computing nodes usually send data to storage nodes for partitioning and storage first, and after all data is saved in the storage node, the computing node performs reduction operations on the data in the same partition. However, this way of processing data is not only inefficient, but also seriously wastes network resources of storage nodes. Summary of the invention

[0004] The present application provides a data processing system to improve the efficiency of data processing and alleviate the problem of network resource waste. In addition, the present application also provides a corresponding data processing method, a computing device, a computer-readable storage medium, and a computer program product.

[0005] In the first aspect, the present application provides a data processing system, including multiple computing nodes and multiple storage nodes. Exemplarily, the computing node may be a node with computing capability such as a server, and the storage node may be a node with storage capability such as a storage server. The multiple computing nodes include a first computing node that executes a Map task and a second computing node that executes a specification reduce operation. Among them, the first computing node refers to one or more computing nodes, each of which executes at least one Map task. Similarly, the second computing node may also be one or more computing nodes. In addition, the first computing node and the second computing node may be the same computing node or different computing nodes, which is not limited. The first computing node is used to determine the partition to which the data corresponding to each Map task in the multiple Map tasks belongs. In the actual application scenario, the data corresponding to each Map task may include data belonging to multiple partitions. In addition, the first computing node is also used to determine the storage node for storing the data of each partition, and store the first metadata, which includes the partition information of the partition to which the data corresponding to each Map task belongs (which can be used to describe which partition the data belongs to) and the information of the storage node for storing the data of each partition (which can be used to describe which storage node the data of the partition is stored in). In addition, the first computing node is also used to write the data corresponding to each Map task to multiple storage nodes. Multiple storage nodes are used to aggregate the data belonging to the same partition in the data corresponding to each Map task, wherein the data belonging to the same partition can be written to the storage area corresponding to the partition in the storage node after the aggregation is completed. In actual applications, when storing the data corresponding to the Map task, the storage node will also store the second metadata, and the second metadata is used to describe the storage location of the data in the storage node. The second computing node is used to obtain the first metadata during the process of the first computing node writing the data corresponding to each Map task to the multiple storage nodes, and obtain the data of the partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata.

[0006] Because the first computing node generates the first metadata that records the partition information corresponding to the data of the Map task and the information of the storage node where the partition is located before storing the data of the Map task to the storage node (i.e., before executing shufflewrite), this allows the second computing node that executes the Reduce operation to obtain the first metadata during the process of the first computing node writing data to multiple storage nodes (i.e., the process of executing shufflewrite), and read the partition data that has been saved and corresponds to the reduce operation from the corresponding storage node according to the first metadata (i.e., execute the shuffle read operation), so that the second computing node performs the corresponding reduce operation on the read partition data. In this way, during the process of the first computing node writing data to multiple storage nodes, the second computing node that executes the reduce operation can execute the process of reading the partition data in parallel, that is, the second computing node can start the reduce operation before the data of the Map task is completely written to the storage node, which not only improves the efficiency of data processing, but also can utilize the uplink bandwidth resources and downlink bandwidth resources of multiple storage nodes in parallel, and improve the utilization rate of the network resources of the multiple storage nodes.

[0007] In a possible implementation, the first computing node is also used to instruct the second computing node to obtain the data of the partition corresponding to the reduce operation from the multiple storage nodes after the first computing node saves the first metadata corresponding to all the Map tasks during the process of the first computing node writing the data corresponding to each Map task to the multiple storage nodes. In this way, the first computing node can store the first metadata corresponding to the data corresponding to all the Map tasks, that is, it can save the complete first metadata. In this way, when the second computing node executes the reduce task, it can access the data of the corresponding partition based on the complete first metadata, thereby improving the reliability of the second computing node in executing the reduce task.

[0008] In a possible implementation, when the first computing node stores the first metadata, taking the first metadata corresponding to the target Map task in multiple Map tasks as an example, the first computing node specifically stores the first metadata corresponding to the target data after writing the target data corresponding to the target Map task in multiple Map tasks into the sending buffer in the first computing node; then, after the first computing node saves the first metadata corresponding to all Map tasks, when the first computing node instructs the second computing node to obtain the data of the partition corresponding to the reduce operation from multiple storage nodes, the number of sending buffers in the first computing node can be increased, and the remaining data in the data corresponding to the multiple Map tasks that have not been sent to the storage in the multiple storage nodes are all written into the sending buffer, and the first metadata corresponding to the remaining data is saved, so that the first computing node instructs the second computing node to obtain the data of the partition corresponding to the reduce operation from the multiple storage nodes. In this way, the first computing node can complete the process of writing the data corresponding to all Map tasks into the sending buffer by increasing the number of sending buffers, so that the first computing node can store the first metadata corresponding to the data corresponding to all Map tasks, that is, it can save the complete first metadata. At this time, the first computing node triggers the second computing node to execute the reduce task, so that the second computing node can access the data of the corresponding partition based on the complete first metadata, thereby ensuring the reliability of the second computing node in executing the reduce task.

[0009] In a possible implementation, when the first computing node increases the number of sending buffers in the first computing node, it can specifically estimate the first duration, which is the duration for the data corresponding to the Map tasks that have been saved in multiple storage nodes to be transmitted to the second computing node. In addition, the first computing node is also used to obtain the second duration, which is the remaining duration for the second computing node to save the data corresponding to all Map tasks to multiple storage nodes. Thus, the first computing node is also used to increase the number of sending buffers in the first computing node when the first duration is greater than or equal to the second duration. In this way, the first computing node can trigger the increase in the number of sending buffers by estimating the duration for the remaining data to be transmitted to multiple storage nodes for storage and the duration for the second computing node to read the saved data from the storage node, so that when the data corresponding to all Map tasks are cached in the sending buffer, the second computing node is automatically triggered to execute the reduce task in advance, thereby not only improving the overall efficiency of the second computing node in reducing data, but also making the best use of the uplink bandwidth resources and downlink bandwidth resources of multiple storage nodes.

[0010] In one possible implementation, when estimating the first duration, the first computing node is specifically used to: determine the amount of data corresponding to multiple Map tasks that has been saved to multiple storage nodes; and estimate the first duration based on the amount and the bandwidth of the first computing node writing data to multiple storage nodes.

[0011] In a possible implementation, the first computing node is also used to: obtain the available resources of the second computing node; then, when the first computing node instructs the second computing node to obtain the data of the partition corresponding to the reduction operation from multiple storage nodes, it is specifically used to: when the available resources of the second computing node meet the task execution conditions and the first duration is greater than or equal to the second duration, instruct the second computing node to obtain the data of the partition corresponding to the reduction operation from multiple storage nodes. In this way, instructing the second computing node to start executing the reduce task in combination with the available resources of the second computing node can realize the execution of the reduce task when the available resources of the second computing node are sufficient, thereby avoiding the impact of insufficient resources of the second computing node on the execution of the reduce task, and improving the rationality and reliability of task scheduling.

[0012] In a possible implementation, when the second computing node obtains the data of the partition corresponding to the reduce operation from multiple storage nodes according to the first metadata, it is specifically used to: read the second metadata of the data of the partition corresponding to the reduce operation from the storage node indicated by the first metadata, the second metadata is used to indicate the storage location of the data of the partition corresponding to the reduce operation in the storage node; and read the data of the partition corresponding to the reduce operation from the storage node indicated by the first metadata according to the second metadata. In this way, the second computing node can first determine the storage node where the data of each partition corresponding to the reduce operation is located according to the first metadata, so that the second computing node can access all the data belonging to the same partition according to the second metadata on the storage node, thereby ensuring that the second computing node can read all the data and ensuring the reliability of the second computing node in executing the reduce task.

[0013] In a possible implementation, when the second computing node obtains the data of the partition corresponding to the reduce operation from multiple storage nodes according to the first metadata, it is specifically used to: read part of the data of the first partition corresponding to the reduce operation from multiple storage nodes according to the first metadata during the process of the first computing node writing the data corresponding to each Map task to multiple storage nodes; then, the second computing node is also used to: read the remaining part of the data of the first partition corresponding to the reduce operation from multiple storage nodes according to the first metadata after the first computing node writes the data corresponding to all Map tasks to multiple storage nodes. In this way, during the process of the first computing node writing data to multiple storage nodes, the second computing node can execute the process of reading the data of the first partition in parallel, which can realize the parallel use of the uplink bandwidth resources and downlink bandwidth resources of multiple storage nodes, improve the utilization rate of the network resources of the multiple storage nodes, and also improve the overall efficiency of the second computing node in reducing the data of the first partition.

[0014] In a possible implementation, when the second computing node obtains the data of the partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata, it is specifically used to: read part of the data of the first partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata during the process of the first computing node writing the data corresponding to each Map task to the multiple storage nodes; and after reading part of the data of the first partition corresponding to the reduce operation, read part of the data of the second partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata; then, the second computing node is also used to: read the remaining part of the data of the first partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata after the first computing node writes the data corresponding to all Map tasks to the multiple storage nodes, and read the remaining part of the data of the second partition corresponding to the reduce operation from the multiple storage nodes. In this way, during the process of the first computing node writing data to the multiple storage nodes, the second computing node can execute the process of reading the data of the multiple partitions in parallel, which can realize the parallel utilization of the uplink bandwidth resources and downlink bandwidth resources of the multiple storage nodes, improve the utilization rate of the network resources of the multiple storage nodes, and also improve the overall efficiency of the second computing node in performing reduce processing on the data of the multiple partitions.

[0015] In a possible implementation, the data stored in the multiple storage nodes include compressed data, which is obtained by compressing the data corresponding to the multiple Map tasks. In this way, the first computing node compresses the data corresponding to each Map task and then writes it into the sending buffer area, which can not only reduce the storage space required for the data in each Map task in the sending buffer area, but also reduce the amount of data sent by the first computing node to the multiple computing nodes, reduce the data transmission delay, and reduce the storage space required for the data to be stored in multiple storage nodes.

[0016] In a second aspect, the present application provides a data processing method, which is applied to a data processing system, wherein the data processing system includes multiple computing nodes and multiple storage nodes, wherein the multiple computing nodes include a first computing node that executes a mapping Map task and a second computing node that executes a reduction operation, and each first computing node executes at least one Map task; the method includes: the first computing node determines the partition to which the data corresponding to each Map task in the multiple Map tasks belongs, and determines the storage node used to store the data of each partition, and stores first metadata, the first metadata including partition information of the partition to which the data corresponding to each Map task belongs and information of the storage node used to store the data of each partition; the first computing node writes the data corresponding to each Map task to multiple storage nodes; the multiple storage nodes aggregate the data belonging to the same partition in the data corresponding to each Map task; in the process of the first computing node writing the data corresponding to each Map task to multiple storage nodes, the second computing node obtains the first metadata, and obtains the data of the partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata.

[0017] In a possible implementation, the method further includes: in the process of the first computing node writing the data corresponding to each Map task to multiple storage nodes, after the first computing node saves the first metadata corresponding to all Map tasks, the first computing node instructs the second computing node to obtain the data of the partition corresponding to the reduce operation from multiple storage nodes.

[0018] In one possible implementation, storing the first metadata includes: after the first computing node writes the target data corresponding to the target Map task among multiple Map tasks into a sending buffer in the first computing node, the first computing node stores the first metadata corresponding to the target data; after the first computing node saves the first metadata corresponding to all Map tasks, the first computing node instructs the second computing node to obtain the data of the partition corresponding to the reduce operation from multiple storage nodes, including: the first computing node increases the number of sending buffers in the first computing node; the first computing node writes all the remaining data corresponding to the multiple Map tasks that have not been sent to the multiple storage nodes for storage into the sending buffer, and saves the first metadata corresponding to the remaining data; the first computing node instructs the second computing node to obtain the data of the partition corresponding to the reduce operation from the multiple storage nodes.

[0019] In one possible implementation, the first computing node increases the number of sending buffer areas in the first computing node, including: the first computing node estimates a first duration, the first duration being the duration for transmitting data corresponding to Map tasks already saved in multiple storage nodes to the second computing node; the first computing node obtains a second duration, the second duration being the remaining duration for the second computing node to save data corresponding to all Map tasks to multiple storage nodes; when the first duration is greater than or equal to the second duration, the first computing node increases the number of sending buffer areas in the first computing node.

[0020] In a possible implementation, the first computing node stores first metadata, including: after writing target data corresponding to a target Map task among multiple Map tasks into a sending buffer in the first computing node, the first computing node stores the first metadata corresponding to the target data; the first computing node instructs the second computing node to obtain data of partitions corresponding to reduce operations from multiple storage nodes, including: when the first duration is greater than or equal to the second duration, the first computing node increases the number of sending buffers; the first computing node writes all remaining data among the data corresponding to multiple Map tasks that have not been sent to multiple storage nodes for storage into the sending buffer, and saves the first metadata corresponding to the remaining data; the first computing node instructs the second computing node to obtain data of partitions corresponding to reduce operations from multiple storage nodes.

[0021] In one possible implementation, the first computing node estimates the first duration, including: the first computing node determines the amount of data corresponding to multiple Map tasks that has been saved to multiple storage nodes; the first computing node estimates the first duration based on the amount and the bandwidth of the second computing node reading data from the multiple storage nodes.

[0022] In a possible implementation, the method also includes: the first computing node obtains the available resources of the second computing node; the first computing node instructs the second computing node to obtain the data of the partition corresponding to the reduce operation from multiple storage nodes, including: when the available resources of the second computing node meet the task execution conditions and the first duration is greater than or equal to the second duration, the first computing node instructs the second computing node to obtain the data of the partition corresponding to the reduce operation from multiple storage nodes.

[0023] In a possible implementation, the second computing node obtains data of the partition corresponding to the reduce operation from multiple storage nodes based on the first metadata, including: the second computing node reads second metadata of the data of the partition corresponding to the reduce operation from the storage node indicated by the first metadata, the second metadata being used to indicate the storage location of the data of the partition corresponding to the reduce operation in the storage node; the second computing node reads the data of the partition corresponding to the reduce operation from the storage node indicated by the first metadata based on the second metadata.

[0024] In a possible implementation, the second computing node obtains data of a partition corresponding to a reduce operation from multiple storage nodes based on the first metadata, including: in the process of the first computing node writing data corresponding to each Map task to multiple storage nodes, the second computing node reads part of the data of the first partition corresponding to the reduce operation from multiple storage nodes based on the first metadata; the method also includes: after the first computing node writes data corresponding to all Map tasks to multiple storage nodes, the second computing node reads the remaining part of the data of the first partition corresponding to the reduce operation from multiple storage nodes based on the first metadata.

[0025] In a possible implementation, the second computing node obtains data of a partition corresponding to a reduce operation from multiple storage nodes according to the first metadata, including: in the process of the first computing node writing data corresponding to each Map task to multiple storage nodes, the second computing node reads part of the data of the first partition corresponding to the reduce operation from multiple storage nodes according to the first metadata; after reading part of the data of the first partition corresponding to the reduce operation, the second computing node reads part of the data of the second partition corresponding to the reduce operation from multiple storage nodes according to the first metadata; the method also includes: after the first computing node writes data corresponding to all Map tasks to multiple storage nodes, the second computing node reads the remaining part of the data of the first partition corresponding to the reduce operation from multiple storage nodes according to the first metadata, and reads the remaining part of the data of the second partition corresponding to the reduce operation from multiple storage nodes.

[0026] In a possible implementation, the data stored in the multiple storage nodes includes compressed data, and the compressed data is obtained by compressing data corresponding to the multiple Map tasks.

[0027] The data processing method provided in the second aspect corresponds to the data processing system provided in the first aspect. Therefore, the technical effects of the second aspect and any implementation method of the second aspect can be found in the relevant description of the technical effects of the corresponding implementation method in the above-mentioned first aspect, and will not be repeated here.

[0028] In a third aspect, the present application provides a computing device, which includes a processor and a memory. The processor and the memory communicate with each other. The processor is used to execute instructions stored in the memory so that the computing device executes a data processing method executed by the first computing node in the second aspect or any implementation of the second aspect. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The computing device may also include a bus. The processor is connected to the memory via a bus. The memory may include a readable memory and a random access memory.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computing device, the computing device executes the operating steps of the data processing method described in the second aspect or any one of the implementations of the second aspect.

[0030] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device, enables the computing device to execute the operating steps of the data processing method described in the second aspect or any one of the implementations of the second aspect.

[0031] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of the structure of an exemplary data processing system provided by the present application;

[0033] Figure 2a A flowchart of the computing node 102 executing the reduce task after all data is written into the storage cluster 200;

[0034] Figure 2b It is a flowchart of the computing node 102 executing the reduce task in the process of writing data into the storage cluster 200;

[0035] Figure 3A flowchart of a data processing method provided in this application;

[0036] Figure 4 A flowchart of executing a Map task and a Reduce task in a data processing system 10;

[0037] Figure 5 A schematic diagram of the overall latency that can be reduced by executing reduce tasks serially and in parallel by computing node 101;

[0038] Figure 6 A schematic diagram of the hardware structure of a computing device provided for this application. DETAILED DESCRIPTION

[0039] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is just a way of distinguishing objects with the same attributes when describing the embodiments of this application.

[0040] The technical solution in this application will be described below in conjunction with the drawings provided in this application.

[0041] See also Figure 1 , which is a schematic diagram of the structure of an exemplary data processing system 10. Figure 1 As shown, the data processing system 10 includes multiple computing nodes and multiple storage nodes. For ease of understanding and description, Figure 1 In the example, two computing nodes (computing node 101 and computing node 102) and two storage nodes (storage node 201 and storage node 202) are used for description. The computing node 101, the computing node 102, the storage node 201 and the storage node 202 may be connected via a network, such as a wired network.

[0042] Among them, a computing node (such as computing node 101) refers to a node with data computing capabilities, which can be implemented by a computing device including a processor, such as a server. Among them, the processor, for example, can be a central processing unit (CPU), an accelerator, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system on chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, a data processing unit (DPU), and any other processor or any combination thereof. Among them, an accelerator, for example, can be a graphics processing unit (GPU), a neural network processor (NPU), or a tensor processing unit (TPU). In actual application scenarios, multiple computing nodes in the data processing system 10 can form a computing cluster, such as Figure 1 A computing cluster 100 is shown.

[0043] A storage node (such as storage node 201) refers to a node with data storage capabilities, and can be implemented, for example, by a device including a storage medium, such as a storage server, etc. The storage medium can be, for example, a solid state disk (SSD), a shingled magnetic recording (SMR) hard disk, a hard disk drive (HDD), a dynamic random access memory (DRAM), a dual in-line memory module, or a dual in-line memory module (DIMM), etc., or can be other types of storage media. In actual application scenarios, multiple storage nodes in the data processing system 10 can form a storage cluster, such as Figure 1 Storage cluster 200 is shown.

[0044] In a big data scenario, the computing node 101 can generate raw data (taking a single computing node generating raw data as an example), and the amount of raw data generated is usually large, while the storage space in the computing node 101 is usually limited, making it difficult to store all the raw data. Therefore, the computing node 101 can first write the data into the storage cluster 200 for storage, so that the data stored in the storage cluster 200 can be processed step by step later.

[0045] In specific implementation, when the data processing system 10 adopts the RSS (remote shuffle service) architecture, multiple map tasks can be generated on the computing node 101, and each map task is used to convert the original data into a key-value pair. Among them, the value in the key-value pair can be the original data itself, and the key can be the offset or line number of the original data. Then, the computing node 101 can write the data (key-value pairs) contained in the multiple map tasks into one or more storage nodes in the storage cluster 200, each storage node can also be called an RSS node, and the storage cluster 200 composed of the multiple storage nodes can be called an RSS cluster. The storage cluster 200 can provide a shuffle service, which is used to implement the segmentation and aggregation of the data (key-value pairs) contained in the multiple map tasks written into the storage cluster 200 to obtain multiple partitions. When the storage cluster 200 aggregates the received data, specifically, the data belonging to the same partition is written into the storage area corresponding to the partition, and the data in different partitions will be stored in different storage areas in the storage cluster 200, and the storage area corresponding to each partition can be pre-configured. After all data in the same partition is aggregated, the metadata corresponding to the data in the partition will be saved in the computing node 101, such as Figure 2a As shown, the metadata can describe the storage location of the data in each partition in the storage node. After the storage cluster 200 successfully stores all the data, the computing node 102 can execute multiple reduce tasks, each of which is used to perform a reduce operation on the data in a partition stored in the storage cluster 200, such as summing or averaging the data in a partition. In the process of executing the corresponding reduce task for the data in each partition, the computing node 102 will first read the metadata stored in the computing node, so as to read the data in the same partition from the storage cluster 200 according to the metadata and perform a reduce operation on it, such as Figure 2a In this way, after the computing node 102 executes the corresponding reduce task for the data in each partition, the reduce processing for the data in all partitions can be realized.

[0046] Since the computing node 102 performs a reduce process on the data based on the partition granularity, and the metadata of the data in each partition stored by the computing node 101 needs to wait for the storage cluster 200 to aggregate the data belonging to the same partition before it can be stored in the computing node 101, this makes the computing node 102 usually need to wait for all the data to be written into the storage cluster 200 before the data belonging to each partition can be aggregated, so that the computing node 102 can use the metadata of the data in each partition stored in the computing node 101 to determine the storage location of all the data belonging to the same partition in the storage node, read the data in the partition, and perform a reduce operation on the data in the partition. In this way, during the entire process of the computing node 101 sending data to the storage cluster 200, the computing node 102 will not read the data in the storage cluster 200, which makes the uplink bandwidth resources of the storage cluster 200 idle for a long time, thereby generating a waste of network resources. In addition, the computing node 102 can only start to execute the reduce task after waiting for the data in all partitions to be saved in the storage cluster 200, which will also reduce the overall efficiency of the data processing system 10 in processing the original data.

[0047] Based on this, Figure 1 In the data processing system 10 shown, the computing node 101 determines the partition to which the data corresponding to each Map task in the plurality of Map tasks belongs, and determines the storage node for storing the data of each partition (that is, determines which storage node or nodes the data of each partition is stored in). In addition, the computing node 101 also stores the first metadata, such as Figure 2bAs shown. Among them, the first metadata includes the partition information (such as partition identifier, etc.) of the partition to which the data corresponding to each Map task belongs and the information of the storage node used to store the data of each partition. That is, the first metadata can describe the partition to which the data corresponding to each Map task belongs and the storage node where the data of each partition is located. The computing node 101 will also write the data corresponding to each Map task to the storage cluster 200. Correspondingly, the storage nodes in the storage cluster 200 will aggregate the data belonging to the same partition in the data corresponding to each Map task, and specifically, the data belonging to the same partition in the data corresponding to different Map tasks may be written to the storage area specified by the partition in the storage node. In actual application, the storage node will store the second metadata of the data while storing the data in multiple partitions obtained by aggregation. The second metadata is used to describe the storage location of the data in the storage node, such as Figure 2b As shown. Assuming that computing node 102 is responsible for executing the reduce task, specifically, performing the reduce operation on the data belonging to the same partition, then, in the process of computing node 101 writing the data corresponding to each Map task into storage cluster 200, computing node 102 can read the partition data corresponding to the reduce operation from storage cluster 200 according to the first metadata saved by computing node 101. In specific implementation, computing node 101 can access the corresponding storage node in storage cluster 200 according to the first metadata, and access the data belonging to the same partition in the corresponding storage location according to the second metadata saved in the storage node, so that computing node 102 can subsequently perform the corresponding reduce operation on the accessed data belonging to the same partition to complete the reduce task corresponding to the partition, as shown in FIG. Figure 2b shown.

[0048] Since the first metadata at the partition level (i.e., metadata describing the storage node where the data in the partition is located) is saved in the computing node 101, when the computing node 101 writes the data corresponding to each Map task to the storage cluster 200, the computing node 102 can determine the storage node where the data of the partition corresponding to the reduce operation is located based on the first metadata, and further read the data from the storage node. In this way, when the computing node 101 writes data to the storage cluster 200, the computing node 102 can execute the process of reading the data of the partition corresponding to the reduce operation, so as to subsequently execute the reduce operation on the data in the partition read, which can realize the parallel use of the uplink bandwidth resources and downlink bandwidth resources of the storage cluster 200, thereby improving the utilization rate of the network resources of the storage cluster 200.

[0049] At the same time, since the data corresponding to multiple Map tasks (that is, the data in multiple partitions) have all been successfully written into the storage cluster 200, the computing node 102 has already read part of the data required to perform the reduce operation from the storage cluster 200, and there is no need to execute the waiting process of transmitting the part of the data from the storage cluster 200 to the computing node 102, this can improve the efficiency of the computing node 102 in obtaining all the data required to perform the reduce operation, thereby improving the overall efficiency of the computing node 102 in reducing the data (that is, improving the efficiency of the computing node 102 in executing the reduce task).

[0050] It is worth noting that the above is explained by taking the example of computing node 101 writing data to storage cluster 200 and computing node 102 reading data corresponding to the reduce operation from storage cluster 200 to execute the reduce operation. In actual application, computing node 101 may also read data corresponding to the reduce operation from storage cluster 200, and there is no limitation on this.

[0051] In addition, the above Figure 1 The data processing system 10 shown is only used as an exemplary illustration and is not intended to be limiting. For example, in other possible implementations, the data processing system may include a greater number of computing nodes or storage nodes. In this case, the computing nodes that write data to the storage cluster 200 may be multiple computing nodes, or the computing nodes that read the data of the partition corresponding to the reduce operation from the storage cluster 200 may be multiple computing nodes. Alternatively, in other possible data processing systems, other types of nodes may also be included, such as a management node that can collect resource usage of multiple computing nodes, so as to select a computing node with sufficient resources based on the resource usage of multiple computing nodes to perform corresponding reduce operations on the data of each partition.

[0052] For ease of understanding, an embodiment of the data processing method provided by the present application is described below in conjunction with the accompanying drawings.

[0053] See also Figure 3 , Figure 3 A flowchart of a data processing method provided in an embodiment of the present application, which can be applied to Figure 1 The data processing system 10 described above may be applied to other applicable data processing systems. Figure 1 The data processing system 10 shown is used as an example for illustrative description, in which the computing node 101 is responsible for data writing and the computing node 102 is responsible for executing the reduce operation.

[0054] in, Figure 3The data processing method shown may specifically include:

[0055] S301: The computing node 101 determines the partition to which the data corresponding to each Map task among multiple Map tasks belongs, and determines the storage node used to store the data of each partition, and stores first metadata, which includes partition information of the partition to which the data corresponding to each Map task belongs and information of the storage node used to store the data of each partition.

[0056] S302 : The computing node 101 writes the data corresponding to each Map task to multiple storage nodes in the storage cluster 200 .

[0057] In actual application scenarios, computing node 101 can generate raw data, and the raw data generated by computing node 101 may need to be subjected to a reduce operation. For example, computing node 101 can generate multiple transaction records based on the transaction behaviors of multiple users, and data processing system 10 can calculate the sum of the transaction amounts of each user based on the transaction amounts in the transaction records. However, the storage space of computing node 101 is limited and cannot save all the raw data generated by it. Therefore, computing node 101 can first save the raw data to storage cluster 200, specifically, the raw data can be saved to storage node 201 and / or storage node 202 in storage cluster 200, so that the saved raw data can be processed step by step later.

[0058] In a possible implementation, the computing node 101 may divide the original data into multiple data slices, and generate a Map task for each data slice, each Map task is used to convert the original data in a data slice into a key-value pair. The value in each key-value pair is an original data, and the key can be an offset or a row number, such as Figure 4As shown. Then, the computing node 101 can execute each Map task to convert the original data in each data slice into a corresponding key-value pair. A send buffer can be configured in the computing node 101, and after the computing node 101 completes each Map task, the data (key-value pair) in each Map task can be written into the send buffer. In actual application scenarios, the number of send buffers created by the computing node 101 in the initial state can be limited to a specified number, such as the computing node 101 can only create 2 send buffers. Therefore, the computing node 101 can first write the data included in some Map tasks into the send buffer. Then, the sending thread on the computing node 101 can send the data in the send buffer to the storage node 201 and the storage node 202. Among them, the sending thread can send the data included in each Map task to the storage node 201 and the storage node 202 based on a preset strategy. The strategy can be, for example, a load balancing strategy, or other types of strategies, which are not limited. After all the data in the sending buffer is written into the storage cluster 200 , the computing node 101 may continue to write data included in other Map tasks into the sending buffer, so that the sending thread continues to execute the process of writing the data in the sending buffer into the storage cluster 200 .

[0059] Furthermore, before writing the data included in the Map task into the sending buffer, the computing node 101 may first compress the data to obtain compressed data. Figure 4 As shown, the computing node 101 writes the compressed data into the sending buffer, thereby reducing the storage space required for the data in each Map task in the sending buffer, and also reducing the amount of data sent by the computing node 101 to the storage cluster 200, thereby reducing the data transmission delay between the storage cluster 200 and each computing node, and reducing the storage space required for data to be stored in the storage cluster 200.

[0060] Generally, a single Map task generated by the computing node 101 may include data in multiple partitions; or, some Map tasks may only include data in one partition, which is not limited. In this embodiment, before sending data, the computing node 101 may determine the partition to which the data in each Map task belongs, such as data with the same key may belong to the same partition, or a hash operation may be performed on the key, and data with the same hash value of the key may belong to the same partition, etc. The number of data in different partitions may be different. Then, taking the target Map task among multiple Map tasks as an example, the computing node 101 may write the target data corresponding to the target Map task into the sending buffer area, and then generate the first metadata corresponding to the target Map task. The first metadata corresponding to the Map task includes the partition information of the partition to which the target data corresponding to the target Map task belongs and the information of the storage node used to store the data of each partition. Among them, the partition information may be, for example, the identifier of the partition, such as the number of the partition. The information of the storage node may be, for example, the identifier of the storage node. In this way, the first metadata corresponding to each Map task can describe the partition to which each piece of data corresponding to the Map task belongs, and describe the storage node where the data belonging to each partition in the Map task is located (that is, describe in which storage node / s in the storage cluster 200 the data in each partition is stored). For example, suppose that one of the Map tasks includes data A belonging to partition 1 and data B belonging to partition 2, and the computing node 101 writes the data A belonging to partition 1 in the Map task to the storage node 201, and writes the data B belonging to partition 2 to the storage node 202, so that the computing node 101 can target the first metadata corresponding to the Map task, and the first metadata includes partition information indicating that data A belongs to partition 1 and data B belongs to partition 2, and information indicating that the data of partition 1 is located in the storage node 201 and the data of partition 2 is located in the storage node 202. Furthermore, the first metadata can also be used to describe the number of data included in each partition (or the size of the partition), etc.

[0061] In actual application scenarios, computing node 101 can send data belonging to the same partition in a Map task to a single storage node for storage. Alternatively, computing node 101 can send data belonging to the same partition in a Map task to different storage nodes for storage based on a load balancing strategy or other types of strategies. For example, data in the same partition can be divided into multiple groups of data, and the multiple groups of data are sent to multiple storage nodes for storage, and each storage node stores a group of data belonging to the partition. At this time, the first metadata of each partition can describe that the data in the partition is located in multiple storage nodes. For example, the first metadata can include the identifier of the partition and the identifiers of multiple storage nodes.

[0062] In this embodiment, after the computing node 101 writes the data in the Map task into the sending buffer area, the first metadata of the partition to which the data in the Map task belongs can be saved in the computing node 101, such as Figure 4 As shown. Among them, when the computing node 101 saves the first metadata corresponding to multiple Map tasks respectively, the first metadata can be merged. For example, assume that Map task 1 includes data belonging to partition 1 (to be stored in storage node 201) and data belonging to partition 2, and Map task 2 includes data belonging to partition 1 (to be stored in storage node 202) and data belonging to partition 3. Then, the computing node 101 can summarize the first metadata corresponding to partition 1 corresponding to the data in Map task 1 (describing that the data is stored in storage node 201), and the first metadata corresponding to partition 1 corresponding to the data in Map task 2 (describing that the data is stored in storage node 202). The computing node 101 can summarize and save these two copies of the first metadata, so that the first metadata recorded on the computing node 101 can describe that the storage nodes where the data in partition 1 is located include storage node 201 and storage node 202.

[0063] S303: The storage cluster 200 aggregates the data belonging to the same partition in the data corresponding to each Map task.

[0064] Assuming that the storage cluster 200 uses the storage node 201 and the storage node 202 to store data, when the storage cluster 200 receives data sent by the computing node 101, the network card in the storage node 201 and the network card in the storage node 202 can cache the received data in the receive buffer respectively, such as Figure 4 As shown. Then, storage node 201 and storage node 202 respectively aggregate the cached data, and write the aggregated data in each partition into the corresponding storage area in the storage medium (such as SSD, etc.). Among them, in the process of aggregating data belonging to the same partition, the storage cluster 200 will determine the partition to which the data belongs based on the key of the received data, such as by performing a hash operation on the key, and determining the partition to which the data belongs based on the hash value of the key. After determining the partition to which the data belongs, the storage cluster 200 can save the data in the storage area corresponding to the partition, thereby realizing the aggregation of data in the same partition. Based on a similar process, the storage cluster 200 can aggregate the data of each partition.

[0065] Exemplarily, the storage node 201 and the storage node 202 may store the data in each partition in units of blocks. In the process of storing the data in each partition, the storage node 201 and the storage node 202 may generate second metadata for the data in each partition, and the second metadata is used to describe the storage location of the data in the partition in the storage node 201 and / or the storage node 202.

[0066] When computing node 101 sends data to storage cluster 200 for storage, the network cards in storage nodes 201 and 202 can receive data using downstream ports, so that downstream bandwidth resources of storage nodes 201 and 202 are in use.

[0067] S304: When the computing node 101 writes the data corresponding to the multiple Map tasks into the storage cluster 200, the computing node 101 obtains the first metadata, and obtains the data of the partition corresponding to the reduce operation from the storage cluster 200 according to the first metadata.

[0068] S305 : After all the data corresponding to the multiple Map tasks are saved in the storage cluster 200 , the computing node 102 reads the remaining data of the partition corresponding to the reduce operation from the storage cluster 200 .

[0069] In this embodiment, the computing node 101 can generate multiple reduce tasks for the data in multiple partitions, and each reduce task is responsible for performing a reduce operation on the data in one partition, such as accumulating, multiplying, averaging, or finding the maximum / minimum value of the data in the same partition. Then, when the computing node 101 writes data to the storage cluster 200, the computing node 101 can schedule the generated reduce tasks to the computing node 102, such as Figure 4 As shown, the computing node 102 is triggered to execute the reduce task (in this embodiment, it is assumed that the computing node 102 executes the reduce task). In an actual application scenario, a driver process may be running in the computing node 101, and the driver process may generate multiple reduce tasks according to the first metadata, and schedule each reduce task to be executed on the computing node 102.

[0070] In a possible implementation, computing node 101 may determine a timing to instruct computing node 102 to execute a reduce task based on the progress of data corresponding to multiple Map tasks being written into storage cluster 200 .

[0071] In specific implementation, when computing node 101 writes data corresponding to multiple Map tasks to storage cluster 200 (via a sending thread), computing node 101 may determine to instruct computing node 102 to start executing a reduce task after saving the first metadata corresponding to all Map tasks.

[0072] For example, computing node 101 may estimate the first duration of pulling the data corresponding to the saved Map task from storage cluster 200 (that is, the duration of transmitting the data saved in storage cluster 200 from storage cluster 200 to computing node 102). For example, computing node 101 may obtain in real time the first quantity of data that computing node 101 has sent to storage cluster 200, such as the computing node 101 may notify computing node 101 of the first quantity in real time, and computing node 101 may also obtain the uplink bandwidth supported by storage cluster 200, so that computing node 101 may calculate the first ratio between the first quantity and the uplink bandwidth, and obtain the first duration required for transmitting data based on the uplink bandwidth.

[0073] Furthermore, the computing node 101 may obtain a second duration, which refers to the remaining duration for saving the data corresponding to all the Map tasks to the storage cluster 200, that is, the duration required for transmitting the remaining untransmitted data in the data corresponding to all the Map tasks from the computing node 101 to the storage cluster 200. Exemplarily, the computing node 101 may calculate a second ratio between the second number of data included in the Map tasks that have not been written into the sending buffer (and the number of data that have not been sent in the sending buffer) and the downstream bandwidth supported by the storage cluster 200, and obtain the second duration required for transmitting the data based on the downstream bandwidth. The downstream bandwidth supported by the storage cluster 200 is generally the bandwidth for the computing node 101 to write data to the storage cluster 200. Alternatively, the computing node 101 may count the second number of data that is not currently sent, and calculate the second ratio between the second number and the downstream bandwidth between the storage cluster 200, to obtain the second duration, and notify the computing node 101, etc. of the second duration.

[0074] Next, computing node 101 may compare the first duration with the second duration. When the first duration is less than the second duration, it indicates that the remaining time for computing node 101 to write data to storage cluster 200 is longer (longer than the duration for pulling the stored data from storage cluster 200). At this time, computing node 101 may wait for computing node 101 to continue writing data to storage cluster 200. When the first duration is greater than or equal to the second duration, it indicates that the remaining time for computing node 101 to write data to storage cluster 200 is similar to the duration for pulling the stored data from storage cluster 200. At this time, computing node 101 may assign a reduce task to computing node 102 to trigger computing node 102 to execute the reduce task and start pulling the stored data from storage cluster 200 that belongs to the partition corresponding to the reduce task.

[0075] It should be noted that the above-mentioned computing node 101 determines the timing of instructing computing node 102 to execute the reduce task only as an implementation example. In actual application, computing node 101 can also determine when to instruct computing node 102 to execute the reduce task based on other methods.

[0076] For example, computing node 101 can monitor the number of Map tasks whose data has not been written to the sending buffer in computing node 101, and when the number of Map tasks whose data has not been written to the sending buffer reaches a specified number, or the number of Map tasks whose data has not been written to the sending buffer is less than the number of Map tasks whose data has been written to the sending buffer, computing node 101 can determine to start instructing computing node 102 to execute a reduce task.

[0077] For another example, while computing node 101 is writing data corresponding to multiple Map tasks to storage cluster 200, computing node 102 may also start a reduce task, which may be assigned to computing node 102 by computing node 101. Thus, in the case where computing node 101 does not store complete first metadata, computing node 102 may also obtain the saved first metadata from computing node 101, and read the data of the partition corresponding to the reduce operation from storage cluster 200 according to the currently obtained first metadata. In this way, while computing node 101 is writing data to storage cluster 200, computing node 102 may simultaneously start the process of reading data from storage cluster 200, thereby further improving the utilization rate of network bandwidth resources of storage cluster 200.

[0078] It can be understood that when it is determined that the first duration is greater than or equal to the second duration, the data in the computing node 101 may still have data in some Map tasks that have not been written into the sending buffer. Based on this, when the computing node 101 determines that the first duration is greater than or equal to the second duration, the number of sending buffers in the computing node 101 can be increased, such as the corresponding number of sending buffers can be increased according to the number of Map tasks whose data has not been written into the sending buffer. In this way, after the computing node 101 increases the sending buffer, it can use the increased sending buffer to cache the remaining data, which is the data in the Map task whose data has not been written into the sending buffer, that is, the data in the data in multiple partitions that has not been written into the sending buffer. In addition, after the computing node 101 successfully writes the remaining data into the newly added sending buffer, the first metadata corresponding to the remaining data is saved to the computing node 101. Among them, the first metadata corresponding to the remaining data includes the partition information of the partition to which the remaining data belongs and the information of the storage node used to store the remaining data. Therefore, the computing node 101 instructs the computing node 102 to start executing the reduce task when the first metadata corresponding to the data in all the Map tasks is saved. The first metadata corresponding to the data in all Map tasks includes the first metadata corresponding to the remaining data and the first metadata that has been saved before the computing node 101 increases the number of sending buffers. In this way, the computing node 101 can save the complete first metadata (that is, the first metadata corresponding to the data in all Map tasks) by increasing the number of sending buffers, which enables the subsequent computing node 102 to determine the complete information of the storage node where the data in the same partition is located in the storage cluster 200 based on the complete first metadata when starting to execute the reduce task, so that the data required to execute the reduce task can be read from the storage node.

[0079] Exemplarily, the computing node 101 may be configured with a driver module 1011 and a scheduling management module 1012. The driver module 1011 may be responsible for estimating the first duration and the second duration, and after determining that the first duration is greater than or equal to the second duration, notify the scheduling management module 1012. The scheduling management module 1012 may be responsible for instructing the computing node 101 to increase the number of sending buffers, such as Figure 4 Thus, when determining that the first metadata corresponding to all the data in the multiple partitions has been saved, the driver module 1011 instructs the computing node 102 to start executing the reduce task.

[0080] The driver module 1011 and the scheduling management module 1012 can be implemented by software or hardware. When implemented by software, the driver module 1011 and the scheduling management module 1012 can be threads or hardware in the computing node 101. When implemented by hardware, the driver module 1011 and the scheduling management module 1012 can be hardware such as ASIC in the computing node 101.

[0081] In this embodiment, the computing node 102 executing the reduce task is taken as an example for explanation. In actual application, multiple computing nodes included in the computing cluster 100 may execute multiple reduce tasks, and different computing nodes are responsible for executing different reduce tasks.

[0082] As an implementation example, computing node 101 can obtain the available resources of each computing node in computing cluster 100. Among them, the available resources of the computing node, for example, may include the number of available processors in the computing node, the size of the available storage area in the memory / hard disk, and the size of the available network bandwidth, etc. Then, when the first duration is greater than or equal to the second duration, computing node 101 determines one or more computing nodes whose available resources meet the task execution conditions, and the determined computing nodes are computing nodes that subsequently execute the reduce task and pull data from the storage cluster 200. Among them, the available resources meet the task execution conditions, which means that the available resources on the computing node can support the resource consumption required for the computing node to execute the reduce task. For ease of description, the following continues to be explained by taking the example of computing node 101 determining to use computing node 102 to execute the reduce task. In actual application scenarios, the reduce task can also be executed by computing node 101 or other computing nodes in computing cluster 100.

[0083] In other implementations, the data processing system 10 may also include a management node ( Figure 1 (not shown), the management node can collect the available resources of each computing node, and determine one or more computing nodes that can execute the reduce task based on the available resources of each computing node (the available resources meet the task execution conditions). Then, the management node can provide the identifiers of the one or more computing nodes that can execute the reduce task to the computing node 101, so that the computing node 101 can select at least one computing node from one or more computing nodes to execute the reduce task based on the received identifier, such as randomly selecting some computing nodes to execute the reduce task, or selecting the computing node with the closest communication distance to the storage cluster 200 to execute the reduce task, etc., which is not limited to this.

[0084] After determining that the computing node that executes the reduce task is computing node 102 and determining the timing to start executing the reduce task, computing node 101 can assign the reduce task to computing node 102 to instruct computing node 102 to start reading the data required for executing the reduce task that has been stored in storage cluster 200. In this embodiment, it is assumed that the reduce task assigned to computing node 102 is used to process data in partition 1, and partition 1 is one of the multiple partitions. The process of computing node 102 reading data from storage cluster 200 is introduced below.

[0085] In a possible implementation, computing node 102 may send an access request to computing node 101, and the access request may carry an identifier of partition 1, so that computing node 101 may query the first metadata stored locally based on the access request, and feed back the first metadata about partition 1 to computing node 102, such as Figure 4 shown.

[0086] Then, the computing node 102 can access one or more storage nodes indicated by the first metadata in the storage cluster 200 according to the first metadata about the partition, and obtain the second metadata of the data that has been stored in the storage node and belongs to the partition 1, so that the computing node 102 can access the data belonging to the partition 1 that has been stored in the storage node according to the second metadata. When the data in the partition 1 is stored in multiple storage nodes, the computing node 102 can access each of the multiple storage nodes respectively to read the data belonging to the partition 1 in each storage node. In the process of pulling data from the storage cluster 200, the computing node 102 can first pull the data to the receiving buffer area of ​​the computing node 102, such as Figure 4 As shown, the computing node 102 can subsequently perform a corresponding reduce operation on the data in the receiving buffer area. In this way, before all the data belonging to partition 1 are written to the storage cluster 200, the computing node 102 can start reading the data belonging to partition 1 from the storage cluster 200 according to the first metadata and the second metadata.

[0087] It is worth noting that, since there is data in the computing node 101 that has not been stored in the storage cluster 200, the data may include data belonging to partition 1 and other partitions. Therefore, the data belonging to partition 1 currently accessed by the computing node 102 from the storage cluster 200 is not all the data in partition 1. However, in the process of computing node 101 writing data to the storage cluster 200, computing node 102 can execute the process of reading data from the storage cluster 200 in parallel, which makes the uplink bandwidth resources and downlink bandwidth resources of the storage cluster 200 used, thereby alleviating the problem of waste of network resources of the storage cluster 200.

[0088] In actual application, computing node 102 can execute multiple reduce tasks generated by computing node 101. Then, when computing node 101 writes data to storage cluster 200, it can only read data required to execute one reduce task, or can read data required to execute multiple reduce tasks respectively.

[0089] In the first implementation, the computing node 102 can execute each reduce task in series. For example, the instruction node 202 schedules n reduce tasks to the computing node 102, where n is an integer greater than 1. Then, when the computing node 102 executes the first reduce task, it can read the data belonging to partition 1 required for executing the first reduce task while the computing node 101 writes data to the storage cluster 200. Since all the data belonging to partition 1 may not be written to the storage cluster 200 when the computing node 102 reads the data. Therefore, the data read by the computing node 102 is part of the data belonging to partition 1 that has been saved by the storage cluster 200 at the first moment. Among them, the first moment can be, for example, the moment when the computing node 102 accesses the storage cluster 200, or it can be any moment before accessing the storage cluster 200, which is not limited. In addition, in the process of the computing node 102 accessing the storage cluster 200, both the uplink bandwidth resources and the downlink bandwidth resources of the storage cluster 200 are used, so as to alleviate the problem of waste of network resources of the storage cluster 200.

[0090] Then, computing node 102 can wait for computing node 101 to write all the data in multiple partitions to storage cluster 200, and after storage cluster 200 successfully saves all the data in multiple partitions, computing node 102 can continue to read the remaining data that has not been saved before the first moment and belongs to partition 1 from storage cluster 200. In this way, computing node 102 can perform corresponding reduce operations based on all the read data belonging to partition 1, thereby completing the first reduce task. In this way, during the entire process of executing the first reduce task, computing node 102 can obtain part of the data from storage cluster 200 in advance, which can reduce the overall delay of computing node 102 in obtaining data, thereby improving the overall efficiency of computing node 102 in executing the first reduce task. Figure 5 As shown, the overall delay of computing node 102 executing the first reduce task can be reduced by the duration T0.

[0091] After completing the execution of the first reduce task, the computing node 102 can continue to execute the second reduce task to the nth reduce task in sequence. At this time, the T0 duration saved by the computing node 102 executing the first reduce task is the total duration saved by executing multiple reduce tasks. Since the data in multiple partitions have all been written to the storage cluster 200, when the computing node 100 executes the remaining reduce tasks, it can directly read all the data belonging to the same partition required by the reduce task from the storage cluster 200.

[0092] In actual application scenarios, when the data processing system 10 is applied to the RSS architecture, since the computing nodes under the RSS architecture usually execute reduce tasks serially, by adding plug-ins to the computing nodes under the RSS architecture, the computing nodes can read the data required to execute the reduce tasks from the storage cluster 200 in advance, thereby achieving non-invasive modification of the code in the computing nodes under the RSS architecture, reducing the difficulty of implementing the solution.

[0093] In the second implementation, computing node 102 can execute multiple reduce tasks in parallel, taking the case where computing node 102 can execute two reduce tasks in parallel as an example. Assume that computing node 101 schedules n reduce tasks to computing node 102, where n is an integer greater than 2. Then, when computing node 102 executes the first reduce task, it can read part of the data belonging to partition 1 that has been saved in storage cluster 200 at the first moment (i.e., the data required to execute the first reduce task) during the process of computing node 101 writing data to storage cluster 200. The first moment, for example, can be the moment when computing node 102 accesses storage cluster 200 for the first reduce task, or can be any moment before that moment.

[0094] Then, the computing node 102 may suspend the execution of the first reduce task, start the execution of the second reduce task, and read part of the data belonging to partition 2 that has been saved in the storage cluster 200 at the second moment (i.e., the data required for executing the second reduce task) during the time period between the first moment and the second moment. The second moment may be, for example, the moment when the computing node 102 accesses the storage cluster 200 for the second reduce task, or may be any moment before this moment. In this way, in the process of the computing node 102 accessing the storage cluster 200, the computing node 102 may read part of the data from the storage cluster 200 in advance for each of the multiple reduce tasks, which enables both the uplink bandwidth resources and the downlink bandwidth resources of the storage cluster 200 to be used, thereby alleviating the problem of waste of network resources of the storage cluster 200.

[0095] Assume that when the computing node 102 finishes reading the data belonging to partition 2 that has been saved by the storage cluster 200 before the second moment, the storage cluster 200 has already stored all the data in multiple partitions. Then, the computing node 102 can continue to read the remaining data belonging to partition 1 for the first reduce task, that is, read the data belonging to partition 1 that has not been saved by the storage cluster 200 before the first moment. Then, after reading and obtaining all the data required to execute the first reduce task (that is, all the data in partition 1), the computing node 102 can continue to read the remaining data belonging to partition 2 for the second reduce task, that is, read the data belonging to partition 2 that has not been saved by the storage cluster 200 before the second moment.

[0096] In actual application, when the number of reduce tasks executed in parallel by computing node 102 exceeds 2, computing node 102 can read the data belonging to partition 3 for the third reduce task in accordance with the above method, and so on, which will not be elaborated herein.

[0097] After all the data in multiple partitions are written to the storage cluster 200, for the reduce tasks that have read part of the data in advance, the computing node 102 can continue to read the remaining data from the storage cluster 200 for the reduce task to meet the data requirements when executing the reduce task. In this way, during the entire process of executing the reduce task, the computing node 102 can obtain part of the data from the storage cluster 200 in advance, which can reduce the overall latency of the computing node 102 in obtaining data, thereby improving the overall efficiency of the computing node 102 in executing the reduce task. As for the remaining reduce tasks that have not read part of the data in advance, the computing node 102 can directly read all the data required to execute the reduce task from the storage cluster 200 when executing the reduce task. Figure 5 As shown, the overall delay of the computing node 102 executing the first reduce task and the second reduce task can be reduced by T1. At this time, the T1 saved by the computing node 102 is the total time saved by executing multiple reduce tasks.

[0098] Furthermore, when computing node 102 is executing a reduce task, if the data read from storage cluster 200 is compressed data (obtained by computing node 101 compressing the data in the Map task), computing node 102 may first decompress the read compressed data, and then perform a corresponding reduce operation based on the compressed data.

[0099] It should be noted that in this embodiment, the example in which computing node 101 writes data in multiple partitions to multiple storage nodes and computing node 102 pulls data from multiple storage nodes is used for illustration, and this is not intended to be limiting.

[0100] For example, in other embodiments, the computing node 101 may execute the operation of pulling data from multiple storage nodes in parallel while writing data to multiple storage nodes.

[0101] For another example, in other embodiments, multiple computing nodes in the data processing system 10 may all write data to the storage cluster 200 , and each computing node may write different data in the same partition to the storage cluster 200 .

[0102] For another example, in other embodiments, computing node 101 may instruct multiple computing nodes in data processing system 10 to execute reduce tasks simultaneously, and different computing nodes execute different reduce tasks.

[0103] It is worth noting that other reasonable step combinations that can be thought of by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be familiar with the fact that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by this application.

[0104] Combination of the above Figures 1 to 5 The data processing system and data processing method provided in the embodiments of the present application are introduced, and then the structure of the computing device provided in the embodiments of the present application is introduced in conjunction with the accompanying drawings.

[0105] Figure 6 The hardware structure diagram of a computing device 600 provided in this application is shown in FIG. 6 . The computing device 600 can implement the above Figure 3 The computing node 101 or computing node 102 etc. in the illustrated embodiment.

[0106] like Figure 6 As shown, the computing device 600 includes a processor 601, a memory 602, and a communication interface 603. The processor 601, the memory 602, and the communication interface 603 communicate via a bus 604, and may also communicate via other means such as wireless transmission. The memory 602 is used to store instructions, and the processor 601 is used to execute the instructions stored in the memory 602. Furthermore, the computing device 600 may also include a memory unit 605, and the memory unit 605 may be connected to the processor 601, the storage medium 602, and the communication interface 603 via the bus 604. The memory 602 stores program codes, and the processor 601 may use the program codes stored in the memory 602 to execute the above-mentioned Figure 3 The operations performed by the computing node 101 in the embodiment shown in the figure, or performing the above Figure 3 Operations performed by computing node 102 in the illustrated embodiment.

[0107] It should be understood that in this embodiment, the processor 601 may be a CPU, and the processor 601 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0108] The memory 602 may include a read-only memory and a random access memory, and provides instructions and data to the processor 601. The memory 602 may also include a nonvolatile random access memory.

[0109] The memory 602 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0110] The communication interface 603 is used to communicate with other devices connected to the computing device 600. The bus 604 may include a power bus, a control bus, a status signal bus, etc. in addition to the data bus. However, for the sake of clarity, various buses are labeled as bus 604 in the figure.

[0111] It should be understood that the computing device 600 of the present application embodiment can be used to execute Figure 3 The method performed by the computing node 101 in the method shown is used to perform Figure 3 The method performed by the computing node 102 in the method shown, so the above and other operations and / or functions implemented by the computing device 600 are respectively to implement Figure 3 For the sake of brevity, the process of the corresponding method in will not be repeated here.

[0112] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned data processing method.

[0113] The embodiment of the present application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part.

[0114] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer or data center to another website, computer or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0115] The computer program product may be a software installation package, which may be downloaded and executed on a computing device when any of the aforementioned data processing methods is required.

[0116] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0117] The terms used in the above embodiments are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; the character " / " generally indicates that the objects associated with each other are in an "or" relationship. In the embodiments of the present application. "Simultaneously" refers to the same time period, including the situation at the same time.

[0118] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0119] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data processing system, characterized in that: The data processing system includes a plurality of computing nodes and a plurality of storage nodes, wherein the plurality of computing nodes include a first computing node that executes a mapping task and a second computing node that executes a specification reduce operation, and each first computing node executes at least one Map task; The first computing node is used to determine the partition to which the data corresponding to each Map task in the plurality of Map tasks belongs, determine the storage node used to store the data of each partition, and store first metadata, wherein the first metadata includes partition information of the partition to which the data corresponding to each Map task belongs and information of the storage node used to store the data of each partition; Writing data corresponding to each Map task to the multiple storage nodes; The multiple storage nodes are used to aggregate the data belonging to the same partition in the data corresponding to each Map task; The second computing node is used to obtain the first metadata during the process of the first computing node writing the data corresponding to each Map task to the multiple storage nodes, and obtain the data of the partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata.

2. The system according to claim 1, characterized in that The first computing node is further configured to: In the process of the first computing node writing the data corresponding to each Map task to the multiple storage nodes, after the first computing node saves the first metadata corresponding to all Map tasks, the second computing node is instructed to obtain the partitioned data corresponding to the reduce operation from the multiple storage nodes.

3. The system according to claim 2, characterized in that When storing the first metadata, the first computing node is specifically used to: After writing target data corresponding to a target Map task among the multiple Map tasks into a sending buffer in the first computing node, storing first metadata corresponding to the target data; After the first computing node saves the first metadata corresponding to all Map tasks, the first computing node instructs the second computing node to obtain the data of the partition corresponding to the reduce operation from the multiple storage nodes, specifically for: Increasing the number of send buffers in the first computing node; Writing all remaining data that has not been sent to the multiple storage nodes and is stored in the data corresponding to the multiple Map tasks into the sending buffer area, and saving the first metadata corresponding to the remaining data; Instruct the second computing node to obtain data of the partition corresponding to the reduce operation from the multiple storage nodes.

4. The system according to claim 3, characterized in that When the first computing node increases the number of the sending buffer areas in the first computing node, the first computing node is specifically configured to: Estimate a first duration, where the first duration is the duration for data corresponding to the Map tasks stored in the multiple storage nodes to be transmitted to the second computing node; Obtain a second duration, where the second duration is a remaining duration for the second computing node to save the data corresponding to all the Map tasks to the multiple storage nodes; When the first duration is greater than or equal to the second duration, the number of sending buffers in the first computing node is increased.

5. The system according to claim 4, characterized in that When estimating the first duration, the first computing node is specifically configured to: Determine the amount of data corresponding to the multiple Map tasks that has been saved to the multiple storage nodes; The first duration is estimated based on the number and the bandwidth of the second computing node reading data from the multiple storage nodes.

6. The system according to claim 4 or 5, characterized in that: The first computing node is further configured to: Acquire available resources of the second computing node; When the first computing node instructs the second computing node to obtain the data of the partition corresponding to the reduce operation from the multiple storage nodes, the first computing node is specifically used to: When the available resources of the second computing node meet the task execution conditions and the first duration is greater than or equal to the second duration, instruct the second computing node to obtain data of the partition corresponding to the reduce operation from the multiple storage nodes.

7. The system according to any one of claims 1 to 6, characterized in that: When the second computing node obtains the data of the partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata, it is specifically used to: Reading second metadata of the data of the partition corresponding to the reduce operation from the storage node indicated by the first metadata, wherein the second metadata is used to indicate a storage location of the data of the partition corresponding to the reduce operation in the storage node; The data of the partition corresponding to the reduce operation is read from the storage node indicated by the first metadata according to the second metadata.

8. The system according to any one of claims 1 to 7, characterized in that: When the second computing node obtains the data of the partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata, it is specifically used to: In the process of the first computing node writing the data corresponding to each Map task to the multiple storage nodes, reading part of the data of the first partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata; The second computing node is further used for: After the first computing node writes the data corresponding to all the Map tasks to the multiple storage nodes, the remaining data of the first partition corresponding to the reduce operation is read from the multiple storage nodes according to the first metadata.

9. The system according to any one of claims 1 to 7, characterized in that: When the second computing node obtains the data of the partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata, it is specifically used to: In the process of the first computing node writing the data corresponding to each Map task to the multiple storage nodes, reading part of the data of the first partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata; After reading part of the data of the first partition corresponding to the reduce operation, reading part of the data of the second partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata; The second computing node is further used for: After the first computing node writes the data corresponding to all Map tasks to the multiple storage nodes, the remaining data of the first partition corresponding to the reduce operation is read from the multiple storage nodes according to the first metadata, and the remaining data of the second partition corresponding to the reduce operation is read from the multiple storage nodes.

10. The system according to any one of claims 1 to 9, characterized in that: The data stored in the multiple storage nodes include compressed data, and the compressed data is obtained by compressing the data corresponding to the multiple Map tasks.

11. A data processing method, characterized in that: The method is applied to a data processing system, the data processing system includes a plurality of computing nodes and a plurality of storage nodes, the plurality of computing nodes include a first computing node that executes a mapping task and a second computing node that executes a specification reduce operation, each first computing node executes at least one Map task; The method comprises: The first computing node determines a partition to which data corresponding to each Map task in the plurality of Map tasks belongs, determines a storage node for storing the data of each partition, and stores first metadata, wherein the first metadata includes partition information of the partition to which the data corresponding to each Map task belongs and information of a storage node for storing the data of each partition; The first computing node writes the data corresponding to each Map task to the multiple storage nodes; The multiple storage nodes aggregate the data belonging to the same partition in the data corresponding to each Map task; During the process of the first computing node writing the data corresponding to each Map task to the multiple storage nodes, the second computing node obtains the first metadata, and obtains the partitioned data corresponding to the reduce operation from the multiple storage nodes according to the first metadata.

12. The method according to claim 11, characterized in that The method further comprises: In the process of the first computing node writing the data corresponding to each Map task to the multiple storage nodes, after the first computing node saves the first metadata corresponding to all Map tasks, the first computing node instructs the second computing node to obtain the data of the partition corresponding to the reduce operation from the multiple storage nodes.

13. The method according to claim 12, characterized in that The storing of the first metadata comprises: After the first computing node writes the target data corresponding to the target Map task among the multiple Map tasks into a sending buffer area in the first computing node, the first computing node stores the first metadata corresponding to the target data; After the first computing node saves the first metadata corresponding to all Map tasks, the first computing node instructs the second computing node to obtain the partition data corresponding to the reduce operation from the multiple storage nodes, including: The first computing node increases the number of sending buffers in the first computing node; The first computing node writes all remaining data of the data corresponding to the multiple Map tasks that have not been sent to the multiple storage nodes for storage into the sending buffer area, and saves the first metadata corresponding to the remaining data; The first computing node instructs the second computing node to obtain data of the partition corresponding to the reduce operation from the multiple storage nodes.

14. The method according to claim 13, characterized in that The first computing node increases the number of sending buffers in the first computing node, including: The first computing node estimates a first duration, where the first duration is the duration for data corresponding to the Map tasks stored in the plurality of storage nodes to be transmitted to the second computing node; The first computing node acquires a second duration, where the second duration is a remaining duration for the second computing node to save data corresponding to all Map tasks to the multiple storage nodes; When the first duration is greater than or equal to the second duration, the first computing node increases the number of sending buffers in the first computing node.

15. The method according to claim 14, characterized in that The first computing node estimating the first duration includes: The first computing node determines the amount of data corresponding to the plurality of Map tasks that has been saved to the plurality of storage nodes; The first computing node estimates the first duration according to the quantity and a bandwidth of the second computing node reading data from the multiple storage nodes.

16. The method according to claim 14 or 15, characterized in that The method further comprises: The first computing node obtains available resources of the second computing node; The first computing node instructs the second computing node to obtain data of the partition corresponding to the reduce operation from the multiple storage nodes, including: When the available resources of the second computing node meet the task execution conditions and the first duration is greater than or equal to the second duration, the first computing node instructs the second computing node to obtain data of the partition corresponding to the reduce operation from the multiple storage nodes.

17. The method according to any one of claims 11 to 16, characterized in that The second computing node obtains data of the partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata, including: The second computing node reads second metadata of the data of the partition corresponding to the reduce operation from the storage node indicated by the first metadata, where the second metadata is used to indicate a storage location of the data of the partition corresponding to the reduce operation in the storage node; The second computing node reads data of the partition corresponding to the reduce operation from the storage node indicated by the first metadata according to the second metadata.

18. The method according to any one of claims 11 to 17, characterized in that The second computing node obtains data of the partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata, including: In the process of the first computing node writing the data corresponding to each Map task to the multiple storage nodes, the second computing node reads part of the data of the first partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata; The method further comprises: After the first computing node writes the data corresponding to all Map tasks to the multiple storage nodes, the second computing node reads the remaining data of the first partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata.

19. The method according to any one of claims 11 to 17, characterized in that The second computing node obtains data of the partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata, including: In the process of the first computing node writing the data corresponding to each Map task to the multiple storage nodes, the second computing node reads part of the data of the first partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata; After reading the partial data of the first partition corresponding to the reduce operation, the second computing node reads the partial data of the second partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata; The method further comprises: After the first computing node writes the data corresponding to all Map tasks to the multiple storage nodes, the second computing node reads the remaining data of the first partition corresponding to the reduce operation from the multiple storage nodes according to the first metadata, and reads the remaining data of the second partition corresponding to the reduce operation from the multiple storage nodes.

20. The method according to any one of claims 11 to 19, characterized in that The data stored in the multiple storage nodes include compressed data, and the compressed data is obtained by compressing the data corresponding to the multiple Map tasks.

21. A computer-readable storage medium, characterized in that: The method comprises instructions which, when executed on at least one computing device, cause the at least one computing device to perform the steps of the method according to any one of claims 11 to 20.

22. A computer program product comprising instructions, characterized in that When it is executed on at least one computing device, the at least one computing device is caused to execute the method according to any one of claims 11 to 20.

Images