Data processing method and device and computing cluster

Abstract

The invention provides a data processing method and device and a computing cluster, relates to the technical field of computers, and can reduce data transmission delay and improve data processing efficiency. A plurality of first computing nodes in the computing cluster are used for executing N first tasks for processing to-be-processed data, M first results are generated after each first task is executed, and the M first results are respectively marked as A0-AM-1; each first data processing unit in the plurality of first data processing units is used for executing M second tasks, each second task in the M second tasks is used for processing the first results with the same identifier to generate second results, M second results are generated after the M second tasks are executed, and the M second results are respectively identified as B0-BM-1; the plurality of second computing nodes are used for executing M third tasks, each third task is used for processing the plurality of second results with the same identifier to generate third results, and M third results are generated after the M third tasks are executed.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a data processing method, apparatus, and computing cluster. Background Art

[0002] With the rapid development and popularization of data analysis technology, "big data" has become a household name in this era, and how to process big data has attracted much attention. Generally, large-scale data parallel processing can be carried out based on a programming model.

[0003] For example, a commonly used big data programming model is the Map-Reduce model. The process of using this model for data processing includes a Map stage and a Reduce stage, and each of the Map stage and the Reduce stage executes some operators. In the Map stage, a complex task is decomposed into several simple subtasks, so as to reduce the data processing scale, perform data local processing, and parallel processing; in the Reduce stage, the processing results generated in the Map stage are summarized.

[0004] The several subtasks divided in the above Map stage are executed by multiple computing nodes, and usually a computing node executes multiple subtasks. When the Reduce stage further processes the data generated in the Map stage, it is necessary to obtain (or pull) the data processing results of each subtask. Since the number of subtasks in the Map stage is large, a large number of data pulls are required in the Reduce stage, resulting in a high data transmission delay in the data processing process. Summary of the Invention [[ID=1⑧]]

[0005] This application provides a data processing method, apparatus, and computing cluster, which can reduce the data transmission delay in the data processing process and improve the efficiency of data processing.

[0006] This application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a computing cluster, which includes: a plurality of first computing nodes, a plurality of first data processing units, each first computing node is connected to a first data processing unit, and a plurality of second computing nodes. The plurality of first computing nodes are configured to execute N first tasks, where the N first tasks are used to process N data shards of the data to be processed, and generate M first results after executing each first task, and are respectively identified as A0 to AM-1; each first data processing unit is configured to execute M second tasks, where each of the M second tasks is used to process the first results with the same identifier in the first computing node connected to the first data processing unit, and generate second results after processing, and generate M second results after executing the M second tasks, and the M second results are identified as B0 to BM-1; the plurality of second computing nodes are configured to execute M third tasks, and each third task is used to process the second results with the same identifier in the plurality of first data processing units to generate third results, and generate M third results after executing the M third tasks.

[0008] In the computing cluster of the present application, since the first data processing unit connected to each first computing node can first execute the second task on the first results generated after the first computing node executes multiple first tasks, generate the second results, and then the second computing node executes the third task to process the second results generated by the multiple computing nodes, it is possible to avoid the direct pulling of the data generated after the same computing node executes multiple first tasks by the second computing node for multiple times and for a large amount of data, thereby solving the problem of large data transmission delay caused by multiple and large amount of data pulling, being able to reduce the data transmission delay in the data processing process, and improving the efficiency of data processing.

[0009] In a possible implementation, the first task is a map task, and the second task and the third task are reduce tasks.

[0010] In a possible implementation, the third computing node among the plurality of first computing nodes is further configured to send a first request to the connected first data processing unit to instruct the first data processing unit to execute M second tasks.

[0011] In a possible implementation, the first request carries the operator identifier of the operator to be called during the execution of the second task and the storage location information of the M first results.

[0012] In a possible implementation, the computing cluster provided by this application further includes: a plurality of second data processing units, and each second computing node is connected to one second data processing unit. The fourth computing node among the plurality of second computing nodes is further configured to send a second request to the connected second data processing unit to instruct the second data processing unit to execute M third tasks. In this application, offloading the M third tasks executed by the second computing node to the second data processing unit for execution can reduce the load on the second computing node, thereby saving the resources of the second computing node.

[0013] In a possible implementation, the second request carries the operator identifier of the operator to be called during the execution of the third task and the storage location information of the M second results.

[0014] In a possible implementation, the third data processing unit among the plurality of first data processing units is further configured to send a notification to the fifth computing node among the plurality of second computing nodes to indicate that the execution of the M second tasks has ended, thereby triggering the second computing node to execute the M third tasks.

[0015] In a possible implementation, the fourth data processing unit among the plurality of first data processing units is further configured to receive a third request to instruct to update the operator to be called by the fourth data processing unit, and load the operator logic microcode of the operator to be updated into the fourth data processing unit. Since the hardware physical resources of a data processing unit (such as a DPU) are limited, usually only a small number of frequently used hardware operators can be solidified. In the embodiments of this application, the logic circuit of the data processing unit supports the update of hardware operators. Therefore, the operators in the data processing unit can be updated flexibly, expanding the range of operators that the data processing unit can execute.

[0016] In a possible implementation, the third request carries the operator logic microcode of the operator to be updated.

[0017] In a possible implementation, the operator to be updated is a custom operator. The user designs and generates the operator logic microcode according to the actual data processing requirements, and updates the custom operator to the data processing unit, that is, persists the operator logic microcode into the data processing unit. In this application, supporting user-defined operators can meet the needs of different users, making the application range of the computing cluster wider.

[0018] In a second aspect, this application provides a data processing method, which is applied to a computing cluster. The computing cluster includes a plurality of first computing nodes, a plurality of first data processing units, and a plurality of second computing nodes, where each first computing node is connected to one first data processing unit; the method includes: the plurality of first computing nodes execute N first tasks, and generate M first results after executing each first task, and are respectively identified as A0 to AM-1 ; wherein, the N first tasks are used to process N data shards of the data to be processed; and each first data processing unit executes M second tasks, and after executing the M second tasks, M second results are generated, and the M second results are labeled as B0 to B M-1 ; wherein, each of the M second tasks is used to process the first results with the same identifier in the first computing node connected to the first data processing unit, and a second result is generated after the processing; furthermore, multiple second computing nodes execute M third tasks, and after executing the M third tasks, M third results are generated; each third task is used to process the second results with the same identifier in the multiple first data processing units to generate a third result.

[0019] In a possible implementation manner, the above-mentioned first task is a map task, and the above-mentioned second task and third task are reduce tasks.

[0020] In a possible implementation manner, the data processing method provided by this application further includes: a third computing node among the multiple first computing nodes sends a first request to the connected first data processing unit to instruct the first data processing unit to execute M second tasks.

[0021] In a possible implementation manner, the first request carries the operator identifier of the operator to be called during the execution of the second task and the storage location information of the M first results.

[0022] In a possible implementation manner, the computing cluster further includes multiple second data processing units, and each second computing node is connected to a second data processing unit; based on this, the data processing method provided by this application further includes: a fourth computing node among the multiple second computing nodes sends a second request to the connected second data processing unit to instruct the second data processing unit to execute M third tasks; wherein, the second request carries the operator identifier of the operator to be called during the execution of the third task and the storage location information of the M second results.

[0023] In a possible implementation manner, the data processing method provided by this application further includes: a third data processing unit among the multiple first data processing units sends a notice to a fifth computing node among the multiple second computing nodes to indicate that the execution of the M second tasks ends.

[0024] In a possible implementation manner, the data processing method provided by this application further includes: a fourth data processing unit among the multiple first data processing units receives a third request to instruct to update the operator to be called by the fourth data processing unit, and the third request carries the operator logic microcode of the operator to be updated; and the fourth data processing unit loads the operator logic microcode of the operator to be updated.

[0025] In a possible implementation, the operator to be updated above is a custom operator.

[0026] In a third aspect, the present application provides a computing node, including a memory and at least one processor connected to the memory. The memory is used to store computer program code, and the computer program code includes computer instructions. When the computer instructions are executed by the at least one processor, the computing node is enabled to perform the actions performed by the first computing node in the first aspect or the second aspect; or, perform the actions performed by the second computing node in the first aspect or the second aspect.

[0027] In a fourth aspect, the present application provides a data processing unit, including a memory and at least one processor connected to the memory. The memory is used to store computer program code, and the computer program code includes computer instructions. When the computer instructions are executed by the at least one processor, the data processing unit is enabled to perform the actions performed by the first data processing unit in the first aspect or the second aspect; or, perform the actions performed by the second data processing unit in the first aspect or the second aspect.

[0028] In a fifth aspect, the present application provides a computer-readable storage medium storing computer instructions, and when the computer instructions run on a computer, they execute the method in any one of the second aspect and its possible implementations.

[0029] In a sixth aspect, the present application provides a computer program product, which includes computer instructions, and when the computer instructions run on a computer, they execute the method in any one of the second aspect and its possible implementations.

[0030] In a seventh aspect, the present application provides a chip system, including: a processor, configured to call and run a computer program from a memory, so that a device equipped with the chip system executes the method in any one of the second aspect and its possible implementations.

[0031] It should be understood that for the beneficial effects obtained by the technical solutions of the second aspect to the seventh aspect of the present application and the corresponding possible implementation manners, reference may be made to the technical effects of the first aspect and its corresponding possible implementation manners described above, and details are not described herein again. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 One of the framework diagrams for data processing based on the Map-Reduce model provided by the embodiments of the present application;

[0033] Figure 2 Another framework diagram for data processing based on the Map-Reduce model provided by the embodiments of the present application;

[0034] Figure 3One of the schematic diagrams of the architecture of a computing cluster provided by an embodiment of the present application;

[0035] Figure 4 Another schematic diagram of the architecture of a computing cluster provided by an embodiment of the present application;

[0036] Figure 5 Another schematic diagram of the architecture of a computing cluster provided by an embodiment of the present application;

[0037] Figure 6 Another schematic diagram of the architecture of a computing cluster provided by an embodiment of the present application;

[0038] Figure 7 One of the schematic flowcharts of a data processing method provided by an embodiment of the present application;

[0039] Figure 8 Another schematic flowchart of a data processing method provided by an embodiment of the present application;

[0040] Figure 9 Schematic diagram of an operator update method provided by an embodiment of the present application. Detailed implementation manners

[0041] In this article, the term "and / or" is merely a description of the association relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0042] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order of the objects. For example, the first task and the second task are used to distinguish different tasks, rather than to describe a specific order of the tasks; the first request and the second request are used to distinguish different requests, rather than to describe a specific order of the requests.

[0043] In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly, using words such as "exemplary" or "for example" aims to present relevant concepts in a specific manner.

[0044] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality" refers to two or more, and "a plurality" can also be described as "at least two".

[0045] The data processing method, apparatus, and computing cluster provided by the embodiments of this application are mainly applied to the scenario of distributed big data processing, and the programming model for big data processing should satisfy at least one of the following conditions:

[0046] 1) The programming model includes different processing stages. There are data dependencies between different processing stages (for example, the subsequent processing stage depends on the processing result of the previous processing stage). Different processing stages respectively include multiple tasks distributed on different computing nodes (i.e., tasks executed by different computers), and multiple tasks are executed on one computing node.

[0047] 2) The subsequent processing stage needs to obtain a large number of data shards in the processing result generated by the previous processing stage.

[0048] It should be noted that the programming model applicable to the embodiments of this application can be any programming model that meets the above conditions, such as, but not limited to, the Map-Reduce model for data processing. In the embodiments of this application, the Map-Reduce model is mainly used as an example to describe the relevant content of the data processing method.

[0049] First, a brief introduction to the relevant knowledge of the Map-Reduce model is given.

[0050] The Map-Reduce model is a big data programming paradigm, that is, a data processing task (hereinafter also simply referred to as a task) is divided into multiple Map-Reduces. One Map-Reduce includes a Map stage and a Reduce stage. In each Map-Reduce, the Map stage and the Reduce stage execute the operators corresponding to each stage. The types of operators executed in the Map stage and the Reduce stage are related to the actual business requirements, and are not limited in the embodiments of this application.

[0051] The Map-Reduce model is used for big data processing in a computing cluster. The computing cluster includes computing nodes that execute tasks, and may also include management nodes for task scheduling and resource management. Taking the computing cluster as a distributed computing cluster as an example, the resource manager pools the computing resources (processing cores) and memory resources of all computing nodes in the computing cluster. For a data processing task to be executed, the task scheduler divides the input data (also referred to as raw data) into multiple data shards according to a certain granularity (such as 64M, etc.), and then applies for sufficient resources from the resource pool according to a certain container specification (such as 4 cores + 12G) for this data processing task to execute the task.

[0052] Specifically, refer to Figure 1The framework diagram for data processing based on the Map-Reduce model shown. In the Map stage, according to the number of data shards after partitioning the above input data (for example, N data shards, where N is an integer greater than 1), N map tasks (which can also be called map tasks) for the data processing task are generated. The N map tasks can be homogeneous (that is, the N map tasks execute the same operator or function). Each of the N map tasks performs operator calculations, and each map task generates M data shards. In the Reduce stage, M reduce tasks (which can also be called reduce tasks) for this data processing task are generated. Among them, for each reduce task, it is necessary to pull the data shard with a certain subscript corresponding to the reduce task among the M data shards generated by all map tasks, and then execute the operator in the Reduce stage. Exemplarily, as Figure 1 shown, reduce task 1 needs to pull data shard 1_1 among the M data shards generated by map task 1, data shard 2_1 among the M data shards generated by map task 2, and so on, until it pulls data shard M_1 among the M data shards generated by map task N. It can be seen that for one reduce task, it is necessary to pull data shards N times.

[0053] Optionally, the data generated by each reduce task can be further divided into finer granularity, for example, divided into K data shards, and then these K data shards with finer granularity are used as input data to execute the next Map-Reduce.

[0054] It should be understood that generally, the N map tasks generated in the Map stage are executed by different first computing nodes, and multiple map tasks are executed on one first computing node. The M reduce tasks generated in the Reduce stage can also be executed by different second computing nodes, and multiple reduce tasks can also be executed on one second computing node.

[0055] Optionally, the map tasks and reduce tasks can be executed by the same computing node (that is, the first computing node and the second computing node are the same computing node), or can be executed by different computing nodes (that is, the first computing node and the second computing node are different computing nodes), and the embodiments of the present application do not make limitations.

[0056] Refer to Figure 2The framework diagram for data processing based on the Map-Reduce model in the shown computing cluster. For the data to be processed, the data to be processed is divided into N data shards. Then, in the Map stage, N map tasks need to be executed, and the N map tasks are executed by multiple computing nodes in the computing cluster. For example, as Figure 2 shown, 3 computing nodes process N data shards. Among them, computing node 1 executes N1 map tasks, computing node 2 executes N2 map tasks, and computing node 3 executes N3 map tasks, where N1 + N2 + N3 = N. After each map task finishes execution, M data shards are generated.

[0057] Continue to refer to Figure 2 , in the Reduce stage, M reduce tasks need to be executed, and the M reduce tasks can be executed by one or more computing nodes. Figure 2 What is shown in Figure 2 is that M reduce tasks are respectively executed by M computing nodes. For example, computing node 4 executes reduce task 1, computing node 5 executes reduce task 2, and computing node ⑥ executes reduce task M. As Figure 2 shown, reduce task 1 needs to pull the first data shard among all the data shards generated by map tasks. Specifically, each of the N1 map tasks on computing node 1 generates M data shards, so reduce task 1 needs to pull the data shards from computing node 1 for N1 times; each of the N2 map tasks on computing node 2 generates M data shards, so reduce task 1 needs to pull the data shards from computing node 2 for N2 times; each of the N3 map tasks on computing node 3 generates M data shards, so reduce task 1 needs to pull the data shards from computing node 3 for N3 times. Reduce task 1 needs to pull a total of N (N1 + N2 + N3 = N) times; similarly, reduce task 2 needs to pull the second data shard among all the data shards generated by map tasks. For example, reduce task 2 pulls the data shards from computing node 1 for N1 times, from computing node 2 for N2 times, and from computing node 3 for N3 times, and it needs to pull a total of N times.

[0058] Combined with Figure 2, currently, in one implementation, when the map task and the reduce task are executed by different computing nodes, during the above data pulling process, it is necessary to pull data across computing nodes. For example, the reduce task pulls the corresponding data shards from the data shards generated by multiple map tasks through shuffle (the operation of data pulling). Shuffle is a data distribution mechanism that crosses nodes and processes within the computing cluster, and can also be understood as the data exchange between the Reduce stage and the Map stage.

[0059] For more content about shuffle and the detailed process of pulling data through shuffle, reference can be made to the existing technical materials, and this application will not elaborate.

[0060] During the above shuffle process, it is necessary to pull cross-node data through the network. Since the network bandwidth resources in the computing cluster are limited, pulling a large number of data shards generated by map tasks through shuffle will occupy more network bandwidth. Moreover, when there are a large number of map tasks, pulling a large number of data shards multiple times results in a relatively large number of read and write (I / O) operations (input / output operations per second, IOPS) per second on the network, which is likely to cause network congestion, thereby leading to a relatively large transmission delay from end to end (i.e., from the computing node executing the map task to the computing node executing the reduce task). In addition, when network congestion occurs due to excessive network load, the network performance deteriorates sharply, affecting the normal operation of other jobs.

[0061] It should be understood that in the Reduce stage, after the reduce task pulls a certain data shard generated by all map tasks through shuffle, the types of operators that the reduce task can execute can include, but are not limited to, operators such as Join (data splicing), Aggregate (aggregation), Filter (filtering), Sort (sorting), distinct (duplicate removal), subtract (set difference), sample (sampling), etc. Among them, each type of operator can also include different types of operators. For example, the Join operator can include types of operators such as cross join, inner join, outer join, self join, etc.

[0062] Among the operators executed in the Reduce stage, some operators are characterized by a significant reduction in the amount of data before and after the operation, that is, the input data volume of the operator is very large, and after the operator operation, the output data volume is very small. That is to say, relative to these operators, when shuffling through the network, a huge amount of original data (i.e., input data) is actually pulled, and the data transmission delay will be more obvious.

[0063] In view of the problem of large data transmission delay caused by the reduce task pulling all the data shards generated by the map tasks during the above data processing process, the embodiments of the present application provide a data processing method, apparatus, and computing cluster. The computing cluster includes multiple first computing nodes, multiple first data processing units, and multiple second computing nodes. Among them, each first computing node is connected to a first data processing unit, and multiple first computing nodes can execute N first tasks (such as map tasks). The N first tasks are used to process N data shards of the data to be processed. After each first task is executed, M first results (the M first results are M data shards) are generated and are respectively identified as A0 to A M-1 ; each first data processing unit executes M second tasks. Each second task in the M second tasks is used to process the first results with the same identifier in the first computing node connected to the first data processing unit. After the processing, second results are generated. After the M second tasks are executed, M second results are generated, and the M second results are identified as B0 to B M-1 ; and then multiple second computing nodes execute M third tasks. Each third task is used to process the second results with the same identifier in multiple first data processing units to generate third results. After the M third tasks are executed, M third results are generated.

[0064] In this method, during the data processing process, the first results generated after multiple first tasks executed by the first computing node connected to each first data processing unit are first processed by the second tasks to generate second results, and then the second computing node executes the third tasks to process the second results generated by multiple computing nodes, which can avoid the direct pulling of the data generated after multiple first tasks executed by the same computing node by the second computing node multiple times and in large quantities, thereby solving the problem of large data transmission delay caused by multiple and large-scale data pulling, that is, it can reduce the data transmission delay during the data processing process and improve the efficiency of data processing.

[0065] The data processing method, apparatus, and computing cluster provided by the embodiments of the present application can be applied to the scenario of big data distributed computing based on the Map-Reduce model. The computing cluster and data processing method provided by the embodiments of the present application will be described in detail below.

[0066] An embodiment of the present application provides a computing cluster. Figure 3 Figure 3 is a schematic architecture diagram of a computing cluster provided by an embodiment of the present application. The computing cluster includes a plurality of first computing nodes 301 and a plurality of second computing nodes 302. Each first computing node 301 is connected to a first data processing unit 303.

[0067] The computing nodes (including the first computing nodes and the second computing nodes) in the computing cluster can be servers, such as central servers, edge servers, or local servers in a local data center. In some embodiments, the computing nodes can also be terminal devices such as desktops, laptops, or smartphones.

[0068] Optionally, the data processing unit provided by the embodiment of the present application can be a DPU (data processing unit, DPU). It can be understood that the DPU is a dedicated data processing unit (or data processor) with strong data processing capabilities. As an intelligent network card, the DPU is connected to a host (such as a computing node) through a network card interface and realizes data processing functions under the control of the computing node. Of course, the data processing unit connected to each computing node can also be other data processing devices, which are not limited in the embodiments of the present application.

[0069] For a data processing task, which is used to process the data to be processed and divides the data to be processed into N data shards for processing. In the embodiment of the present application, executing the data processing task includes executing N first tasks, M second tasks, and M third tasks. Both M and N are integers greater than or equal to 2. Taking the data processing model as the Map-Reduce model as an example, the above-mentioned first task is a map task (hereinafter simply referred to as a map task), and the above-mentioned second task and third task are reduce tasks (hereinafter simply referred to as reduce tasks). The N first tasks (map tasks) are executed by a plurality of first computing nodes 301, the M second tasks are executed by the first data processing unit 303, and the M third tasks are executed by a plurality of second computing nodes 302.

[0070] In the embodiment of the present application, the Map-Reduce model is used to process the N data shards obtained by dividing the data to be processed.

[0071] Combined with Figure 3 , such as Figure 4As shown in the figure, there are multiple first computing nodes 301, which are used to execute N first tasks (map tasks). The N first tasks are used to process N data shards of the data to be processed, that is, one first task processes one data shard (the N first tasks correspond to the N data shards one by one). Among them, each first computing node 301 executes multiple first tasks to process multiple data shards among the N data shards, and generates M first results after executing each first task. The M first results are respectively labeled as A0 to A M-1 .

[0072] Each first data processing unit 303 among the multiple first data processing units 303 is used to execute M second tasks (reduce tasks). Each of the M second tasks is used to process the first results with the same label in the first computing node connected to the first data processing unit, and generates a second result after processing. After executing the M second tasks, M second results are generated. The M second results are labeled as B0 to B M-1 . Refer to Figure 4 , taking one second task executed by a first data processing unit as an example. When executing the second task, from the multiple first results obtained after the multiple first tasks are executed by the connected first computing node, pull the multiple first results with the label A0, and then execute the second task to generate a second result B0.

[0073] There are multiple second computing nodes 302, which are used to execute M third tasks (reduce tasks). Each third task is used to process the second results with the same label in the multiple first data processing units and generate a third result. After executing the M third tasks, M third results are generated. Among them, each second computing node 302 executes one or more third tasks to process the second results generated by the multiple first data processing units communicatively connected to the second computing node 302. For example Figure 4 in, the first second computing node 302 executes n1 + 1 reduce tasks, and the generated third results are labeled as C0 to C n1 . Refer to Figure 4 , taking the third task executed by a second computing node 302 as an example. When executing the third task, from the multiple second results generated by the multiple first data processing units, pull the multiple second results with the label B0, and then execute the third task to generate a third result C0.

[0074] It should be noted that the input data of each of the multiple first tasks (i.e., one data shard among N data shards) is different. After executing the multiple first tasks, the first results generated after each first task are different. Similarly, the input data of each of the multiple second tasks is different. After the multiple first data processing units execute the multiple second tasks, the generated second results are different. Therefore, Figure 4 in M-1 ), the identification of the first result (A0 to A M-1 ) and the second result (B0 to B

[0075] is only used to identify the data shards in each processing result, and does not limit the content of the data shards in the processing result.

[0075] Referring to Figure 4 , for example, the first result (i.e., the first data shard) among the M first results generated by different map tasks is all identified as A0, but it does not mean that the contents of the multiple first results identified as A0 are the same. Another example is that the first result among the M second results generated after different first data processing units execute the first reduce task is all represented as B0, but it does not mean that the contents of the multiple second results identified as B0 are the same.

[0076] Combined with Figure 4 , as Figure 5 shown, the computing cluster provided by the embodiments of the present application may further include multiple second data processing units 304, and each second computing node 302 is connected to a second data processing unit 304. When the computing cluster includes multiple second data processing units 304, the process of the multiple second computing nodes 302 executing M third tasks (reduce tasks) described above may be replaced by the second data processing units 304 connected to the second computing nodes 302. For example, each second computing node may instruct the second data processing unit 304 connected thereto to process one or more third tasks to process the second results with the same identification among the multiple first data processing units. After the multiple second computing nodes execute the M third tasks, M third results are generated.

[0077] In the embodiments of the present application, offloading the M third tasks executed by the second computing nodes to the second data processing units for execution can reduce the load of the second computing nodes, thereby saving the resources of the second computing nodes.

[0078] Optionally, the computing cluster may further include other nodes, such as a management node. The management node is used to manage the computing nodes in the computing cluster. The management node may have functions of task scheduling and resource management. For example, a task scheduler and / or a resource manager may be deployed on the management node to schedule data processing tasks and allocate computing resources and memory resources for data processing tasks.

[0079] In summary, for the computing cluster provided in the embodiments of the present application, since the first data processing unit connected to each first computing node can first execute a second task on the first results generated after multiple first tasks executed by the first computing node to generate second results, and then the second computing node executes a third task to process the second results generated by multiple computing nodes, it is possible to avoid the second computing node directly pulling multiple times and a large amount of data from the data generated after the first computing node executes multiple first tasks, thereby solving the problem of large data transmission delay caused by multiple times and a large amount of data pulling, being able to reduce the data transmission delay during the data processing process, and improving the data processing efficiency.

[0080] It can be understood that multiple first computing nodes 301, multiple second computing nodes 302, multiple first data processing units 303, and multiple second data processing units 304 in the computing cluster interact with each other to complete the processing process of the data to be processed.

[0081] In one implementation, the third computing node among the multiple first computing nodes 301 is further configured to send a first request to the first data processing unit 303 connected thereto to instruct the first data processing unit 303 to execute M second tasks, where the third computing node can be any one of the multiple first computing nodes 301. It can be understood that after each first computing node 301 executes multiple first tasks (map tasks), it sends a first request to the first data processing unit 303 connected thereto, so that the first data processing unit 303 starts to execute the second tasks on the multiple first results generated within the first computing node.

[0082] Optionally, the above first request carries the operator identifier of the operator to be called during the execution of the second task and the storage location information of the M first results. The operator identifier is used to indicate the operator to be called included in the second task, and the storage location information is used to indicate the storage location of the M first results, such as the physical address of the M first results in the memory. In this way, the first data processing unit 303 reads the M first results according to the storage location information and calls the operator indicated by the operator identifier to execute the second task.

[0083] In one implementation, the third data processing unit among the multiple first data processing units 303 is further configured to send a notification to the fifth computing node among the multiple second computing nodes to indicate the end of the execution of the M second tasks, where the third data processing unit is any one of the multiple first data processing units 303, and the fifth computing node is the second computing node corresponding to the third data processing unit (i.e., the second computing node for processing the result generated by the third data processing unit). Optionally, after the third data processing unit finishes executing the M second tasks, it sends a notification indicating the end of the execution of the M second tasks to the first computing node 301 connected to it. Further, the first computing node 301 forwards this notification to the second computing node 302.

[0084] Optionally, the notification indicating the end of the execution of the M second tasks carries the storage location information of the M second results. In this way, after receiving this notification, the second computing node reads the M second results according to the storage location information to execute the third task.

[0085] In one implementation, when the computing cluster includes multiple second data processing units 304, the fourth computing node among the multiple second computing nodes 302 is further configured to send a second request to the connected second data processing unit 304 to indicate that the second data processing unit 304 executes the M third tasks, where the fourth computing node can be any one of the multiple second computing nodes 301. It can be understood that after each first data processing unit 303 finishes executing the multiple second tasks (reduce tasks), it sends a first request to the first data processing unit 303 connected to it, so that the first data processing unit 303 starts to execute the second tasks on the multiple first results generated in the first computing node.

[0086] Optionally, the second request carries the operator identifier of the operator to be called during the execution of the third task and the storage locations of the M second results. In this way, the second data processing unit 304 reads the M second results according to the storage location information and calls the operator indicated by the operator identifier to execute the third task.

[0087] It can be understood that each first data processing unit 303 executes the M second tasks, and the process of executing the second tasks is actually to call the operators to be called included in the second tasks.

[0088] Based on this, in one implementation, the fourth data processing unit among multiple first data processing units 303 is further configured to receive a third request to instruct to update the operator to be called by the fourth data processing unit, and load the operator logic microcode of the operator to be updated into the fourth data processing unit, so as to persist the operator to be updated in the fourth data processing unit. Wherein, the fourth data processing unit may be any one of the multiple first data processing units 303.

[0089] Optionally, the operator logic microcode of the operator to be updated carried in the above third request may further include an operator identifier, so as to store the operator logic microcode according to the correspondence between the operator identifier and the operator logic microcode.

[0090] It should be understood that generally, due to the limited hardware physical resources of a data processing unit (such as a DPU), usually only a small number of frequently used hardware operators can be solidified. In the embodiments of the present application, the logic circuit of the data processing unit supports the update of hardware operators. Therefore, the operators in the data processing unit can be flexibly updated, expanding the range of operators executable by the data processing unit.

[0091] Optionally, in the embodiments of the present application, the operator to be updated may be a user-defined operator. The user designs and generates the operator logic microcode according to actual data processing requirements, and updates the user-defined operator to the data processing unit, that is, persists the operator logic microcode in the data processing unit. Supporting user-defined operators can meet the needs of different users and make the application range of the computing cluster wider.

[0092] The following embodiments describe the data processing process from the perspective of the internal module interaction between each computing node and data processing unit in the computing cluster. Taking Figure 5 the shown computing cluster as an example, for the sake of convenience of description, the data processing process of the computing cluster is described by taking 2 first computing points and 1 second computing node in the computing cluster as an example.

[0093] As Figure 6 shown, for the first computing node 301a in the computing cluster, the first data processing unit 303a connected to the first computing node 301a, the first computing node 301b, the first data processing unit 303b connected to the first computing node 301b, the second computing node 302, and the second data processing unit 304 connected to the second computing node 302.

[0094] Among them, the first computing nodes 301a, 301b and the second computing node 302 all include an agent module. The agent module is responsible for converting the received request from the service semantics into a request for hardware computing. The first data processing units 303a, 303b and the second data processing unit 304 all include a request handler module and a data offloading engine (DOE). The DOE is a hardware module for operator operations.

[0095] Among them, after the first computing node 301a finishes executing multiple map tasks (the first tasks), each map task generates M first results (labeled as A0 to A M-1 ), and the agent module in the first computing node 301a sends a first request to the request handling module in the first data processing unit 303a to instruct the first data processing unit 303a to execute a reduce task (the second task) on the multiple first results generated by the first computing node 301b.

[0096] The request handling module in the first data processing unit 303a forwards the first request to the DOE of the first data processing unit 303a to enable the DOE to execute the reduce task. For example, referring to Figure 6 , for each first computing node 301a, the first data processing unit 303a pulls the first result A0 generated by each map task and executes the reduce task (operators such as Join operator and Aggregate operator need to be called during the execution of the reduce task). After the execution, a second result B0 is obtained.

[0097] It should be understood that the process of data processing by the first computing node 301b and the first data processing unit 303b can refer to the process of data processing by the first computing node 301a and the first data processing unit 303a, which will not be elaborated here.

[0098] Furthermore, after the first data processing unit 303a and the first data processing unit 303b both finish executing multiple reduce tasks, they notify the agent module of the second computing node 302 through the agent module. Then, the agent module of the second computing node 302 sends a second request to the request handling module of the second data processing unit 304 to instruct the second data processing unit 304 to execute M reduce tasks (the third task).

[0099] The request processing module of the second data processing unit 304 forwards the second request to the DOE of the second data processing unit 304, so that the DOE executes the reduce task. Figure 6 The second data processing unit 304 pulls the second result B0 generated by the first data processing unit 303a and the second result B0 generated by the first data processing unit 303b, and performs reducetask, and obtains a third result C0 after the execution is completed.

[0100] Based on the above content, it can be seen that in the embodiment of the present application, all operator operations in the Reduce stage originally completed by the second computing node 302 are first performed on the node-level reducetask inside the first computing node according to the granularity of the computing node to obtain M second results. In this way, the number of times and the amount of data pulled by the second computing node are reduced.

[0101] Based on the architecture of the computing cluster described in the above embodiments, the embodiments of the present application also provide a data processing method. The data processing method provided in the embodiments of the present application is described in more detail below from the perspective of interaction between various modules within a first computing node, a first data processing unit (first DPU), a second computing node, and a second data processing unit (second DPU).

[0102] like Figure 7 As shown, in one implementation, the data processing method provided in the embodiment of the present application includes the following steps.

[0103] S701: After executing a plurality of first tasks, the processing module of the first computing node sends a notification message to the agent module of the first computing node to indicate completion of execution of the plurality of first tasks.

[0104] After each first task is executed, M first results are generated (respectively marked as A0~A M-1 ). Optionally, the first result generated after the first computing node executes each first task is cached in the memory of the first computing node.

[0105] S702. The agent module of the first computing node sends a first request to the request processing module of the first DPU.

[0106] The first request is used to instruct the first DPU to execute M second tasks. The first request carries the operator identifier of the operator to be called during the execution of the second task and the storage location information of the M first results. For the description of the first request, please refer to the description of the above embodiment and will not be repeated here.

[0107] S703: After receiving the first request, the request processing module of the first DPU applies for a cache that meets the required size.

[0108] The application cache is used to store the results generated after the first DPU executes tasks.

[0109] S704. The request processing module of the first DPU sends a first request to the DOE of the first DPU to trigger the DOE of the first DPU to execute M second tasks.

[0110] Optionally, the request processing module of the first DPU fills the operator identifier of the operator to be called and the storage location information of the M first results into the register of the DOE of the first DPU to trigger the DOE to execute M second tasks.

[0111] S705. The DOE of the first DPU executes M second tasks, generating M second results (respectively identified as B0 to B M-1 ).

[0112] After the DOE of the first DPU obtains the operator identifier of the operator to be called and the storage location information of the M first results, it obtains the M first results and executes M second tasks to obtain M second results.

[0113] The operators called in the execution of the second tasks may include operators with the following characteristics: operators with a large amount of input data and a significantly reduced amount of output data after executing the operator. For example, the Join operator, Aggregate operator, etc. mentioned in the above embodiments. The operators called in the second tasks may also include other operators, such as the Sort operator, etc.

[0114] S706. The DOE of the first DPU writes the M second results into the application cache.

[0115] S707. The first DPU sends a notification message to the proxy module of the first computing node through the request processing module of the first DPU. This notification message is used to notify that the execution of the M second tasks has ended.

[0116] This notification message may carry the storage location information of the M second results and the data lengths of the M second results.

[0117] S708. The proxy module of the first computing node forwards the notification message in S707 to the proxy module of the second computing node.

[0118] S709. The second computing node sends a second request to the request processing module of the second DPU through its processing module and proxy module.

[0119] This second request is used to instruct the second data processing unit to execute M third tasks. The second request carries the operator identifier of the operator to be called during the execution of the third tasks and the storage location information of the M second results.

[0120] Optionally, the proxy module of the second computing node fills the storage location information of the M second results generated by the first DPU and the operator identifiers of the operators to be called during the execution of the third task into the second request in accordance with the interface format and sends it to the request processing module of the second DPU.

[0121] S710. The request processing module of the second DPU obtains the M second results generated by the first DPU from the cache.

[0122] Specifically, the request processing module of the second DPU obtains the M second results from the cache corresponding to each first DPU, that is, the request processing module of the second DPU obtains the M second results generated by connecting multiple first DPU by multiple first computing nodes.

[0123] S711. The request processing module of the second DPU applies for a cache.

[0124] S712. The request processing module of the second DPU sends a second request to the DOE of the second DPU.

[0125] The second request also carries the M second results obtained from the cache (generated by each first DPU executing M second tasks).

[0126] S713. The DOE of the second DPU executes M third tasks and generates M third results.

[0127] S714. The DOE of the second DPU writes the M third results into the cache.

[0128] S715. The DOE of the second DPU sends a notification message indicating the end of the execution of the M third tasks to the second computing node through the request processing module of the second DPU.

[0129] S716. The second computing node obtains the M third results from the cache.

[0130] As Figure 8 shown, in another implementation manner, the data processing method provided by the embodiments of the present application includes the following steps.

[0131] S801. After the processing module of the first computing node executes multiple first tasks, it sends a notification message to the proxy module of the second computing node to indicate the end of the execution of the multiple first tasks.

[0132] Each first task generates M first results after execution.

[0133] S802. The proxy module of the second computing node sends a fourth request to the proxy module of the first computing node.

[0134] The fourth request is used to instruct the first DPU connected to the first computing node to execute M second tasks.

[0135] In one implementation, the second computing node can obtain the list of computing nodes where the data (M first results) required to execute the second task is located from the metadata server node (which can be implemented by the management node), and send a fourth request to the agent module of the first computing node.

[0136] The fourth request carries the index information of the data and the operator identifier of the operator to be called during the execution of the second task. Among them, the index information of the data is used to indicate the characteristics of the data required for the second task. The index information of the data may include the identifier information of the application (such as APP id), shuffle id, and stage id. Among them, the APP id is used to indicate which application the input data comes from, the shuffle id is used to indicate which shuffle operation of the corresponding application the input data is, and the stage id is used to indicate the task (indicating the second task).

[0137] S803. According to the fourth request, the agent module of the first computing node sends a first request to the request processing module of the first DPU.

[0138] For the description of the first request, reference can be made to the relevant description in the above embodiments.

[0139] The first computing node determines the storage location (such as the physical address) of the M first results generated after the first computing node executes multiple first tasks in the memory through the index information in the mapping table (the mapping table of index information and address).

[0140] S804. After receiving the first request, the request processing module of the first DPU applies for a cache that meets the required size.

[0141] S805. The request processing module of the first DPU sends a first request to the DOE of the first DPU to trigger the DOE of the first DPU to execute M second tasks.

[0142] S806. The DOE of the first DPU executes M second tasks and generates M second results.

[0143] S807. The DOE of the first DPU writes the M second results into the applied cache.

[0144] S808. The first DPU sends a notification message to the agent module of the first computing node through the request processing module of the first DPU. The notification message is used to notify that the execution of M second tasks is completed.

[0145] The proxy module of the first computing node forwards the notification message in S808 to the proxy module of the second computing node.

[0146] S810. The second computing node sends a second request to the request processing module of the second DPU through its processing module and proxy module.

[0147] For the description of the second request, reference can be made to the relevant description in the above embodiments.

[0148] S811. The request processing module of the second DPU obtains M second results generated by the first DPU from the cache.

[0149] S812. The request processing module of the second DPU applies for a cache.

[0150] S813. The request processing module of the second DPU sends a second request to the DOE of the second DPU.

[0151] The second request also carries M second results obtained from the cache (generated by each first DPU executing M second tasks).

[0152] S814. The DOE of the second DPU executes M third tasks and generates M third results.

[0153] S815. The DOE of the second DPU writes the M third results into the cache.

[0154] S816. The DOE of the second DPU sends a notification message indicating the end of the execution of M third tasks to the second computing node through the request processing module of the second DPU.

[0155] S817. The second computing node obtains M third results from the cache.

[0156] In summary, for the data processing methods described in S701 - S716 and S801 - S817, since the first data processing unit connected to each first computing node can first execute the second task on the first results generated after the first computing node executes multiple first tasks to generate second results, and then the second computing node executes the third task to process the second results generated by multiple computing nodes, it is possible to avoid the direct pulling of data generated after a single computing node executes multiple first tasks by the second computing node multiple times and for a large amount of data, thereby solving the problem of large data transmission delay caused by multiple and large amounts of data pulling, being able to reduce the data transmission delay in the data processing process, and improving the efficiency of data processing.

[0157] In combination with the contents of the above embodiments, in an embodiment of the present application, the DPU may include a programmable logic circuit (such as an FPGA), which can support users to update custom operators to the DPU, that is, persist them to the DPU, so that operators can be updated flexibly during the DPU execution of tasks.

[0158] In the embodiment of the present application, the operator update can be achieved by interacting with the DPU connected to the computing node, refer to Figure 9 ,The operator update process includes the following steps.

[0159] S901. The proxy module of the computing node receives a third request to instruct the DPU connected to the computing node to update an operator to be called.

[0160] The third request carries the operator identifier (such as operator ID) and the operator logic microcode of the operator to be updated.

[0161] Optionally, in an embodiment of the present application, the third request may be triggered by a user, and the user may send a request for online update of the hardware operator to the proxy module of the computing node through the configuration interface.

[0162] S902: The proxy module of the computing node sends a third request to the request processing module of the DPU.

[0163] S903: The request processing module of the DPU forwards the third request to the DOE of the DPU.

[0164] S904. The DOE of the DPU stores the operator logic microcode corresponding to the operator identifier.

[0165] After the DPU's DOE receives the third request, it first determines whether the operator indicated by the operator identifier in the third request is a new operator, that is, determines whether the operator logic microcode is a new operator logic microcode. If it is a new operator logic microcode, the DOE will persist the operator logic microcode in the DPU's flash memory.

[0166] S905. The DPU's DOE loads the operator logic microcode from the flash into the DOE's hardware engine pool.

[0167] It can be understood that when DOE receives the first request or the second request in the above embodiment, it loads the operator logic microcode of the operator from the flash into the hardware engine pool of DOE according to the operator identifier carried in the request, thereby executing the corresponding task.

[0168] In summary of the descriptions of S901 - S905, in the embodiments of the present application, the logic circuit of the data processing unit supports hardware operator updates. Therefore, the operators in the data processing unit can be flexibly updated, expanding the range of operators executable by the data processing unit. And the operator to be updated can be a user - defined operator. Supporting user - defined operators can meet the needs of different users, making the application scope of the computing cluster wider.

[0169] The embodiments of the present application further provide a computing node, including a memory and at least one processor connected to the memory. The memory is used to store computer program code, and the computer program code includes computer instructions. When the computer instructions are executed by the at least one processor, the computing node performs the actions executed by the first computing node in the computing cluster; or, performs the actions executed by the second computing node in the computing cluster.

[0170] The embodiments of the present application provide a data processing unit, including a memory and at least one processor connected to the memory. The memory is used to store computer program code, and the computer program code includes computer instructions. When the computer instructions are executed by the at least one processor, the data processing unit performs the actions executed by the first data processing unit in the computing cluster; or, performs the actions executed by the second data processing unit in the computing cluster.

[0171] The embodiments of the present application further provide a computer - readable storage medium storing computer instructions. When the computer instructions run on a computer, they execute the data processing method described in the above embodiments.

[0172] The embodiments of the present application further provide a computer program product. The computer program product contains computer instructions. When the computer instructions run on a computer, they execute the data processing method described in the above embodiments.

[0173] The embodiments of the present application further provide a chip system, including: a processor for calling and running a computer program from a memory, so that a device installed with the chip system executes the data processing method described in the above embodiments.

[0174] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, it can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media integrated. The available medium can be a magnetic medium (such as a floppy disk, magnetic disk, magnetic tape), an optical medium (such as a digital video disc (DVD)), or a semiconductor medium (such as a solid state drive (SSD)), etc.

[0175] Through the description of the above embodiments, those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0176] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0177] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] In addition, each functional unit in various embodiments of the present application may be integrated in a processing unit, may exist separately as individual physical units, or two or more units may be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0179] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk, or optical disk, etc., which can store program codes.

[0180] As described above, the above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A computing cluster, characterized in that, Including: Multiple first computing nodes, which are used to execute N first tasks. The N first tasks are used to process N data shards of the data to be processed. After each first task is executed, M first results are generated and are respectively identified as A0 to A M-1 ; Multiple first data processing units, each first computing node is connected to one first data processing unit; each first data processing unit is configured to execute M second tasks, and each of the M second tasks is used to process the first results with the same identifier in the first computing node connected to the first data processing unit, and generate second results after processing. After executing the M second tasks, M second results are generated, and the M second results are labeled as B0 to B M-1 ; A plurality of second computing nodes, configured to execute M third tasks, each third task being configured to process second results with the same identifier among the plurality of first data processing units to generate third results, and generating M third results after executing the M third tasks.

2. The computing cluster according to claim 1, wherein: The first task is a map task, and the second task and the third task are reduce tasks.

3. The computing cluster according to claim 1 or 2, wherein: The third computing node among the plurality of first computing nodes is further configured to send a first request to the connected first data processing unit to instruct the first data processing unit to execute the M second tasks.

4. The computing cluster according to claim 3, wherein: The first request carries the operator identifier of the operator to be called during the execution of the second task and the storage location information of the M first results.

5. The computing cluster according to any one of claims 1 to 4, characterized in that Further including: A plurality of second data processing units, each second computing node being connected to a second data processing unit; The fourth computing node among the plurality of second computing nodes is further configured to send a second request to the connected second data processing unit to instruct the second data processing unit to execute the M third tasks; the second request carries the operator identifier of the operator to be called during the execution of the third task and the storage location information of the M second results.

6. The computing cluster according to any one of claims 1 to 5, wherein: The third data processing unit among the plurality of first data processing units is further configured to send a notification to the fifth computing node among the plurality of second computing nodes to indicate the end of the execution of the M second tasks.

7. The computing cluster according to any one of claims 1 to 6, wherein: The fourth data processing unit among the plurality of first data processing units is further configured to receive a third request to instruct to update the operator to be called by the fourth data processing unit; the third request carries the operator logic microcode of the operator to be updated; and load the operator logic microcode of the operator to be updated.

8. The computing cluster according to claim 7, wherein: The operator to be updated is a custom operator.

9. A data processing method, characterized in that, Applied to a computing cluster, the computing cluster includes a plurality of first computing nodes, a plurality of first data processing units, and a plurality of second computing nodes, wherein each first computing node is connected to a first data processing unit; the method includes: Multiple first computing nodes execute N first tasks, generate M first results after executing each first task, and are respectively labeled as A0 to A M-1 ; The N first tasks are used to process N data shards of the data to be processed; Each first data processing unit executes M second tasks, and generates M second results after executing the M second tasks. The M second results are identified as B0 to B M-1 ; Each of the M second tasks is used to process the first results with the same identifier in the first computing node connected to the first data processing unit, and generates the second result after processing; A plurality of second computing nodes execute M third tasks, and generate M third results after executing the M third tasks; each third task is configured to process second results with the same identifier among the plurality of first data processing units to generate the third results.

10. The method according to claim 9, wherein: The first task is a map task, and the second task and the third task are reduce tasks.

11. The method according to claim 9 or 10, characterized in that, The method further includes: The third computing node among the plurality of first computing nodes sends a first request to the connected first data processing unit to instruct the first data processing unit to execute the M second tasks.

12. The method according to claim 11, wherein: the operator identifier of the operator to be called during the execution of the second task and the storage location information of the M first results are carried in the first request.

13. The method according to any one of claims 9 to 12, characterized in that The computing cluster further includes: a plurality of second data processing units, each second computing node is connected to a second data processing unit; the method further includes: a fourth computing node among the plurality of second computing nodes sends a second request to the connected second data processing unit to instruct the second data processing unit to execute the M third tasks; the operator identifier of the operator to be called during the execution of the third task and the storage location information of the M second results are carried in the second request.

14. The method according to any one of claims 9 to 13, characterized in that, The method further includes: a third data processing unit among the plurality of first data processing units sends a notification to a fifth computing node among the plurality of second computing nodes to indicate the end of the execution of the M second tasks.

15. The method according to any one of claims 9 to 14, characterized in that The method further includes: a fourth data processing unit among the plurality of first data processing units receives a third request to instruct to update the operator to be called by the fourth data processing unit, and the operator logic microcode of the operator to be updated is carried in the third request. The fourth data processing unit loads the operator logic microcode of the operator to be updated.

16. The method according to claim 15, wherein: the operator to be updated is a custom operator.

17. A data processing device, characterized in that, Comprising a memory and at least one processor connected to the memory, the memory is used to store computer program code, the computer program code includes computer instructions, when the computer instructions are executed by at least one processor, the data processing device executes the method according to any one of claims 9 to 16.

18. A computer-readable storage medium, characterized in that, Stored with computer instructions, when the computer instructions are run on a computer, the method according to any one of claims 9 to 16 is executed.

19. A computer program product, characterized in that, Containing instructions, when the instructions are run on a computing device, the computing device is caused to execute the method according to any one of claims 9 to 16.

Images