A distributed task processing method, a distributed system and a first device

Abstract

The embodiments of this specification provide a method for processing a distributed task, a distributed system, and a first device, wherein the distributed task includes at least two subtasks respectively executed by at least two devices in the distributed system, and the at least two devices in the distributed system include the first device. The processor of the first device stores the result data of the executed first subtask in the memory, thereby reducing the interaction with the disk during the data calculation stage. At the same time, since the result data is stored in the memory, the network card of the first device can directly transmit the result data in the memory to the network card of the second device through the network, thereby reducing the interaction with the disk and the consumption of computing resources during the data exchange stage. Since the consumption of computing resources is reduced in both the data calculation stage and the data exchange stage, the execution time of the distributed task is shortened, which is beneficial to the execution of distributed tasks with greater real-time requirements.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of big data technology, and in particular, to a distributed task processing method, a distributed system, and a first device. Background Art

[0002] With the rapid development of Internet technology, the extensive interconnection between intelligent machines and humans, and between machines, has generated massive amounts of big data. Faced with massive amounts of big data, it is necessary to jointly maintain the massive amounts of data through a distributed system. A distributed system includes multiple nodes, each of which maintains a portion of the entire data. In a distributed system, when the execution of a task requires the use of data stored in different nodes, a task can be divided into multiple subtasks. Each subtask is scheduled to be executed in the node that stores the required data. Such a task that is collaboratively executed by multiple nodes can be called a distributed task.

[0003] In the process of executing distributed tasks, since each node is only responsible for the operation of a part of the data, data exchange (shuffle) between nodes is an indispensable process. However, in the related technology, the computer resources occupied by the data exchange process are relatively large. How to reduce the computing resources consumed by the data exchange process is a technical problem to be solved in this field. Summary of the invention

[0004] The embodiments of this specification provide a distributed task processing method, a distributed system, and a first device to reduce computing resources consumed during data exchange and improve the execution efficiency of distributed tasks.

[0005] According to a first aspect of an embodiment of the present specification, a method for processing a distributed task is provided, wherein the distributed task includes at least two subtasks respectively executed by at least two devices in a distributed system; the at least two devices in the distributed system include a first device; the method includes:

[0006] The processor of the first device reads the data in the memory of the first device to execute the first subtask corresponding to the first device, and after obtaining the result data of the first subtask, stores the result data in the memory of the first device;

[0007] The network card of the first device transmits the result data of the first subtask in the memory of the first device to the network card of the second device of the distributed system through the network, so that the network card of the second device writes the result data of the first subtask into the memory of the second device.

[0008] In some examples, the network card of the second device writes the result data of the first subtask into the memory of the second device, including:

[0009] The processor of the second device reads the result data in the memory of the second device to execute the second subtask corresponding to the second device; and / or

[0010] The network card of the second device transmits the result data in the memory of the second device to the network card of the third device of the distributed system through the network, so that the third device uses the result data to execute the third subtask corresponding to the third device.

[0011] In some examples, after the network card of the second device writes the result data of the first subtask into the memory of the second device, the method further includes:

[0012] The processor of the second device sequentially writes the result data in the memory of the second device to the disk of the second device.

[0013] In some examples, after the processor of the second device sequentially writes the result data in the memory of the second device to the disk of the second device, the method further includes:

[0014] The processor of the second device deletes the result data in the memory of the second device;

[0015] After the processor of the second device receives the result data sending instruction, it sequentially reads the result data from the disk of the second device into the memory of the second device, so that the network card of the second device transmits the result data in the memory of the second device to the network card of the third device of the distributed system through the network.

[0016] In some examples, the storage area of ​​the memory of the first device includes a sub-area for storing application data; the result data of the first subtask is stored in the sub-area of ​​the memory, and the network card of the first device transmits the result data of the first subtask in the memory of the first device to the network card of the second device of the distributed system through the network, including:

[0017] The network card of the first device transmits the result data in the sub-area of ​​the memory to the network card of the second device through remote direct access technology.

[0018] According to a second aspect of an embodiment of the present specification, there is provided a distributed system, the distributed system being used to execute a distributed task and comprising at least two devices, the at least two devices comprising a first device; the distributed task comprising at least two subtasks respectively executed by the at least two devices;

[0019] The processor of the first device is used to read the data in the memory of the first device to execute the first subtask corresponding to the first device, and store the result data of the first subtask in the memory of the first device after obtaining the result data;

[0020] The network card of the first device is used to transmit the result data of the first subtask in the memory of the first device to the network card of the second device of the distributed system through the network;

[0021] The network card of the second device is used to receive the result data of the first subtask and write the result data of the first subtask into the memory of the second device.

[0022] In some examples, the processor of the second device is used to read the result data in the memory of the second device to execute the second subtask corresponding to the second device; and / or

[0023] The network card of the second device is used to transmit the result data in the memory of the second device to the network card of the third device of the distributed system through the network, so that the third device uses the result data to execute the third subtask corresponding to the third device.

[0024] In some examples, the processor of the second device is further configured to sequentially write the result data in the memory of the second device to the disk of the second device.

[0025] In some examples, the processor of the second device is also used to delete the result data in the memory of the second device; and after receiving an instruction to send result data, sequentially read the result data from the disk of the second device into the memory of the second device, so that the network card of the second device transmits the result data in the memory of the second device to the network card of a third device of the distributed system through the network.

[0026] In some examples, the storage area of ​​the memory of the first device includes a sub-area for storing application data; the result data of the first subtask is stored in the sub-area of ​​the memory,

[0027] The network card of the first device is further used to transmit the result data in the sub-area of ​​the memory to the network card of the second device through remote direct access technology.

[0028] According to a third aspect of an embodiment of the present specification, a first device of a distributed system is provided, the distributed system is used to execute a distributed task, and includes at least two devices, the at least two devices include a first device; the distributed task includes at least two subtasks respectively executed by the at least two devices; the first device includes:

[0029] processor;

[0030] a memory for storing processor-executable instructions;

[0031] Network card;

[0032] Memory;

[0033] The processor is configured to: read data in the memory of the first device to execute the first subtask corresponding to the first device, and store the result data of the first subtask in the memory of the first device after obtaining the result data;

[0034] The network card is configured to: transmit the result data of the first subtask in the memory of the first device to the network card of the second device of the distributed system through the network; so that the network card of the second device writes the result data of the first subtask into the memory of the second device.

[0035] According to a fourth aspect of the embodiments of the present specification, there is provided a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method as described in any example of the first aspect are implemented.

[0036] According to a fifth aspect of the embodiments of the present specification, a computer-readable storage medium is provided, on which a number of computer instructions are stored. When the computer instructions are executed, the method described in any example of the first aspect is performed.

[0037] The technical solutions provided by the embodiments of the present specification may have the following beneficial effects:

[0038] The embodiments of this specification provide a method for processing a distributed task, a distributed system, and a first device, wherein the distributed task includes at least two subtasks respectively executed by at least two devices in the distributed system, and the at least two devices in the distributed system include the first device. The processor of the first device stores the result data of the executed first subtask in the memory, which reduces the interaction with the disk during the data calculation stage. At the same time, since the result data is stored in the memory, the network card of the first device can directly transmit the result data in the memory to the network card of the second device through the network, which also reduces the interaction with the disk and the consumption of computing resources during the data exchange stage. The above method reduces the consumption of computing resources in both the data calculation stage and the data exchange stage, shortens the execution time of the distributed task, and is conducive to the execution of distributed tasks with greater real-time requirements.

[0039] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the embodiments of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings herein are incorporated in the specification and constitute a part of the embodiments of the specification, illustrate embodiments consistent with the embodiments of the specification, and together with the description, are used to explain the principles of the embodiments of the specification.

[0041] Figure 1 It is a schematic diagram of a distributed system shown in an embodiment of this specification according to an embodiment.

[0042] Figure 2 This is a flowchart of a distributed task processing method shown in an embodiment of this specification according to an embodiment.

[0043] Figure 3 It is a schematic diagram of the Spark architecture shown in an embodiment of this specification according to an embodiment.

[0044] Figure 4 This is a flowchart of a distributed task processing method shown in an embodiment of this specification according to another embodiment.

[0045] FIG5( a ) is a schematic diagram of the structure of a distributed system according to an embodiment of the present specification.

[0046] FIG5( b ) is a schematic diagram of the structure of a distributed system according to another embodiment of the present specification.

[0047] Figure 6 It is a hardware structure diagram of a first device of a distributed system shown in an embodiment of this specification according to an embodiment. DETAILED DESCRIPTION

[0048] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of the embodiments of this specification as detailed in the appended claims.

[0049] The terms used in the embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of this specification. The singular forms of "a", "said" and "the" used in the embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items.

[0050] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0051] With the rapid development of Internet technology, the extensive interconnection between intelligent machines and humans, and between machines, has generated massive amounts of big data. Faced with massive amounts of big data, it is necessary to jointly maintain the massive amounts of data through distributed systems. Distributed systems can be implemented based on machine clusters. For example, Figure 1 A schematic diagram of a distributed system is shown. The distributed system 100 may include multiple nodes, such as nodes 110-140 shown in the figure. Each node may store a portion of data and maintain the stored data. On the one hand, in a distributed system, when the execution of a task requires the use of data stored in different nodes, a task may be divided into multiple subtasks. Each subtask is dispatched to the node storing the required data for execution. On the other hand, when executing tasks such as data storage, management, and analysis, since the complexity of the task greatly exceeds the processing capacity of a single device, the complex task may be divided into multiple subtasks and assigned to multiple nodes for execution. The above-mentioned tasks executed by multiple nodes in collaboration may be called distributed tasks. The subtasks divided by the distributed task may be multiple subtasks processed synchronously and in parallel; or one part of the subtasks may be executed based on the execution results of another part of the subtasks, that is, there is a sequential execution order between the subtasks, which are multiple subtasks processed asynchronously. Different nodes may be different in hardware configuration, for example, different nodes have different computing capabilities or data transmission and reception capabilities. The subtasks may be sent to different nodes for processing according to the attributes of each subtask and / or the data required to execute each subtask.

[0052] In the process of multiple nodes collaboratively executing distributed tasks, since each node is only responsible for the operation of a part of the data, data exchange (shuffle) between nodes is an indispensable process. In the related art, when the processor of a node executes the assigned subtask, the intermediate data of the subtask is read and written in the memory and disk many times. After the processor obtains the result data corresponding to the subtask, it can store the result data from the memory to a storage device such as a disk. In the data exchange stage, the processor of the node reads the result data from the disk into the memory through random read / write (Input / Output, I / O), and then sends the result data to the node of the next subtask through a network protocol such as the TCP / IP protocol.

[0053] However, random I / O of the disk during the data exchange phase consumes a lot of disk performance, especially the input / output operations per second (IOPS) of the disk. Data exchange occupies a lot of computing resources. How to reduce the computing resources consumed during data exchange is a technical problem that needs to be solved urgently in this field.

[0054] To this end, the embodiment of this specification proposes a method for processing a distributed task, wherein the distributed task is executed by a distributed system. The distributed system includes at least two devices, and the distributed task includes at least two subtasks. The at least two subtasks are respectively executed by at least two devices included in the distributed system. Among them, the device for executing the subtask includes at least a first device. The above method includes the following steps: Figure 2 The steps described:

[0055] Step 210: The processor of the first device reads data in the memory of the first device to execute a first subtask corresponding to the first device, and after obtaining result data of the first subtask, stores the result data in the memory of the first device;

[0056] Step 220: The network card of the first device transmits the result data of the first subtask in the memory of the first device to the network card of the second device of the distributed system through the network, so that the network card of the second device writes the result data of the first subtask into the memory of the second device.

[0057] Step 210 and step 220 may be executed by different execution entities, for example, step 210 may be executed by a processor of the first device, and step 220 may be executed by a network card of the first device.

[0058] Distributed systems can be Figure 1In the distributed system 100 shown, the first device and the second device may be any of the nodes 110-140. The distributed system may be equipped with a big data storage computing architecture based on memory computing, such as the Spark computing architecture (hereinafter referred to as Spark). Taking Spark as an example, Figure 3 The Spark architecture is shown in FIG. 3. The Spark architecture includes a driver Spark Driver 310, a cluster resource manager Cluster Manager 320, and one or more worker nodes Worker Node 330. Figure 3 Take two as an example. Worker Node 330 includes Executor 331. Spark Driver 310 can be installed in the nodes of the distributed system, which is the execution entry of the Spark program and is used to build a directed acyclic graph (DAG), apply for cluster resources, create accumulators, and broadcast variables; some nodes in the distributed system can be used as Cluster Manager 320 to provide external services of computing resources for the program; another part of the nodes in the distributed system can be used as Worker Node 330, which is the working node in the cluster and is responsible for task computing; Executor 331 is a process in WorkerNode 330, which is used to manage task computing on one or more CPU threads.

[0059] Generally speaking, the processing of data by a node may include data calculation and data exchange. Data calculation is to use the data stored in this node to execute the scheduled subtask and obtain the result data corresponding to the subtask. Data exchange is to transmit the result data of the subtask to other nodes. For a distributed system equipped with a memory computing architecture such as Spark, during the data calculation process, that is, when executing the first subtask, the first device can store the result data of the first subtask in memory based on memory computing instead of on disk. Compared with the disk-based computing architecture, the memory-based computing architecture has higher throughput and lower access latency because it reduces the interaction with the disk during the calculation process, that is, it reduces the interaction with the disk from the data calculation stage and saves computing resources.

[0060] Subsequently, the network card of the first device transmits the result data stored in the memory to the network card of the second device through the network, so that the network card of the second device writes the result data into the memory of the second device, thereby completing the data exchange process between the first device and the second device.

[0061] The embodiment of this specification provides a method for processing a distributed task. On the one hand, the first device stores the result data of the first subtask in the memory during the data calculation phase, thereby reducing the interaction with the disk. On the other hand, since the result data of the first subtask is stored in the memory during the calculation phase, the network card of the first device can directly transmit the result data stored in the memory to the network card of the second device through the network during the data exchange phase, so that the network card of the second device writes the result data into the memory of the second device. In both the calculation phase and the data exchange phase, the interaction with the disk and the consumption of computing resources are reduced. Therefore, the execution time of the distributed task is shortened, which is conducive to the execution of distributed tasks with greater real-time requirements.

[0062] The processor of the first device reads the required data from the memory of the first device when executing the first subtask. In some embodiments, the required data may be stored on a disk or other storage device before being loaded into the memory of the first device.

[0063] In some embodiments, the first device and the second device may be nodes in a distributed system for executing subtasks included in a distributed task. Thus, after the network card of the second device writes the result data of the first subtask into the memory of the second device, the processor of the second device may read the result data of the first subtask in the memory of the second device to execute the second subtask corresponding to the second device.

[0064] In some embodiments, to prevent data loss, the processor of the second device may sequentially write the result data of the first subtask to the disk of the second device while the network card of the second device writes the result data of the first subtask to the memory of the second device.

[0065] In some embodiments, after the result data of the first subtask are written into the memory and hard disk of the second device, if the second subtask does not meet the execution condition, the processor of the second device may delete the result data of the first subtask in the memory of the second device. And when the second subtask meets the execution condition, the processor of the second device sequentially reads the result data of the first subtask from the disk to the memory of the second device to execute the second subtask. The execution condition may include but is not limited to: reaching the execution time and / or the second device stores all the data required to execute the task.

[0066] In the case where both the first device and the second device are nodes for executing subtasks, the data calculation stage and the data exchange stage are completed in the first device. However, the data calculation stage and the data exchange stage have different requirements for the hardware configuration of the node. For example, nodes that perform data calculations will have higher requirements on computing power; nodes that perform data exchange will have higher requirements on data receiving and sending capabilities. If the first device is responsible for data calculation and data exchange at the same time, it will have higher requirements on the computing power and data receiving and sending capabilities of the first device, which will bring a certain burden to the first device. As such, in some embodiments, the Remote Shuffle Service (RSS) can be used to achieve the decoupling of data calculation and data exchange. Figure 4 As shown, the first device can be a node in the distributed system for executing subtasks included in the distributed task; the second device can be a server for storing result data of the subtask, such as an RSS server. In addition to storing the result data of the first subtask, the second device can also store result data of subtasks executed by other nodes in the distributed system. The distributed system also includes a third device, which can be a node for executing subtasks included in the distributed task. In this way, a distributed task processing method of this embodiment can include the following: Figure 4 Steps shown:

[0067] Step 411: The processor of the first device reads data required to execute the first subtask from the memory of the first device;

[0068] Step 412: The processor of the first device executes the first subtask and obtains result data of the first subtask;

[0069] Step 413: The processor of the first device stores the result data of the first subtask in the memory of the first device;

[0070] Step 414: The network card of the first device reads the result data of the first subtask from the memory of the first device;

[0071] Step 415: The network card of the first device transmits the result data of the first subtask to the network card of the RSS server through the network;

[0072] Step 421: the network card of the RSS server writes the result data of the first subtask into the memory of the RSS server;

[0073] Step 422: the network card of the RSS server reads the result data of the first subtask from the memory of the RSS server;

[0074] Step 423: The network card of the RSS server transmits the result data of the first subtask to the network card of the third device through the network;

[0075] Step 431: The network card of the third device writes the result data of the first subtask into the memory of the third device;

[0076] Step 432: The processor of the third device reads the result data of the first subtask from the memory of the third device;

[0077] Step 433: The processor of the third device uses the result data of the first subtask to execute the third subtask corresponding to the third device.

[0078] In some embodiments, the RSS server may execute step 422 after receiving the result data sending instruction of the first subtask.

[0079] In some embodiments, in order to prevent data loss, while the RSS server is executing step 421 , the processor of the RSS server may also sequentially write the result data of the first subtask into the disk of the RSS server.

[0080] In some embodiments, after the result data of the first subtask are written into the memory and hard disk of the RSS server, if the RSS server does not receive the instruction to send the result data of the first subtask, the processor of the RSS server may delete the result data of the first subtask in the memory of the RSS server. When receiving the instruction to send the result data of the first subtask, the processor of the RSS server sequentially reads the result data of the first subtask from the disk into the memory of the RSS server, and then the RSS server executes step 422.

[0081] In some embodiments, after executing the third subtask, the third device can send the result data of the third subtask to the RSS server for storage so as to be called by other nodes in the distributed system. The sending process of the result data of the third subtask can refer to the above embodiment, and this specification embodiment will not be repeated here.

[0082] In this way, after the first device completes the execution of the first subtask, the result data of the first subtask is stored in the RSS server. When the third device needs to use the result data of the first subtask, the third device can request the result data from the RSS server without having to interact with the first device. For the first device, when the first device executes multiple subtasks, the first device can send the result data of multiple subtasks to the RSS server, and use the RSS server to send the result data of multiple subtasks to the next node respectively. Therefore, the first device does not need to exchange data with multiple nodes, which greatly reduces the amount of data sent and received by the first device, thereby decoupling data calculation from data exchange. Since the first device only needs to undertake data calculation work, the hardware configuration of the first device can pay more attention to computing power; while the second device mainly undertakes data exchange, and the hardware configuration can pay more attention to data sending and receiving capabilities.

[0083] As described above, in the traditional data exchange process, the result data is usually sent to the next node through a network protocol such as the TCP / IP protocol. The storage area of ​​the memory may include a sub-area for storing application data, also known as user-state memory, or user-space memory; and also include a sub-area for storing operating system data, also known as kernel-state memory, or kernel-space memory. In the traditional TCP / IP technology, the data sending device needs to first read the data to be transmitted from the disk into the user-state memory, and then the CPU of the data sending device copies the data to be transmitted to the kernel-state memory, and then the network card copies the data to be transmitted in the kernel-state memory to its own buffer, and then sends it to the data receiving device through a physical link after processing. Multiple copies of the data to be transmitted rely on CPU execution, which consumes a lot of CPU. To this end, in some embodiments, the process of transmitting the result data of the first subtask from the first device to the second device may include that the network card of the first device transmits the result data in the user-state memory to the network card of the second device through remote direct access technology (Remote Direct Memory Access, RDMA). Accordingly, the network cards of the first device and the second device may be RDMA network cards. RDMA technology is a new direct memory access technology. With RDMA technology, the network card of the data sending device can directly copy the data to be transmitted in the user-state memory to its own buffer. After assembling the data to be transmitted into messages at each layer, it is sent to the network card of the data receiving device through a physical link. After the network card of the data receiving device receives the data, it strips off the message headers and checksums at each layer, and then directly copies the received data to the user-state memory. Therefore, RDMA technology can directly access data from the memory of one device to the memory of another device, bypassing the copy of kernel-state memory, system calls, and CPU context switching, thereby saving the overhead of the TPC / IP protocol. Compared with traditional TCP / IP technology, RDMA technology greatly reduces CPU consumption and shortens transmission delay during data transmission.

[0084] In this embodiment, the second device may be the RSS server mentioned above, and the distributed system further includes a third device, which may be a node for executing the subtasks included in the distributed task. In this embodiment, the method for distributed task processing may include the above Figure 4Steps. Thus, in this embodiment, the network card of the first device can transmit the result data in the user-state memory to the network card of the RSS server through the RDMA technology. The network card of the RSS server can write the result data of the first subtask into the memory of the RSS server. Subsequently, after receiving the result data sending instruction, the network card of the RSS server can read the result data of the first subtask from the memory of the RSS server, and transmit it to the network card of the third device through the RDMA technology. The network card of the third device writes the result data of the first subtask into the memory of the third device. In the field of distributed computing, many memory-based computing frameworks such as Spark, during the data exchange process, seriously consume CPU, memory, disk, and network resources because there is still a process of data interaction between memory and disk, resulting in resource waste. In this embodiment, the RSS technology is used to pull the data exchange process to the remote end (RSS server), and combined with the RDMA technology to solve the problem of resource consumption during the data exchange process.

[0085] This embodiment provides a method for processing distributed tasks, in which the network card of the first device can directly read the intermediate result data from the user-state memory, and the data transmission bypasses the kernel (kernel bypass) to achieve zero copy. The network card sends the intermediate result data to the network card of the second device based on RDMA technology. After the network card of the second device receives the intermediate result data, it can directly write the data into the user-state memory of the second device. On the one hand, since the intermediate result data is read directly from the memory, the interaction with the disk is reduced; on the other hand, the intermediate result data can be directly transmitted from the user-state memory to the network card without being processed by the CPU, thereby reducing the CPU consumption during the data exchange process and saving computing resources.

[0086] Based on the distributed task processing method described in any of the above embodiments, the present specification also provides a distributed system for executing a distributed task. The distributed task includes at least two subtasks. The at least two subtasks are respectively executed by at least two devices included in the distributed system. Figure 5(a)-Figure 5(b) As shown, the distributed system 500 includes at least a first device 510 for performing the above subtasks, and also includes a second device 520.

[0087] The processor of the first device 510 is used to read the data in the memory of the first device 510 to execute the first subtask corresponding to the first device 510, and store the result data of the first subtask in the memory of the first device 510 after obtaining the result data;

[0088] The network card of the first device 510 is used to transmit the result data of the first subtask in the memory of the first device 510 to the network card of the second device 520 of the distributed system 500 through the network;

[0089] The network card of the second device 520 is used to receive the result data of the first subtask and write the result data of the first subtask into the memory of the second device 520 .

[0090] In some embodiments, the processor of the second device 520 is used to read the result data in the memory of the second device 520 to execute the second subtask corresponding to the second device 520.

[0091] In some embodiments, as shown in FIG5(b), the distributed system 500 further includes a third device 530. The network card of the second device 520 is used to transmit the result data in the memory of the second device 520 to the network card of the third device 530 of the distributed system through the network, so that the third device 530 uses the result data to perform the third subtask corresponding to the third device 530.

[0092] In some embodiments, the processor of the second device 520 is further used to sequentially write the result data in the memory of the second device 520 to the disk of the second device 520 .

[0093] In some embodiments, the processor of the second device 520 is also used to delete the result data in the memory of the second device 520; and after receiving the result data sending instruction, sequentially read the result data from the disk of the second device 520 to the memory of the second device 520, so that the network card of the second device 520 transmits the result data in the memory of the second device 520 to the network card of the third device 530 of the distributed system through the network.

[0094] In some embodiments, the storage area of ​​the memory of the first device 510 includes a sub-area for storing application data; the result data of the first subtask is stored in the sub-area of ​​the memory, and the network card of the first device 510 is also used to transmit the result data in the sub-area of ​​the memory to the network card of the second device 520 through remote direct access technology.

[0095] Based on a distributed task processing method described in any of the above embodiments, the embodiments of this specification also provide Figure 6 The structure diagram of the first device of a distributed system is shown in FIG. The distributed system is used to execute a distributed task and includes at least two devices, the at least two devices including the first device; the distributed task includes at least two subtasks executed by the at least two devices respectively. Figure 6At the hardware level, the first device includes a processor, an internal bus, a network card, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it. The processor is configured to: read the data in the memory of the first device to execute the first subtask corresponding to the first device, and store the result data of the first subtask in the memory of the first device after obtaining it. The network card is configured to: transmit the result data of the first subtask in the memory of the first device to the network card of the second device of the distributed system through the network; so that the network card of the second device writes the result data of the first subtask into the memory of the second device.

[0096] Based on a distributed task processing method described in any of the above embodiments, an embodiment of this specification further provides a computer program product, including a computer program, which can be used to execute the distributed task processing method described in any of the above embodiments when executed by a processor.

[0097] Based on a distributed task processing method described in any of the above embodiments, an embodiment of this specification further provides a computer storage medium, which stores a computer program. When the computer program is executed by a processor, it can be used to execute a distributed task processing method described in any of the above embodiments.

[0098] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0099] Those skilled in the art will readily appreciate other implementations of the embodiments of the present specification after considering the specification and practicing the inventions claimed herein. The embodiments of the present specification are intended to cover any variations, uses, or adaptations of the embodiments of the present specification, which follow the general principles of the embodiments of the present specification and include common knowledge or customary techniques in the art that are not claimed in the embodiments of the present specification. The description and embodiments are to be considered merely as exemplary, and the true scope and spirit of the embodiments of the present specification are indicated by the claims below.

Claims

1. A method for processing a distributed task, wherein the distributed task includes at least two subtasks respectively executed by at least two devices in a distributed system; The at least two devices of the distributed system include a first device; the method includes: The processor of the first device reads the data in the memory of the first device to execute the first subtask corresponding to the first device, and after obtaining the result data of the first subtask, stores the result data in the memory of the first device; The network card of the first device transmits the result data of the first subtask in the memory of the first device to the network card of the second device of the distributed system through the network, so that the network card of the second device writes the result data of the first subtask into the memory of the second device; wherein the second device is a server for storing the result data of the subtasks included in the distributed task; The network card of the second device transmits the result data in the memory of the second device to the network card of a third device in the distributed system through the network, so that the network card of the third device writes the result data into the memory of the third device; The processor of the third device reads the result data from the memory of the third device to execute the third subtask corresponding to the third device.

2. The method according to claim 1, wherein the network card of the second device writes the result data of the first subtask into the memory of the second device, comprising: The processor of the second device reads the result data in the memory of the second device to execute the second subtask corresponding to the second device.

3. The method according to claim 2, after the network card of the second device writes the result data of the first subtask into the memory of the second device, further comprising: The processor of the second device sequentially writes the result data in the memory of the second device to the disk of the second device.

4. The method according to claim 3, after the processor of the second device sequentially writes the result data in the memory of the second device to the disk of the second device, further comprising: The processor of the second device deletes the result data in the memory of the second device; After the processor of the second device receives the result data sending instruction, it sequentially reads the result data from the disk of the second device into the memory of the second device, so that the network card of the second device transmits the result data in the memory of the second device to the network card of the third device of the distributed system through the network.

5. The method according to claim 1, wherein the storage area of ​​the memory of the first device includes a sub-area for storing application data; the result data of the first subtask is stored in the sub-area of ​​the memory, and the network card of the first device transmits the result data of the first subtask in the memory of the first device to the network card of the second device of the distributed system through the network, comprising: The network card of the first device transmits the result data in the sub-area of ​​the memory to the network card of the second device through remote direct access technology.

6. A distributed system, the distributed system being used to execute a distributed task, and comprising at least two devices, the at least two devices comprising a first device; the distributed task comprising at least two subtasks respectively executed by the at least two devices; The processor of the first device is used to read the data in the memory of the first device to execute the first subtask corresponding to the first device, and store the result data of the first subtask in the memory of the first device after obtaining the result data; The network card of the first device is used to transmit the result data of the first subtask in the memory of the first device to the network card of the second device of the distributed system through the network; wherein the second device is a server for storing the result data of the subtasks included in the distributed task; The network card of the second device is used to receive the result data of the first subtask and write the result data of the first subtask into the memory of the second device; and transmit the result data in the memory of the second device to the network card of the third device of the distributed system through the network; The network card of the third device is used to write the result data into the memory of the third device; The processor of the third device is used to read the result data from the memory of the third device to execute the third subtask corresponding to the third device.

7. According to the system of claim 6, the processor of the second device is used to read the result data in the memory of the second device to execute the second subtask corresponding to the second device.

8. According to the system of claim 7, the processor of the second device is further used to sequentially write the result data in the memory of the second device to the disk of the second device.

9. According to the system of claim 8, the processor of the second device is also used to delete the result data in the memory of the second device; and after receiving the result data sending instruction, sequentially read the result data from the disk of the second device into the memory of the second device, so that the network card of the second device transmits the result data in the memory of the second device to the network card of the third device of the distributed system through the network.

10. The system according to claim 6, wherein the storage area of ​​the memory of the first device includes a sub-area for storing application data; the result data of the first subtask is stored in the sub-area of ​​the memory, The network card of the first device is further used to transmit the result data in the sub-area of ​​the memory to the network card of the second device through remote direct access technology.

11. A first device of a distributed system, the distributed system being used to execute a distributed task and comprising at least two devices, the at least two devices comprising a first device; the distributed task comprising at least two subtasks respectively executed by the at least two devices; the first device comprising: processor; a memory for storing processor-executable instructions; Network card; Memory; The processor is configured to: read data in the memory of the first device to execute the first subtask corresponding to the first device, and store the result data of the first subtask in the memory of the first device after obtaining the result data; The network card is configured to: transmit the result data of the first subtask in the memory of the first device to the network card of the second device of the distributed system through the network; so that the network card of the second device writes the result data of the first subtask into the memory of the second device, and transmit the result data in the memory of the second device to the network card of the third device of the distributed system through the network, so that the processor of the third device uses the result data to execute the third subtask corresponding to the third device.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

13. A computer-readable storage medium, wherein a plurality of computer instructions are stored on the computer-readable storage medium, and when the computer instructions are executed, the method according to any one of claims 1 to 5 is performed.

Images