Task execution method, apparatus, computing cluster, storage medium and program product

Abstract

The present application belongs to the technical field of computers. Disclosed are a task execution method, an apparatus, a computing cluster, a storage medium and a program product. A computing model is distributed on multiple nodes for operation. The method comprises: during execution of a first transaction, a first node synchronously executes pre-processing of at least one first communication task that needs to be executed subsequently, the first transaction being a second communication task or a first computing task, the at least one first communication task referring to a communication task that needs to be executed after the first transaction, and the communication task being used for transferring operation data of the computing model between nodes; and if the execution of pre-processing of a reference communication task in the at least one first communication task is completed, executing the reference communication task on the basis of an execution result of the pre-processing of the reference communication task, the reference communication task being any first communication task. Thus, the method can avoid additional time spent on pre-processing, thus starting first communication tasks more quickly, and improving the communication performance.

Description

Task execution methods, devices, computing clusters, storage media, and program products

[0001] This application claims priority to Chinese Patent Application No. 202410814526.7, filed on June 21, 2024, entitled “Task Execution Method, Apparatus, Computing Cluster, Storage Medium and Program Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to a task execution method, apparatus, computing cluster, storage medium, and program product. Background Technology

[0003] As the industry deepens its research into computational models such as artificial intelligence (AI) models and improves its research level, the number of parameters in these models has rapidly increased to the levels of billions, tens of billions, or even trillions, resulting in a rapid increase in computational demands. However, the iteration speed of computing hardware lags far behind the speed of software updates, making it a key research issue for the industry to address how to support computational models with a large number of parameters under limited hardware resources.

[0004] Taking AI models as an example, the main methods currently used in the industry to solve this problem include data parallelism, pipeline parallelism, model parallelism, and sequence parallelism. These methods distribute an AI model across multiple nodes for execution. However, the data related to the operation of an AI model is usually globally correlated. Therefore, when distributing an AI model across multiple nodes, communication between these nodes is necessary to achieve data exchange. Improving communication performance in this situation, thereby enhancing the overall performance of the AI ​​model, is of paramount importance. Summary of the Invention

[0005] This application provides a task execution method, apparatus, computing cluster, storage medium, and program product, which can improve the communication performance of data interaction between nodes when the computing model is distributed across multiple nodes, thereby improving the overall operating performance of the computing model. The technical solution is as follows:

[0006] Firstly, a task execution method is provided, wherein the computational model is distributed across multiple nodes, and the method is applied to the first node among these multiple nodes. The method includes:

[0007] During the execution of the first transaction, preprocessing of at least one first communication task is performed. The preprocessing is used to prepare for the execution of the corresponding communication task. The first transaction is either a second communication task or a first computation task of the above-mentioned computation model. The at least one first communication task refers to a communication task that needs to be executed after the first transaction. The communication task is used to transmit the running data of the computation model between all or some of the above-mentioned nodes. If the preprocessing of the reference communication task in the at least one first communication task is completed, the reference communication task is executed based on the execution result of the preprocessing of the reference communication task. The reference communication task is any one of the first communication tasks.

[0008] In other words, during the execution of the first transaction, the preprocessing of the first communication task that needs to be executed later is performed simultaneously to prepare for the execution of the first communication task in advance, avoiding the extra time spent on preprocessing, making it easier to start the first communication task faster, thereby improving the overall operating efficiency of the computing model.

[0009] In one possible implementation, the first node includes a first main execution unit and a first communication unit, and the method further includes: during the execution of the first transaction, the first main execution unit sends a pre-processing instruction message for the at least one first communication task to the first communication unit;

[0010] The first node performs preprocessing for the at least one first communication task, including: the first communication unit performs preprocessing for the at least one first communication task in response to the preprocessing instruction message for the at least one first communication task.

[0011] That is, the first main execution unit has the function of triggering the first communication unit to perform preprocessing through instructions, and correspondingly, the first communication unit performs the corresponding preprocessing under the instruction of the first main execution unit.

[0012] In one possible implementation, the at least one first communication task includes multiple first communication tasks, and the preprocessing indication message for the at least one first communication task includes multiple first preprocessing indication messages, each corresponding one-to-one with the multiple first communication tasks. The multiple first preprocessing indication messages are carried in the same instruction, or they are carried in multiple instructions, which are sent in multiple installments. That is, one instruction can trigger the preprocessing of at least one first communication task.

[0013] The aforementioned multiple first communication tasks are executed in the order of task execution. If the multiple first preprocessing instruction messages are carried in multiple instructions, these multiple instructions are sent in any of the multiple sending methods, including sending in the order of task execution, sending in any order, and sending in parallel. That is, these multiple instructions can be sent sequentially, out of order, or in parallel.

[0014] In one possible implementation, considering that the preprocessing of each communication task typically takes relatively little time, while the first transaction takes relatively more time, the preprocessing of the aforementioned multiple first communication tasks can be executed in any of a variety of execution methods. These methods include executing the multiple first communication tasks in the order they are executed, executing them in any order, and executing them in parallel. That is, the preprocessing of these multiple first communication tasks can be executed sequentially, out of order, or in parallel.

[0015] The aforementioned first communication task refers to a communication task that needs to be executed after the first transaction; that is, the first communication task has a data dependency relationship with the first transaction. Since the first transaction is either a second communication task or a first computation task, and the data dependency relationship between the first transaction and the first communication task is also different when the first transaction is different, the following will introduce several cases of data dependency relationship between the first transaction and the first communication task.

[0016] In the first scenario, the first transaction is a first computational task, and the data transmitted by the first of the at least one first communication tasks includes first data, which is determined based on the computation result of the first computational task. That is, communication depends on computation.

[0017] In one possible implementation, the at least one first communication task corresponds one-to-one with at least one computation task, and the at least one computation task includes a first computation task; the first node executes the reference communication task based on the execution result of the preprocessing of the reference communication task, including: if the reference computation task is completed, then the reference communication task is executed based on the execution result of the preprocessing of the reference communication task to send reference data to the second node among the plurality of nodes, wherein the reference data is determined based on the computation result of the reference computation task, and the reference computation task refers to the computation task corresponding to the reference communication task.

[0018] In one possible implementation, the first node includes a first main execution unit, a first communication unit, and a first computing unit. If the reference computing task is completed, the reference communication task is executed based on the execution result of the preprocessing of the reference communication task. This includes: if the first computing unit completes the reference computing task, the first main execution unit sends a completion indication message for the reference computing task to the first communication unit; the first communication unit, in response to the completion indication message, executes the reference communication task based on the execution result of the preprocessing of the reference communication task. That is, the first main execution unit has the function of triggering the first communication unit to execute the corresponding communication task through instructions. Accordingly, the first communication unit executes the corresponding communication task under the instruction of the first main execution unit.

[0019] In the second scenario, the first transaction is the second communication task. The data transmitted in the second communication task includes second data from a third node among the aforementioned nodes. This second data is used as input data for the second computation task of the aforementioned computation model after the second communication task is completed. In other words, computation depends on communication.

[0020] In one possible implementation, the second communication task is the first communication task among a plurality of communication tasks, and the at least one first communication task is a communication task other than the first communication task among the plurality of communication tasks. The plurality of communication tasks correspond one-to-one with a plurality of computing tasks, and the plurality of computing tasks include the second computing task. The method further includes: whenever any of the plurality of communication tasks is completed, executing the corresponding computing task based on the data transmitted by the any of the communication tasks.

[0021] In one possible implementation, the first node includes a first main execution unit, a first communication unit, and a first computing unit. Whenever any one of the plurality of communication tasks completes, a corresponding computing task is executed based on the data transmitted by that communication task. This includes: whenever the first communication unit completes any communication task, the first main execution unit sends a completion indication message for that communication task to the first computing unit; and the first computing unit, in response to the completion indication message, executes the corresponding computing task based on the data transmitted by that communication task. That is, the first main execution unit has the function of triggering the first computing unit to execute a corresponding computing task through instructions. Accordingly, the first computing unit executes the corresponding computing task under the instruction of the first main execution unit.

[0022] In the third scenario, the first transaction is a second communication task. The data transmitted by the second communication task includes third data from a fourth node among the aforementioned multiple nodes. The first of the at least one first communication task is used to transmit the fourth data to a fifth node among these multiple nodes, and the fourth data is determined based on the third data. That is, communication depends on communication.

[0023] In one possible implementation, the second communication task is the first communication task among the plurality of communication tasks, and the at least one first communication task is a communication task other than the first communication task among the plurality of communication tasks, the plurality of communication tasks are executed sequentially, and the data to be transmitted by any non-first communication task among the plurality of communication tasks is determined based on the data transmitted by the preceding communication task of any non-first communication task.

[0024] In one possible implementation, when the first transaction is a second communication task, in order to start the second communication task as soon as possible, before the first main execution unit sends the pre-processing instruction message of the at least one first communication task to the first communication unit, the method further includes: the first main execution unit sending the pre-processing instruction message of the second communication task to the first communication unit; the first communication unit performing pre-processing of the second communication task in response to the pre-processing instruction message of the second communication task; and the first communication unit executing the second communication task based on the execution result of the pre-processing of the second communication task. That is, the first main execution unit first sends out the pre-processing instruction message of the first communication task (i.e., the second communication task) to prepare for the execution of the first communication task as soon as possible, and then sends the pre-processing instruction messages of the remaining communication tasks (i.e., the aforementioned at least one first communication task).

[0025] Secondly, a task execution apparatus is provided, which has the function of implementing the task execution method behavior described in the first aspect. The task execution apparatus includes one or more modules for implementing the task execution method provided in the first aspect.

[0026] Thirdly, a computing cluster is provided, comprising multiple nodes, on which a computing model is distributed and runs. A first node among these nodes is used to execute the steps of the task execution method provided in the first aspect. In one possible implementation, the first node is any one of the multiple nodes.

[0027] Fourthly, a computing device is provided, the computing device including a processor and a memory, the memory being used to store a program for executing the task execution method provided in the first aspect, and to store data related to implementing the task execution method provided in the first aspect. The processor is configured to execute the program stored in the memory.

[0028] In one possible implementation, the computing device may further include a communication bus for establishing a connection between the processor and the memory.

[0029] Fifthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the task execution method described in the first aspect.

[0030] In a sixth aspect, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the task execution method described in the first aspect above.

[0031] The technical effects achieved by the second, third, fourth, fifth, and sixth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0032] Figure 1 is a functional schematic diagram of a communication operator provided in an embodiment of this application;

[0033] Figure 2 is a functional schematic diagram of another communication operator provided in an embodiment of this application;

[0034] Figure 3 is a schematic diagram of a method for separating and parallelizing computation and communication in related technologies;

[0035] Figure 4 is a schematic diagram of the related technologies when tasks are not executed in parallel without partitioning, and when tasks are executed in parallel through partitioning.

[0036] Figure 5 is a flowchart of a task execution method provided in an embodiment of this application;

[0037] Figure 6 is a schematic diagram illustrating a communication-dependent computation method provided in an embodiment of this application;

[0038] Figure 7 is a schematic diagram showing that multiple first communication tasks provided in the embodiments of this application each depend on a computing task;

[0039] Figure 8 is another schematic diagram showing that multiple first communication tasks, each dependent on a computing task, are provided in an embodiment of this application.

[0040] Figure 9 is a schematic diagram illustrating a computation-dependent communication method provided in an embodiment of this application;

[0041] Figure 10 is a schematic diagram showing that multiple computing tasks, each dependent on a communication task, are provided in an embodiment of this application.

[0042] Figure 11 is another schematic diagram showing that multiple computing tasks, each dependent on a communication task, are provided in an embodiment of this application.

[0043] Figure 12 is a schematic diagram of a communication-dependent communication provided in an embodiment of this application;

[0044] Figure 13 is a schematic diagram showing the data dependency relationship between multiple communication tasks provided in the embodiments of this application;

[0045] Figure 14 is a schematic diagram of the interaction between a main execution unit and a communication unit provided in an embodiment of this application;

[0046] Figure 15 is a schematic diagram of another interaction between the main execution unit and the communication unit provided in an embodiment of this application;

[0047] Figure 16 is a schematic diagram of the structure of a task execution device provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0049] First, a brief introduction to the background knowledge involved in the embodiments of this application will be given.

[0050] Currently, computational models such as AI models, protein sequence analysis models, and large-scale scientific computing models have reached the levels of billions, tens of billions, or even trillions of parameters, and the corresponding computational demands are rapidly increasing. To support computational models with large parameter counts within limited hardware resources, the industry primarily employs methods such as tensor parallelism, data parallelism, pipelined parallelism, model parallelism, and sequence parallelism. These methods distribute a computational model across multiple nodes. However, the data used in computational models is typically globally correlated, requiring communication between multiple nodes to achieve data interaction.

[0051] Taking the transformer block structure commonly used in AI models as an example, in strategies that enable tensor parallelism and sequence parallelism, data interaction between nodes can be achieved by adding AllGather (AG) communication operators and ReduceScatter (RS) communication operators between computation operators.

[0052] The main function of the AllGather communication operator is to collect data from all nodes and then transmit all the data to all nodes. As shown in Figure 1, the data on nodes 0, 1, 2, and 3 are A, B, C, and D, respectively. The AllGather communication operator can collect data A, B, C, and D and then transmit them to these four nodes, so that all four nodes have data A, B, C, and D. In the specific implementation, node 0 uses the AllGather communication operator to transmit data A to nodes 1, 2, and 3; node 1 uses the AllGather communication operator to transmit data B to nodes 0, 2, and 3; node 2 uses the AllGather communication operator to transmit data C to nodes 0, 1, and 3; and node 3 uses the AllGather communication operator to transmit data C to nodes 0, 1, and 2.

[0053] The main function of the ReduceScatter communication operator is to collect and sum the data from all nodes, then divide the summed data according to the total number of nodes, and distribute the divided data to the corresponding nodes. As shown in Figure 2, the data on node 0 includes A1, B1, C1, and D1; the data on node 1 includes A2, B2, C2, and D2; the data on node 2 includes A3, B3, C3, and D3; and the data on node 3 includes A4, B4, C4, and D4. The ReduceScatter communication operator can sum the data A1, A2, A3, and A4 and distribute it to node 0; sum the data B1, B2, B3, and B4 and distribute it to node 1; sum the data C1, C2, C3, and C4 and distribute it to node 2; and sum the data D1, D2, D3, and D4 and distribute it to node 3.

[0054] As shown in Figures 1 and 2, the communication operator is dependent on the data generated by a specific transaction. Taking Linear->ReduceScatter as an example, Linear is a computation operator, and ReduceScatter is a communication operator; there is a data dependency between them. Only after Linear completes its execution and prepares the data can ReduceScatter execute the data interaction communication task. The communication task mainly includes two processes: preprocessing and data transfer. Preprocessing is actually independent of the data; in this embodiment, the preprocessing of the communication task can be referred to as the head overhead of the communication task. As an additional overhead, the larger the head overhead, the more it affects the execution performance of large AI models. Therefore, the smaller the head overhead in the communication task, the better.

[0055] Referring to Figure 3, the related technology integrates Linear (abbreviated as L) and ReduceScatter into a single operator, dividing their common axis. Performance optimization is achieved by parallelizing multiple computations and communications. Specifically, the data to be processed by the Linear computation operator is divided into multiple parts, each processed by a separate Linear computation operator. The processed data is then transmitted via the ReduceScatter communication operator. The processing of these multiple data parts is performed in parallel, and the processed data parts are transmitted to the same node. After all the Linear and ReduceScatter operations have been completed on each of these multiple data parts, the node then concatenates the received data parts.

[0056] Figure 4 illustrates the time required for tasks to be executed in parallel without partitioning in related technologies, and the time required for tasks to be executed in parallel with partitioning as shown in Figure 3. It can be seen that without partitioning, parallel execution requires waiting for the Linear operation to complete before performing preprocessing for the communication task, and then performing the communication task (i.e., data transfer), resulting in a longer overall time consumption. However, by partitioning a large amount of data into multiple smaller amounts, after performing the Linear computation operator on each smaller amount of data, a ReduceScatter communication operator can be synchronously executed based on the data obtained from the currently completed computation while simultaneously performing the Linear computation operator on the next smaller amount of data to transfer the data obtained from the currently completed computation. This significantly reduces the overall time consumption and improves overall performance.

[0057] In the embodiments of this application, the preprocessing of a communication task mainly includes preparatory work such as task orchestration or communication channel preparation. For example, the preprocessing of ReduceScatter includes task orchestration. The data interaction process of a communication task mainly includes data transmission. In some embodiments, the data interaction process of some communication tasks may also include data processing. For example, the data interaction process of ReduceScatter includes data summation and data transmission.

[0058] Header overhead is a fixed performance overhead, independent of the amount of data being communicated, but related to the number of partitions. The more partitions, the greater the head overhead. To reduce the increased head overhead after parallelization, related technologies primarily adjust the partitioning strategy. One method is to reduce the number of partitions, for example, from four partitions to two, reducing the head overhead by two. However, since the overall amount of data to be computed remains the same, when the number of partitions decreases, the amount of data to be processed per partition increases, resulting in longer computation time. This leads to longer waiting times for communication tasks to start, ultimately reducing overall efficiency.

[0059] To compensate for the longer waiting time for communication tasks to start due to reducing the number of data segments, a non-uniform segmentation method has been proposed. This method divides the total data volume into multiple segments of varying sizes, with the initial segment being smaller, thus reducing the time spent on the first computation and consequently reducing the waiting time for communication tasks to start. However, the non-uniform segmentation results in varying data sizes for each segment. Due to pipeline masking, the reduction in individual communication data sizes may lead to a mismatch between the reduced data size and computation time, resulting in gaps in communication tasks and ultimately, insufficient overall efficiency.

[0060] In summary, related technologies reduce header overhead by adjusting the segmentation strategy; however, communication tasks still require waiting for the data they depend on to be generated before they can start, resulting in lower overall efficiency. This application provides a task execution method that reduces the impact of communication task header overhead on the overall performance of the computational model without changing the segmentation strategy (e.g., without reducing the number of segments).

[0061] The implementation environment involved in the application embodiments will be described next.

[0062] The task execution method provided in this application is applied to a computing cluster, which can also be called a computing system. The computing cluster includes multiple nodes, and the computing model is distributed across these nodes. These nodes are connected via wired or wireless communication.

[0063] In one possible implementation, one of these nodes can be a processor or a computing chip, such as a neural network processing unit (NPU) or a graphics processing unit (GPU). It can also be a general-purpose central processing unit (CPU), a microprocessor, or one or more integrated circuits used to implement the scheme of this application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. In one possible implementation, the PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. This application does not limit the type of node. Different nodes can be of the same or different types; for example, some nodes may be NPUs and others may be GPUs, or all nodes may be NPUs. In this application, a node can also be referred to as a device or card, etc.

[0064] In one possible implementation, a node can be a single-core processor or a multi-core processor. Here, a processor refers to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0065] In this embodiment, each node further includes a storage unit (which may be referred to as on-chip storage space). The storage unit is used to store program code for executing the solution of this application. The node can execute the program code stored in the storage unit to implement the task execution method provided in the embodiment of Figure 5 below. The program code includes one or more software modules.

[0066] The aforementioned multiple nodes are located in at least one cabinet, and each cabinet has at least one node deployed or equipped. For example, these multiple nodes are located in the same cabinet, or some of these multiple nodes are located in cabinet 1, some in cabinet 2, and some in cabinet 3.

[0067] The system architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0068] Figure 5 is a flowchart of a task execution method provided in an embodiment of this application. This method is applied to a computing cluster, which includes multiple nodes, and a computing model is distributed across these nodes and runs on them. The method can be applied to a first node among these multiple nodes, and the first node can be any one of the multiple nodes. Referring to Figure 5, the method includes the following steps.

[0069] Step 501: During the execution of the first transaction, perform preprocessing for at least one first communication task. The first transaction is either a second communication task or a first computation task of the computation model. The at least one first communication task refers to a communication task that needs to be executed after the first transaction.

[0070] The preprocessing step is used to prepare for the execution of the corresponding communication task, which is used to transfer the running data of the computing model between all or some of the nodes in the multiple nodes.

[0071] In this embodiment, during the execution of the second communication task or the first computation task, the preprocessing for the subsequent first communication task is executed simultaneously. This prepares the system in advance for the execution of the first communication task, avoiding the extra time spent on preprocessing and facilitating faster startup of the first communication task, thereby improving the overall operating efficiency of the computation model. In other words, the technical solution provided in this embodiment improves the overall operating efficiency of the computation model by executing the data-independent preprocessing portion in advance.

[0072] In one possible implementation, the first node includes a first main execution unit and a first communication unit. During the execution of the first transaction, the first main execution unit sends a pre-processing instruction message for the at least one first communication task to the first communication unit. In response to the pre-processing instruction message for the at least one first communication task, the first communication unit performs pre-processing for the at least one first communication task. That is, the first main execution unit has the function of triggering the first communication unit to perform pre-processing via instructions, and correspondingly, the first communication unit performs the corresponding pre-processing under the instruction of the first main execution unit.

[0073] The at least one first communication task includes multiple first communication tasks, and the preprocessing indication message for each of the at least one first communication task includes multiple first preprocessing indication messages, each corresponding one-to-one with the multiple first communication tasks. These multiple first preprocessing indication messages are carried in the same instruction, or they are carried in multiple instructions, which are sent in multiple installments. That is, one instruction can trigger the preprocessing of at least one first communication task. For example, one instruction can trigger the preprocessing of one first communication task, or the preprocessing of two first communication tasks, or the preprocessing of all first communication tasks.

[0074] In the embodiments of this application, an instruction can be called a prepare instruction, or simply a P instruction. The main execution unit can also be called the main execution engine, controller, or main executor, or other names. The communication unit can also be called the communication engine, communication management engine, or other names. The computing unit can also be called the computing engine, or other names.

[0075] The aforementioned multiple first communication tasks are executed in the order of task execution. If the multiple first preprocessing indication messages are carried in multiple instructions, these multiple instructions are sent in any of the multiple sending methods, including sending in the order of task execution, sending in any order, and sending in parallel. That is, these multiple instructions can be sent sequentially, out of order, or in parallel. Parallel sending can mean that these multiple instructions are sent simultaneously, or that every first number of instructions is sent simultaneously in parallel. For example, if the first number is 2, then every two instructions are sent simultaneously in parallel, or if the first number is 3, then every three instructions are sent simultaneously. Of course, the first number can also be other values, and this application embodiment does not limit this.

[0076] Considering that the preprocessing of each communication task typically takes relatively little time, while the first transaction takes relatively more time, the preprocessing of these multiple first communication tasks can be executed in any of several ways. These methods include executing them in the order they are executed, executing them in any order, and executing them in parallel. That is, the preprocessing of these multiple first communication tasks can be executed sequentially, out of order, or in parallel. Parallel execution can mean that the preprocessing of these multiple first communication tasks is executed simultaneously, or that every second number of preprocessing steps within these multiple first communication tasks is executed simultaneously. For example, if the second number is 2, then every two preprocessing steps are executed simultaneously; or if the second number is 3, then every three preprocessing steps are executed simultaneously. Of course, the second number can also be other values.

[0077] In one possible implementation, the aforementioned first communication task is a communication task determined by a segmentation strategy in related technologies.

[0078] In this embodiment, the first transaction is either a second communication task or a first computing task. The aforementioned first communication task refers to a communication task that needs to be executed after the first transaction; that is, the first communication task and the first transaction have a data dependency relationship. Since the first transaction is either a second communication task or a first computing task, and the data dependency relationship between the first transaction and the first communication task is different depending on the type of task, the following sections will describe several scenarios of the data dependency relationship between the first transaction and the first communication task.

[0079] In the first scenario, the first transaction is a first computational task. The data transmitted by the first of these at least one first communication tasks includes first data, which is determined based on the computation result of the first computational task. That is, the data to be transmitted for communication depends on the computation result; simply put, communication depends on computation.

[0080] In one possible implementation, the at least one first communication task corresponds one-to-one with at least one computation task, and the at least one computation task includes a first computation task. Specifically, the first first communication task among the at least one first communication tasks corresponds to the first computation task.

[0081] Referring to Figures 6 to 8, Figure 6 is a schematic diagram illustrating a communication-dependent computation scenario according to an embodiment of this application. Figures 7 and 8 are two schematic diagrams illustrating multiple first communication tasks depending on a single computation task, as provided in an embodiment of this application. In Figures 7 and 8, communication 0, communication 1, and communication 2 are three first communication tasks, corresponding one-to-one with computation 0, computation 1, and computation 2. Computation 0 is the first transaction (i.e., the first computation task), and communication 0 is the first first communication task. Communication 0 depends on computation 0, communication 1 depends on computation 1, and communication 2 depends on computation 2. The difference between Figures 7 and 8 is that the preprocessing instruction messages for the three first communication tasks shown in Figure 7 are carried in a single instruction, while the preprocessing instruction messages for the three first communication tasks shown in Figure 8 are carried in multiple instructions.

[0082] In one possible implementation, computation 0, computation 1, and computation 2 in Figures 7 and 8 are three computation tasks determined by a segmentation strategy in related technologies. Correspondingly, communication 0, communication 1, and communication 2 are also three communication tasks that correspond one-to-one with these three computation tasks, determined by the same segmentation strategy.

[0083] In one possible implementation, taking the determination of multiple computational tasks and multiple communication tasks through a segmentation strategy as an example, during the execution of the first computational task, the preprocessing of all communication tasks can be completed. Alternatively, the preprocessing of the first part of the communication tasks can be executed, and the preprocessing of the second part of the communication tasks can be executed during the execution of another computational task, and so on, until the preprocessing of all communication tasks is completed. The second part of the communication tasks is the communication task that needs to be executed after the so-called other computational task. That is, as long as the corresponding preprocessing is completed before the execution of a certain communication task, the embodiments of this application do not limit whether the preprocessing of the communication task must be executed in parallel with the first computational task.

[0084] For example, taking Communication 0, Communication 1, Communication 2, Calculation 0, Calculation 1, and Calculation 2 in Figures 7 and 8 as examples, while executing Calculation 0, the preprocessing of Communication 0 is performed simultaneously; while executing Calculation 1, the preprocessing of Communication 1 and Communication 2 is performed simultaneously. Specifically, during the execution of Calculation 0, Calculation 0 is referred to as the first calculation task, and Communication 0 is referred to as the first communication task. Similarly, during the execution of Calculation 1, Calculation 1 is referred to as the first calculation task, and both Communication 1 and Communication 2 are referred to as the first communication tasks.

[0085] Besides the one-to-one correspondence between multiple first communication tasks and multiple computing tasks described above, in other embodiments, the first first communication task among these multiple first communication tasks has a data dependency relationship with the first computing task, while any first communication task other than the first first communication task may have a data dependency relationship with another communication task, but not with any computing task. That is, this solution does not limit whether the first communication task other than the first first communication task has a data dependency relationship with a computing task or with a communication task.

[0086] In the second scenario, the first transaction is the second communication task. The data transmitted in the second communication task includes second data from a third node among the aforementioned nodes. This second data serves as the input data for the second computation task of the computational model after the second communication task is completed. In other words, the input to the computation depends on the result of the communication; simply put, the computation depends on the communication.

[0087] The second communication task transmits data including data from at least one third node among the aforementioned multiple nodes; that is, there may be one or more third nodes.

[0088] The second communication task is the first communication task among a plurality of communication tasks. The aforementioned at least one first communication task is a communication task other than the first communication task among the plurality of communication tasks. In one possible implementation, the plurality of communication tasks correspond one-to-one with a plurality of computation tasks, and the plurality of computation tasks include the second computation task.

[0089] Referring to Figures 9 to 11, Figure 9 is a schematic diagram illustrating a computation dependent on communication according to an embodiment of this application. Figures 10 and 11 are two schematic diagrams illustrating multiple computation tasks depending on a single communication task according to an embodiment of this application. In Figures 10 and 11, communication 0, communication 1, and communication 2 are three communication tasks. Communication 0 is the first task (i.e., the second communication task), and communication 1 and communication 2 are two first communication tasks. Communication 0, communication 1, and communication 2 correspond one-to-one with computation 0, computation 2, and computation 3, where computation 0 is the second computation task. Computation 0 depends on communication 0, computation 1 depends on communication 1, and computation 2 depends on communication 2. The difference between Figures 10 and 11 is that the preprocessing instruction messages for the two first communication tasks shown in Figure 10 are carried in one instruction, while the preprocessing instruction messages for the two first communication tasks shown in Figure 11 are carried in multiple instructions.

[0090] In one possible implementation, communication 0, communication 1, and communication 2 in Figures 10 and 11 are three communication tasks determined by a segmentation strategy in related technologies. Correspondingly, computation 0, computation 1, and computation 2 are also three computation tasks that correspond one-to-one with these three communication tasks, determined by the same segmentation strategy.

[0091] In one possible implementation, taking the determination of multiple computational tasks and multiple communication tasks through a segmentation strategy as an example, during the execution of the first communication task, the preprocessing of all remaining communication tasks can be completed. Alternatively, the preprocessing of the first part of the remaining communication tasks can be executed, and the preprocessing of the second part of the communication tasks can be executed during the execution of another communication task, and so on, until the preprocessing of all communication tasks is completed. The second part of the communication task is the communication task that needs to be executed after the so-called other communication task. That is, as long as the corresponding preprocessing is completed before the execution of a certain communication task, the embodiments of this application do not limit whether the preprocessing of the communication task must be executed in parallel with the first communication task.

[0092] For example, taking Communication 0, Communication 1, Communication 2, Calculation 0, Calculation 1, and Calculation 2 in Figures 10 and 11 as examples, while executing Communication 0, the preprocessing of Communication 1 is executed simultaneously; while executing Communication 1, the preprocessing of Communication 2 is executed simultaneously. During the execution of Communication 0, Communication 0 is referred to as the second communication task, and Communication 1 is referred to as the first communication task. During the execution of Communication 1, Communication 1 is referred to as the second communication task, and Communication 2 is referred to as the first communication task.

[0093] In other embodiments, besides the one-to-one correspondence between multiple computing tasks and multiple communication tasks described above, the first communication task (i.e., the second communication task) among these multiple communication tasks has a data dependency relationship with the second computing task, while any communication task other than the first communication task (i.e., any first communication task) among these multiple communication tasks may have a data dependency relationship with another communication task, but not with any computing task. That is, this solution does not limit whether any first communication task among these multiple first communication tasks has a data dependency relationship with a computing task or with a communication task.

[0094] In the third scenario, the first transaction is a second communication task. The data transmitted by the second communication task includes third data from a fourth node among the aforementioned multiple nodes. The first of the at least one first communication task is used to transmit the fourth data to a fifth node among these multiple nodes. The fourth data is determined based on the third data. That is, the data to be transmitted in a subsequent communication depends on the result of the previous communication; simply put, communication depends on communication.

[0095] The second communication task transmits data including data from at least one fourth node among the multiple nodes mentioned above; that is, there may be one or more fourth nodes. Similarly, the first communication task transmits fourth data to at least one fifth node among the multiple nodes; that is, there may be one or more fifth nodes.

[0096] The second communication task is the first communication task among a plurality of communication tasks. At least one of these first communication tasks is a communication task other than the first communication task. These communication tasks are executed sequentially. In one possible implementation, the data transmitted by any non-first communication task is determined based on the data transmitted by the preceding communication task. That is, each communication task other than the first communication task has a data dependency on the preceding communication task.

[0097] Referring to Figures 12 and 13, Figure 12 is a schematic diagram illustrating a communication dependency relationship according to an embodiment of this application. Figure 13 is a schematic diagram illustrating a data dependency relationship between multiple communication tasks provided in an embodiment of this application. In Figure 13, communication A0, communication B0, communication A1, communication B1, communication A2, and communication B2 are six communication tasks. Among them, communication A0 is the first transaction (i.e., the second communication task), and the remaining five communication tasks are five first communication tasks. Communication B0 depends on communication A0, communication A1 depends on communication B0, communication B1 depends on communication A1, communication A2 depends on communication B1, and communication B2 depends on communication A2. The preprocessing instruction messages of the five first communication tasks shown in Figure 13 are carried in one instruction. In one possible implementation, the preprocessing instruction messages of these five first communication tasks can also be carried in multiple instructions, such as in five instructions or in three instructions.

[0098] In one possible implementation, the six communication tasks in Figure 13 are communication tasks determined by a segmentation strategy in related technologies.

[0099] In one possible implementation, during the execution of the first communication task out of multiple communication tasks, the preprocessing for all remaining communication tasks can be completed. Alternatively, the preprocessing for the first part of the remaining communication tasks can be executed, and the preprocessing for the second part of the communication tasks can be executed during the execution of another communication task, and so on, until the preprocessing for all communication tasks is completed. The second part of the communication task is the communication task that needs to be executed after the so-called other communication task. That is, as long as the corresponding preprocessing is completed before the execution of a certain communication task, the embodiments of this application do not limit whether the preprocessing of the communication task must be executed in parallel with the first communication task.

[0100] For example, taking the six communication tasks shown in Figure 13 as an example, while executing communication A0, the preprocessing for communication B0 and communication A1 is executed simultaneously. Similarly, while executing communication B0 (or communication A1), the preprocessing for communication B1, communication A2, and communication B2 is executed simultaneously. During the execution of communication A0, communication A0 is referred to as the second communication task, and communication B0 and communication A1 are both referred to as the first communication task. During the execution of communication B0 (or communication A1), communication B0 (or communication A1) is referred to as the second communication task, and communication B1, communication A2, and communication B2 are all referred to as the first communication task.

[0101] In one possible implementation, any of these multiple communication tasks, except for the first and second communication tasks, may not have a data dependency with the previous communication task, but may have a data dependency with a certain computing task, except that this communication task needs to be executed after the first communication task.

[0102] In this embodiment, when the first transaction is the second communication task, in order to start the second communication task as quickly as possible, the first main execution unit sends a pre-processing instruction message for the second communication task to the first communication unit before sending at least one pre-processing instruction message for the first communication task to the first communication unit. The first communication unit responds to the pre-processing instruction message for the second communication task by performing pre-processing for the second communication task, and then executes the second communication task based on the execution result of the pre-processing. That is, the first main execution unit first sends out the pre-processing instruction message for the first communication task (i.e., the second communication task) to prepare for the execution of the first communication task as quickly as possible, and then sends the pre-processing instruction messages for the remaining communication tasks (i.e., the aforementioned at least one first communication task) subsequently.

[0103] As shown in Figures 10 and 11, the preprocessing instruction message for communication 0 is first sent via one instruction, and the preprocessing instruction messages for communication 1 and communication 2 are subsequently sent via one or two instructions. As shown in Figure 13, the preprocessing instruction message for communication A0 is first sent via one instruction, and the preprocessing instruction messages for the remaining five communication tasks are then sent via one instruction, or they may be sent via multiple instructions.

[0104] In one possible implementation, in some other embodiments, when the first transaction is the second communication task, the pre-processing instruction messages of the above-mentioned multiple communication tasks can also be carried in one instruction and sent simultaneously. However, in order to start the first communication task as soon as possible, the first communication unit can first perform the pre-processing of the first communication task.

[0105] Step 502: If the preprocessing of the reference communication task in the at least one first communication task is completed, then the reference communication task is executed based on the execution result of the preprocessing of the reference communication task. The reference communication task is any one of the first communication tasks.

[0106] It should be understood that the first communication task needs to wait for the first transaction to complete before execution. Since the preprocessing of each communication task typically takes relatively little time, while the execution of each computational and communication task usually takes a relatively long time, the preprocessing of at least one first communication task should normally have also been completed by the time the first transaction completes. Therefore, when the first transaction completes, the first node can execute the first communication task based on the execution result of its preprocessing. In other words, if the reference communication task is the first communication task, then when the preprocessing of the first communication task has been completed and the first transaction has been completed, the first node executes the first communication task based on its preprocessing result.

[0107] If other first communication tasks besides the first first communication task also need to wait for the completion of one or more other transactions (communication tasks or computing tasks) before they can be executed, for example, if other first communication tasks have data dependencies on other transactions, then other first communication tasks also need to wait for the completion of the corresponding transactions before they can be executed.

[0108] Any first communication task will be referred to as a reference communication task. The execution process of the reference communication task will be described in detail below.

[0109] In the first scenario, the at least one first communication task corresponds one-to-one with at least one computation task (as shown in Figures 7 and 8), and the at least one computation task includes the first computation task. In this case, if the reference computation task is completed, the first node executes the reference communication task based on the execution result of the preprocessing of the reference communication task to send reference data to the second node among the plurality of nodes. The reference data is determined based on the computation result of the reference computation task, which refers to the computation task corresponding to the reference communication task.

[0110] As shown in Figures 7 and 8, if computation 0 is completed, the first node executes communication 0 based on the execution result of the preprocessing of communication 0 to send data 0 to the second node. Data 0 is determined based on the computation result of computation 0. For example, data 0 is the computation result of computation 0, or it is the data obtained by the first node after processing the computation result of computation 0 using a certain computation operator. If computation 1 is completed, the first node executes communication 1 based on the execution result of the preprocessing of communication 1 to send data 1 to a certain node. Data 1 is determined based on the computation result of computation 1. If computation 2 is completed, the first node executes communication 2 based on the execution result of the preprocessing of communication 2 to send data 2 to a certain node. Data 2 is determined based on the computation result of computation 2.

[0111] In one possible implementation, the first node includes a first main execution unit, a first communication unit, and a first computing unit. If the first computing unit completes the reference computing task, the first main execution unit sends a completion indication message for the reference computing task to the first communication unit. In response to this completion indication message, the first communication unit executes the reference communication task based on the execution result of the preprocessing of the reference communication task. That is, the first main execution unit has the function of triggering the first communication unit to execute the corresponding communication task through instructions. Accordingly, the first communication unit executes the corresponding communication task under the instruction of the first main execution unit.

[0112] In one possible implementation, the first main execution unit detects whether the first computing unit has completed the reference computing task, or the first computing unit sends a completion indication message of the reference computing task to the first main execution unit after completing the reference computing task. The completion indication message here can be exactly the same as the completion indication message sent by the first main execution unit to the first communication unit, or there can be differences, such as format differences.

[0113] In the embodiments of this application, a completion indication message may be referred to as a commit instruction, or simply a C instruction.

[0114] In the second scenario, the second communication task is the first of a plurality of communication tasks. The aforementioned at least one first communication task is one of the communication tasks excluding the first one. These multiple communication tasks correspond one-to-one with these multiple computational tasks (as shown in Figures 10 and 11), and these multiple computational tasks include the second computational task. In this case, whenever any of these multiple communication tasks completes, the first node executes the corresponding computational task based on the data transmitted by that communication task (i.e., the currently completed communication task).

[0115] As shown in Figures 10 and 11, if communication 0 is completed, the first node performs calculation 0 based on the data transmitted in communication 0, for example, using the data transmitted in communication 0 as the input data for calculation 0. If communication 1 is completed, the first node performs calculation 1 based on the data transmitted in communication 1. If communication 2 is completed, the first node performs calculation 2 based on the data transmitted in communication 2.

[0116] In one possible implementation, the first node includes a first main execution unit, a first communication unit, and a first computing unit. Whenever the first communication unit completes any one of a plurality of communication tasks, the first main execution unit sends a completion indication message for that communication task (i.e., the currently completed communication task) to the first computing unit. In response to the completion indication message, the first computing unit executes the corresponding computing task based on the data transmitted in that communication task. That is, the first main execution unit has the function of triggering the first computing unit to execute the corresponding computing task through instructions. Accordingly, the first computing unit executes the corresponding computing task under the instruction of the first main execution unit.

[0117] In one possible implementation, the first main execution unit detects whether the first communication unit has completed any of the multiple communication tasks, or the first communication unit sends a completion indication message for the communication task to the first main execution unit after completing any communication task. The completion indication message here can be exactly the same as the completion indication message sent by the first main execution unit to the first computing unit, or it can be different, such as having a different format.

[0118] In the third scenario, the first transaction is the second communication task, which is the first communication task among multiple communication tasks. At least one of the aforementioned first communication tasks is a communication task other than the first one among these multiple communication tasks. These multiple communication tasks are executed sequentially, and the data to be transmitted by any non-first communication task is determined based on the data transmitted by the preceding communication task. In other words, whenever any communication task other than the last one among these multiple communication tasks completes, the first node executes the next communication task based on the data transmitted by the currently completed communication task.

[0119] As shown in Figure 13, if communication A0 is completed, the first node executes communication B0 based on the data transmitted in communication A0. If communication B0 is completed, the first node executes communication A1 based on the data transmitted in communication B0. If communication A1 is completed, the first node executes communication B1 based on the data transmitted in communication A1. If communication B1 is completed, the first node executes communication A2 based on the data transmitted in communication B1. If communication A2 is completed, the first node executes communication B2 based on the data transmitted in communication A2.

[0120] In one possible implementation, the first node includes a first main execution unit and a first communication unit. Whenever the first communication unit completes any communication task other than the last one among the multiple communication tasks, the first main execution unit sends a completion indication message for the corresponding communication task to the first communication unit. In response to the completion indication message, the first communication unit executes the next communication task based on the data transmitted by the corresponding communication task.

[0121] Of course, the first communication unit can also automatically execute the next communication task after completing a certain communication task, without the instruction of the first main execution unit.

[0122] As described above, in this embodiment, a communication task is actually divided into three stages: preprocessing, waiting for the communication task to complete, and committing. The preprocessing stage is a data-independent stage. By separating the data-independent stage from the communication task and executing it in advance, the extra time spent on preprocessing is reduced, which means reducing the overhead of the communication task.

[0123] In some embodiments, referring to FIG14, FIG14 is a schematic diagram of the interaction between a main execution unit and a communication unit according to an embodiment of the present application. During the execution of a task, the main execution unit sends a prepare instruction to the communication unit, which carries pre-processing indication messages for one or more communication tasks. In response to the prepare instruction, the communication unit executes the pre-processing of the one or more communication tasks. When the data on which a communication task depends is ready, the main execution unit sends a commit instruction to the communication unit, indicating that the corresponding data is ready. Based on the commit instruction, the communication unit executes the corresponding communication task based on the corresponding data. During the execution of a communication task, the main execution engine is in a waiting-done phase, i.e., the wait-done phase. In one possible implementation, during this phase, the main execution unit can execute other tasks.

[0124] In some embodiments, referring to FIG15, FIG15 is a schematic diagram of another interaction between a main execution unit and a communication unit provided in an embodiment of this application. To initiate the first communication task as early as possible, the main execution unit sends a prepare instruction to the communication unit, which carries a pre-processing indication message for the first communication task. In response to the prepare instruction, the communication unit performs pre-processing for the first communication task. When the data upon which the first communication task depends is ready, the main execution unit sends a commit instruction to the communication unit, indicating that the corresponding data is ready. Based on the commit instruction, the communication unit performs the first communication task based on the corresponding data. During the execution of the first communication task, the main execution unit sends another prepare instruction to the communication unit, which carries pre-processing indication messages for the remaining one or more communication tasks. In response to this prepare instruction, the communication unit simultaneously performs pre-processing for the remaining one or more communication tasks during the execution of the first communication task. After the first communication task is completed, the communication unit executes subsequent communication tasks in sequence. During the execution of a certain communication task, the main execution engine is in the waiting-done stage, which is the waiting-done stage. In one possible implementation, the main execution unit can execute other tasks during this stage.

[0125] In summary, in this embodiment, during the execution of the first transaction, the preprocessing for the subsequent first communication task is executed simultaneously. This prepares the first communication task in advance, avoiding extra time spent on preprocessing and allowing for faster initiation of the first communication task. Related technologies reduce the number of data segments, thereby reducing the number of communication tasks and consequently reducing preprocessing time, thus saving on preprocessing time. Therefore, the technical solution of this embodiment does not require changing the data segmentation strategy and can improve the overall operating efficiency of the computational model by executing data-independent preprocessing parts in advance. Of course, regardless of whether the data segmentation strategy is adjusted, the overall operating efficiency of the computational model can be improved by executing data-independent preprocessing parts in advance.

[0126] Figure 16 is a schematic diagram of a task execution device 1600 provided in an embodiment of this application. The device 1600 can be implemented as part or all of a computing device by software, hardware, or a combination of both. This computing device can be any node in the above method embodiments; for example, the computing device can be a first node. In this embodiment, the computing model is distributed across multiple nodes, and the device 1600 is included in the first node among these multiple nodes. Referring to Figure 16, the device 1600 includes: a first execution module 1601 and a second execution module 1602.

[0127] The first execution module 1601 is used to perform preprocessing of at least one first communication task during the execution of the first transaction. The preprocessing is used to prepare for the execution of the corresponding communication task. The first transaction is either a second communication task or a first computing task of the above-mentioned computing model. The at least one first communication task refers to a communication task that needs to be executed after the first transaction. The communication task is used to transmit the running data of the computing model between all or some of the above-mentioned multiple nodes.

[0128] The second execution module 1602 is used to execute a reference communication task based on the execution result of the preprocessing of the reference communication task if the preprocessing of at least one of the first communication tasks is completed. The reference communication task is any one of the first communication tasks.

[0129] In one possible implementation, the first node includes a first main execution unit and a first communication unit, and the device 1600 further includes:

[0130] The first main execution unit is used to send a pre-processing instruction message for the at least one first communication task to the first communication unit during the execution of the first transaction.

[0131] The first execution module 1601 includes:

[0132] A first communication unit is configured to perform preprocessing of the at least one first communication task in response to a preprocessing instruction message for the at least one first communication task.

[0133] In one possible implementation, the first transaction is the second communication task;

[0134] The first main execution unit is further configured to send a preprocessing instruction message for a second communication task to the first communication unit before sending the preprocessing instruction message for the at least one first communication task to the first communication unit.

[0135] The first communication unit is also configured to perform preprocessing of the second communication task in response to the preprocessing instruction message of the second communication task;

[0136] The first communication unit is also used to execute the second communication task based on the execution result of the preprocessing of the second communication task.

[0137] In one possible implementation, the at least one first communication task includes a plurality of first communication tasks, and the preprocessing indication message of the at least one first communication task includes a plurality of first preprocessing indication messages, which correspond one-to-one with the plurality of first communication tasks.

[0138] These multiple first preprocessing instruction messages are carried in the same instruction, or these multiple first preprocessing instruction messages are carried in multiple instructions, which are sent in separate batches.

[0139] In one possible implementation, these multiple first communication tasks are executed in the order of task execution;

[0140] If the aforementioned multiple first preprocessing instruction messages are carried in multiple instructions, then these multiple instructions are sent in any of the multiple sending methods, including sending in the order of task execution, sending in any order, and sending in parallel.

[0141] In one possible implementation, the preprocessing of these multiple first communication tasks is performed in any of a variety of execution methods, including execution in the order of task execution of the multiple first communication tasks, execution in any order, and parallel execution.

[0142] In one possible implementation, the first transaction is a first computation task, and the data used to transmit by the first first communication task in the at least one first communication task includes first data, which is determined based on the computation result of the first computation task.

[0143] In one possible implementation, the above-mentioned at least one first communication task corresponds one-to-one with at least one computing task, and the at least one computing task includes the first computing task.

[0144] The second execution module 1602 is used for:

[0145] If the reference computation task is completed, the reference communication task is executed based on the execution result of the preprocessing of the reference communication task to send reference data to the second node among the multiple nodes. The reference data is determined based on the computation result of the reference computation task, which refers to the computation task corresponding to the reference communication task.

[0146] In one possible implementation, the first node includes a first main execution unit, a first communication unit, and a first computing unit;

[0147] The second execution module 1602 includes:

[0148] The first main execution unit is configured to send a reference computing task completion indication message to the first communication unit if the first computing unit has completed the reference computing task.

[0149] The first communication unit is used to execute the reference communication task based on the execution result of the preprocessing of the reference communication task in response to the completion indication message.

[0150] In one possible implementation, the first transaction is a second communication task. The data transmitted by the second communication task includes second data from a third node among the aforementioned multiple nodes. The second data is used as input data for the second computation task of the aforementioned computation model after the second communication task is completed.

[0151] In one possible implementation, the second communication task is the first communication task among a plurality of communication tasks, and the aforementioned at least one first communication task is a communication task other than the first communication task among the plurality of communication tasks, and the plurality of communication tasks correspond one-to-one with a plurality of computing tasks, including the second computing task.

[0152] The device 1600 also includes:

[0153] The third execution module is used to execute the corresponding computation task based on the data transmitted by any of the above multiple communication tasks once the communication task is completed.

[0154] In one possible implementation, the first node includes a first main execution unit, a first communication unit, and a first computing unit;

[0155] The third execution module includes:

[0156] The first main execution unit is used to send a completion indication message for any of the above communication tasks to the first computing unit whenever the first communication unit finishes executing any of the above communication tasks.

[0157] The first computing unit is configured to, in response to the completion indication message, execute the corresponding computing task based on the data transmitted by any of the communication tasks.

[0158] In one possible implementation, the first transaction is a second communication task, and the data transmitted by the second communication task includes third data from a fourth node among the plurality of nodes. The first of the at least one first communication task is used to transmit the fourth data to a fifth node among the plurality of nodes, and the fourth data is determined based on the third data.

[0159] In one possible implementation, the second communication task is the first communication task among a plurality of communication tasks, and the at least one first communication task is a communication task other than the first communication task among the plurality of communication tasks, which are executed sequentially, and the data to be transmitted by any non-first communication task among the plurality of communication tasks is determined based on the data transmitted by the preceding communication task of that non-first communication task.

[0160] In this embodiment, during the execution of the first transaction, preprocessing for the subsequent first communication task is performed simultaneously. This prepares the first communication task in advance, avoiding extra time spent on preprocessing and allowing for faster initiation of the first communication task. Related technologies reduce the number of data segments, thereby reducing the number of communication tasks and consequently reducing preprocessing time. Therefore, the technical solution of this embodiment does not require changing the data segmentation strategy and can improve the overall operating efficiency of the computational model by executing data-independent preprocessing in advance.

[0161] It should be noted that the task execution device provided in the above embodiments is only illustrated by the division of the above functional modules when executing tasks. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the task execution device and the task execution method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0162] This application embodiment also provides a computing cluster, which includes multiple nodes, and a computing model is distributed and runs on the multiple nodes. The first node among the multiple nodes is used to execute the steps of the task execution method shown in the above method embodiment.

[0163] In one possible implementation, the first node is any one of the plurality of nodes.

[0164] This application also provides a computing device, which includes a processor and a memory. The memory stores a program for executing the task execution method shown in the above-described method embodiments, and stores data related to implementing the task execution method shown in the above-described method embodiments. The processor is configured to execute the program stored in the memory.

[0165] In one possible implementation, the computing device may also include a communication bus for establishing a connection between the processor and the memory.

[0166] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the task execution method shown in the above-described method embodiments.

[0167] This application also provides a computer program product containing instructions that, when run on a computer, causes the computer to perform the task execution method shown in the above method embodiments.

[0168] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital versatile disc (DVD)), or a semiconductor medium (e.g., solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0169] It should be understood that "at least one" as mentioned herein refers to one or more, and "multiple" refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., are not necessarily different.

[0170] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0171] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A task execution method, characterized in that, The computational model runs on multiple nodes, and the method is applied to a first node among the multiple nodes. The method includes: During the execution of the first transaction, preprocessing for at least one first communication task is performed. The preprocessing is used to prepare for the execution of the corresponding communication task. The first transaction is either a second communication task or a first computation task of the computation model. The at least one first communication task refers to a communication task that needs to be executed after the first transaction. The communication task is used to transmit the running data of the computation model between all or some of the nodes in the plurality of nodes. If the preprocessing of the reference communication task in the at least one first communication task is completed, the reference communication task is executed based on the execution result of the preprocessing of the reference communication task, where the reference communication task is any one of the first communication tasks.

2. The method as described in claim 1, characterized in that, The first node includes a first main execution unit and a first communication unit, and the method further includes: During the execution of the first transaction, the first main execution unit sends a pre-processing instruction message for the at least one first communication task to the first communication unit; The preprocessing for executing at least one first communication task includes: The first communication unit performs preprocessing for the at least one first communication task in response to the preprocessing instruction message for the at least one first communication task.

3. The method as described in claim 2, characterized in that, The first transaction is the second communication task; Before the first main execution unit sends the preprocessing indication message for the at least one first communication task to the first communication unit, the method further includes: The first main execution unit sends a pre-processing instruction message for the second communication task to the first communication unit; The first communication unit responds to the preprocessing instruction message of the second communication task and performs preprocessing of the second communication task; The first communication unit executes the second communication task based on the execution result of the preprocessing of the second communication task.

4. The method as described in claim 2 or 3, characterized in that, The at least one first communication task includes a plurality of first communication tasks, and the preprocessing indication message of the at least one first communication task includes a plurality of first preprocessing indication messages, and the plurality of first preprocessing indication messages correspond one-to-one with the plurality of first communication tasks; The multiple first preprocessing indication messages are carried in the same instruction, or the multiple first preprocessing indication messages are carried in multiple instructions, and the multiple instructions are sent in separate batches.

5. The method as described in claim 4, characterized in that, The plurality of first communication tasks are executed in the order of task execution; If the multiple first preprocessing instruction messages are carried in multiple instructions, then the multiple instructions are sent in any of the multiple sending methods, including sending in the order of task execution, sending in any order, and sending in parallel.

6. The method as described in claim 4 or 5, characterized in that, The preprocessing of the plurality of first communication tasks is performed in any of a variety of execution methods, including execution in the order of task execution of the plurality of first communication tasks, execution in any order, and parallel execution.

7. The method according to any one of claims 1-6, characterized in that, The first transaction is the first computing task, and the data used to transmit by the first first communication task in the at least one first communication task includes first data, which is determined based on the computing result of the first computing task.

8. The method as described in claim 7, characterized in that, The at least one first communication task corresponds one-to-one with at least one computing task, and the at least one computing task includes the first computing task; The execution of the reference communication task based on the preprocessing result of the reference communication task includes: If the reference computing task is completed, the reference communication task is executed based on the execution result of the preprocessing of the reference communication task to send reference data to the second node among the plurality of nodes. The reference data is determined based on the calculation result of the reference computing task, and the reference computing task refers to the computing task corresponding to the reference communication task.

9. The method as described in claim 8, characterized in that, The first node includes a first main execution unit, a first communication unit, and a first computing unit; If the reference computing task is completed, then the reference communication task is executed based on the execution result of the preprocessing of the reference communication task, including: If the first computing unit completes the reference computing task, the first main execution unit sends a completion indication message for the reference computing task to the first communication unit. In response to the completion indication message, the first communication unit executes the reference communication task based on the execution result of the preprocessing of the reference communication task.

10. The method according to any one of claims 1-6, characterized in that, The first transaction is the second communication task, and the data transmitted by the second communication task includes second data from a third node among the plurality of nodes. The second data is used as input data for the second computing task of the computing model after the second communication task is completed.

11. The method as described in claim 10, characterized in that, The second communication task is the first communication task among a plurality of communication tasks. The at least one first communication task is a communication task other than the first communication task among the plurality of communication tasks. The plurality of communication tasks correspond one-to-one with a plurality of computing tasks, and the plurality of computing tasks include the second computing task. The method further includes: Whenever any of the plurality of communication tasks is completed, the corresponding computation task is executed based on the data transmitted by that communication task.

12. The method as described in claim 11, characterized in that, The first node includes a first main execution unit, a first communication unit, and a first computing unit; The step of executing a corresponding computation task based on the data transmitted by any of the plurality of communication tasks upon completion includes: Whenever the first communication unit finishes executing any of the communication tasks, the first main execution unit sends a completion indication message for any of the communication tasks to the first computing unit; In response to the completion indication message, the first computing unit executes the corresponding computing task based on the data transmitted by any of the communication tasks.

13. The method according to any one of claims 1-6, characterized in that, The first transaction is the second communication task, and the data transmitted by the second communication task includes third data from the fourth node among the plurality of nodes. The first first communication task in the at least one first communication task is used to transmit the fourth data to the fifth node among the plurality of nodes, and the fourth data is determined based on the third data.

14. The method as described in claim 13, characterized in that, The second communication task is the first communication task among a plurality of communication tasks. The at least one first communication task is a communication task other than the first communication task among the plurality of communication tasks. The plurality of communication tasks are executed sequentially. The data to be transmitted by any non-first communication task among the plurality of communication tasks is determined based on the data transmitted by the preceding communication task of the non-first communication task.

15. A task execution device, characterized in that, The computational model is distributed across multiple nodes, and the device is included in a first node among the multiple nodes. The device includes: The first execution module is used to perform preprocessing of at least one first communication task during the execution of the first transaction. The preprocessing is used to prepare for the execution of the corresponding communication task. The first transaction is either a second communication task or a first computation task of the computation model. The at least one first communication task refers to a communication task that needs to be executed after the first transaction. The communication task is used to transmit the running data of the computation model between all or some of the nodes in the plurality of nodes. The second execution module is configured to execute the reference communication task based on the execution result of the preprocessing of the reference communication task if the preprocessing of the reference communication task in the at least one first communication task is completed, wherein the reference communication task is any one of the first communication tasks.

16. The apparatus as claimed in claim 15, characterized in that, The first node includes a first main execution unit and a first communication unit, and the device further includes: The first main execution unit is configured to send a pre-processing instruction message for the at least one first communication task to the first communication unit during the execution of the first transaction. The first execution module includes: The first communication unit is configured to perform preprocessing of the at least one first communication task in response to the preprocessing instruction message of the at least one first communication task.

17. The apparatus as claimed in claim 16, characterized in that, The first transaction is the second communication task; The first main execution unit is further configured to send a preprocessing indication message for the second communication task to the first communication unit before sending the preprocessing indication message for the at least one first communication task to the first communication unit; The first communication unit is further configured to perform preprocessing of the second communication task in response to the preprocessing instruction message of the second communication task; The first communication unit is further configured to execute the second communication task based on the execution result of the preprocessing of the second communication task.

18. The apparatus as claimed in claim 16 or 17, characterized in that, The at least one first communication task includes a plurality of first communication tasks, and the preprocessing indication message of the at least one first communication task includes a plurality of first preprocessing indication messages, and the plurality of first preprocessing indication messages correspond one-to-one with the plurality of first communication tasks; The multiple first preprocessing indication messages are carried in the same instruction, or the multiple first preprocessing indication messages are carried in multiple instructions, and the multiple instructions are sent in separate batches.

19. The apparatus as claimed in claim 18, characterized in that, The plurality of first communication tasks are executed in the order of task execution; If the multiple first preprocessing instruction messages are carried in multiple instructions, then the multiple instructions are sent in any of the multiple sending methods, including sending in the order of task execution, sending in any order, and sending in parallel.

20. The apparatus as claimed in claim 18 or 19, characterized in that, The preprocessing of the plurality of first communication tasks is performed in any of a variety of execution methods, including execution in the order of task execution of the plurality of first communication tasks, execution in any order, and parallel execution.

21. The apparatus according to any one of claims 15-20, characterized in that, The first transaction is the first computing task, and the data used to transmit by the first first communication task in the at least one first communication task includes first data, which is determined based on the computing result of the first computing task.

22. The apparatus as claimed in claim 21, characterized in that, The at least one first communication task corresponds one-to-one with at least one computing task, and the at least one computing task includes the first computing task; The second execution module is used for: If the reference computing task is completed, the reference communication task is executed based on the execution result of the preprocessing of the reference communication task to send reference data to the second node among the plurality of nodes. The reference data is determined based on the calculation result of the reference computing task, and the reference computing task refers to the computing task corresponding to the reference communication task.

23. The apparatus as claimed in claim 22, characterized in that, The first node includes a first main execution unit, a first communication unit, and a first computing unit; The second execution module includes: The first main execution unit is configured to send a completion indication message of the reference computing task to the first communication unit if the first computing unit has completed the execution of the reference computing task. The first communication unit is configured to execute the reference communication task based on the execution result of the preprocessing of the reference communication task in response to the completion indication message.

24. The apparatus according to any one of claims 15-20, characterized in that, The first transaction is the second communication task, and the data transmitted by the second communication task includes second data from a third node among the plurality of nodes. The second data is used as input data for the second computing task of the computing model after the second communication task is completed.

25. The apparatus as claimed in claim 24, characterized in that, The second communication task is the first communication task among a plurality of communication tasks. The at least one first communication task is a communication task other than the first communication task among the plurality of communication tasks. The plurality of communication tasks correspond one-to-one with a plurality of computing tasks, and the plurality of computing tasks include the second computing task. The device further includes: The third execution module is used to execute the corresponding computation task based on the data transmitted by any of the multiple communication tasks once the communication task is completed.

26. The apparatus as claimed in claim 25, characterized in that, The first node includes a first main execution unit, a first communication unit, and a first computing unit; The third execution module includes: The first main execution unit is configured to send a completion indication message for any communication task to the first computing unit whenever the first communication unit finishes executing any communication task. The first computing unit is configured to, in response to the completion indication message, execute a corresponding computing task based on the data transmitted by any of the communication tasks.

27. The apparatus according to any one of claims 15-20, characterized in that, The first transaction is the second communication task, and the data transmitted by the second communication task includes third data from the fourth node among the plurality of nodes. The first first communication task in the at least one first communication task is used to transmit the fourth data to the fifth node among the plurality of nodes, and the fourth data is determined based on the third data.

28. The apparatus as claimed in claim 27, characterized in that, The second communication task is the first communication task among a plurality of communication tasks. The at least one first communication task is a communication task other than the first communication task among the plurality of communication tasks. The plurality of communication tasks are executed sequentially. The data to be transmitted by any non-first communication task among the plurality of communication tasks is determined based on the data transmitted by the preceding communication task of the non-first communication task.

29. A computing cluster, characterized in that, The computing cluster includes multiple nodes, and the computing model is distributed and runs on the multiple nodes. The first node among the multiple nodes is used to execute the steps of the method according to any one of claims 1-14.

30. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1-14.

31. A computer program product, characterized in that, The computer program product stores computer instructions, which, when executed by a processor, implement the method described in any one of claims 1-14.

Images