Distributed computing method and system, and related device

Abstract

Provided in the present application are a distributed computing method and system, and a related device. The method comprises: a management node issuing a fused operator to a compute node, wherein the fused operator comprises two matrices and an instruction for implementing a matrix multiplication operation of the two matrices; on the basis of the fused operator, a plurality of matrix computation units in the compute node separately executing a partial operation in the multiplication operation of the two matrices, so as to obtain a partial computation result of the matrix multiplication operation; and then executing a communication task to synchronize the partial computation result. The management node only needs to send one fused operator to the compute node, and the compute node can complete computation of one matrix multiplication on the basis of the fused operator, without the need for the management node to split one matrix multiplication operation into a plurality of sub-tasks for computation and communication, and then sequentially send the sub-tasks to the compute node. Thus, idle resources of the compute node caused by the management node splitting and issuing tasks can be avoided, thereby improving the resource utilization and the efficiency of the distributed computing system.

Description

Distributed computing method, system and related device

[0001] The present application claims priority to the Chinese patent application No. 202411267368.4, filed on September 10, 2024, and entitled "Distributed computing method, system and related device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of computer technology, and in particular to a distributed computing method, system and related device. BACKGROUND

[0003] In large-scale computing through multiple computing nodes of a distributed computing system, each computing node is used to execute part of the tasks in the large-scale computing, a management node splits the task to be executed by a computing node into multiple computing tasks and multiple communication tasks, and then sequentially issues the multiple computing tasks and the multiple communication tasks to the computing node. The computing node implements parallel execution of the computing tasks and the communication tasks according to the computing tasks and the communication tasks issued by the management node through a reasonable scheduling strategy.

[0004] Generally, the more the computing tasks and the communication tasks that a computing node needs to execute are split, the more the computing tasks and the communication tasks overlap in parallel execution, which can improve the utilization of computing resources and communication resources, thereby improving the efficiency of the distributed system. However, regardless of the amount of data that a computing task needs to calculate or the amount of data that a communication task needs to transmit, the time for the management node to issue a computing task or a communication task is constant. The more the number of computing tasks that the management node splits from the task to be executed by a computing node, the less the amount of tasks of a single computing task and a single communication task, and the less the time required for the computing node to execute a single computing task and a single communication task. However, the time for the management node to issue a computing task or a communication task is constant. This results in a phenomenon that the time for the management node to issue a computing or communication task to the computing node is greater than the time for the computing node to execute a computing task or a communication task, which causes the computing node to wait for the management node to issue a computing task or a communication task, resulting in waste of computing node resources. SUMMARY

[0005] The present application provides a distributed computing method, system and related device, which can implement parallel computing and communication, and provide resource utilization and efficiency of a distributed computing system.

[0006] In a first aspect, the present application provides a distributed computing method applied to a distributed computing system including a management node and a plurality of computing nodes, the method comprising: the management node sending a fusion operator to each of the plurality of computing nodes, wherein the management node sends a first fusion operator to a first computing node; the first fusion operator including a first matrix and a second matrix, and an instruction for instructing the first computing node to perform a matrix multiplication operation of the first matrix and the second matrix; after receiving the first fusion operator, a plurality of matrix computing units in the first computing node perform part of the matrix multiplication operation of the first matrix and the second matrix according to the first fusion operator, to obtain a first calculation result; and then perform a communication task with other computing nodes in the distributed computing system, the communication task being used for synchronizing the first calculation result with the other computing nodes, and the first computing node also obtains calculation results of the other computing nodes by performing the communication task.

[0007] When the distributed computing system needs to perform distributed computing, and the computing nodes need to perform matrix multiplication, the management node only needs to send a fusion operator to the computing node, the fusion operator including an instruction for performing the matrix multiplication, i.e., the fusion operator including complete computing logic and communication logic for performing the matrix multiplication, and the computing node can complete one matrix multiplication based on the fusion operator. The management node does not need to split one matrix multiplication into a plurality of sub-tasks for computing and for communication, and then send the sub-tasks to the computing node one by one, which can avoid the phenomenon of idle computing node resources caused by the management node splitting and sending tasks, and improve the utilization of resources and the efficiency of the distributed computing system.

[0008] In a possible implementation, each matrix computing unit in the first computing node needs to perform m rounds of part of the matrix multiplication of the first matrix and the second matrix, i.e., each matrix computing unit performs part of the matrix multiplication of the first matrix and the second matrix in each round. The first fusion operator further includes sub-matrix information and a communication trigger parameter n; the sub-matrix information is used for indicating the size of a first sub-matrix and the size of a second sub-matrix participating in the calculation of each round of operation of one matrix computing unit, and the communication trigger parameter n is less than or equal to m, and m is a positive integer greater than 1. The plurality of matrix computing units in the first computing node perform part of the matrix multiplication of the first matrix and the second matrix according to the first fusion operator to obtain the first calculation result, including: each matrix computing unit in the first computing node acquires two sub-matrices from the first matrix and the second matrix according to the sub-matrix information in each round of calculation, performs part of the matrix multiplication of the first matrix and the second matrix, performs part of the matrix multiplication of the first matrix and the second matrix in parallel through the plurality of matrix computing units until n rounds of calculation are performed, to obtain the first calculation result, and then the first computing node starts to perform the communication task with other computing nodes to synchronize the results obtained by each computing node after n rounds of calculation.

[0009] By setting appropriate sub-matrix information and communication trigger parameter n, each matrix calculation unit calculates part of data from the first matrix and the second matrix respectively in each round of calculation, and triggers calculation after each matrix calculation unit performs n rounds of calculation to obtain the first calculation result. In this way, the first calculation results calculated by the plurality of matrix calculation units are communicated in a centralized manner, which can reduce the number of times of synchronizing data between the calculation nodes, and can also avoid the problem that the amount of data of the first calculation result obtained before performing the communication task is large, causing the memory of the calculation node to be insufficient to store the calculation results obtained in subsequent rounds of calculation before the communication task is completed.

[0010] In a possible implementation, the method further includes: when the first calculation node performs the communication task with other calculation nodes in the plurality of calculation nodes, the plurality of matrix calculation units start to perform the n+1th round of calculation, that is, the first calculation node can continue to perform calculation through the matrix calculation unit while performing the communication task, to realize parallel execution of calculation and communication, and improve the resource utilization rate and efficiency of the distributed computing system.

[0011] In a possible implementation, the first fusion operator further includes an output address, and the output address is used to indicate an address of the shared memory space of the first calculation node; the method further includes: the first calculation node stores the first calculation result in the shared memory space according to the output address, where the shared memory space can be read and written by other calculation nodes.

[0012] The output address is included in the first fusion operator issued by the management node, and one matrix calculation unit of the first calculation node can directly write the result matrix obtained after performing one round of matrix multiplication of the sub-matrix into the shared memory space through an instruction. When other calculation nodes need to obtain the first calculation result of the first calculation node, the other calculation nodes can obtain the first calculation result from the shared memory space of the first calculation node through an instruction. This method can reduce the copying operation of data in different memory areas, and improve the communication efficiency.

[0013] In a possible implementation, the first calculation node stores the first calculation result in the shared memory space according to the output address, including: when each matrix calculation unit in the first calculation node stores the multiple rows of data included in the result matrix obtained after performing one round of calculation into the shared memory space in sequence, the i+1th row and the ith row of the result matrix are stored continuously in the shared memory space.

[0014] By storing the i+1th row data and the ith row data of the result matrix of the matrix multiplication of the group of sub-matrices in a continuous memory space, when other computing nodes obtain the result matrix obtained by the matrix multiplication of the group of sub-matrices in the process of performing the communication task, the other computing nodes can obtain the result matrix from the continuous memory space without obtaining the result matrix from different memory spaces, and the communication efficiency can be improved.

[0015] In a possible implementation, the first fusion operator further includes a carry granularity, the carry granularity being used to indicate an amount of data that the first computing node obtains from the shared memory space of another computing node once by executing the obtaining instruction; before the management node issues the first fusion operator to the first computing node in the plurality of computing nodes, the management node determines the carry granularity according to the computation amount and the computation time length of the n rounds of computation and the communication amount and the communication time length of the communication task.

[0016] When the computing node performs the communication task, the computing node obtains data from the shared memory space of another computing node by executing the obtaining instruction, and one execution of the obtaining instruction cannot obtain the complete computation result in another computing node, so the computing node needs to execute the obtaining instruction multiple times to obtain the complete computation result in another computing node. Since the computing node performs the communication task while performing computation, both the communication task and the computation task need to access the memory, and both the communication task and the computation task need to occupy resources such as outstanding resources when accessing the memory. The smaller the carry granularity corresponding to the obtaining instruction is, the easier it is for the computation task to obtain the memory access resource. Therefore, by pre-determining the carry granularity in the communication task, the memory access resource occupied by one execution of the obtaining instruction can be controlled, so that the resources required by the computation task and the communication task can be reasonably allocated, the computation task and the communication task can be reasonably scheduled, the computation task and the communication task can be efficiently and parallelly executed, and the efficiency of the distributed computing system is provided.

[0017] In a possible implementation, the first computing node performs the communication task with other computing nodes in the plurality of computing nodes, including: the first computing node executes the obtaining instruction multiple times to obtain the computation result of each computing node from the shared memory space of the other computing node according to the carry granularity.

[0018] In a possible implementation, the first computing node performs the communication task with other computing nodes based on the memory access instruction.

[0019] The fusion operator includes instructions for implementing two matrix multiplication operations, that is, the application instructs the computing nodes to implement computation and communication at the instruction level, and therefore the matrix computation unit and the vector computation unit in the application can read and write the memory through Memory Tagging Extension (MTE) instructions, such as store / load memory access instructions. In the fusion operator, the address to which the matrix computation unit writes data after computation can be specified, and therefore the computation result obtained by each round of computation of the matrix computation unit can be directly written to the shared memory space.

[0020] In a possible implementation, the first computing node performs a communication task with other computing nodes in the plurality of computing nodes, including: the first computing node querying a flag bit of each computing node in the other computing nodes, the flag bit being used to indicate whether the computing node has completed n rounds of computation; and the first computing node determining that the flag bit of each computing node is the first value, and performing the communication task with the other computing nodes in the plurality of computing nodes; wherein the first value of the flag bit is used to indicate that the computing node has completed n rounds of computation.

[0021] Each matrix computation unit in the first computing node sets the flag bit to 1 through the vector computation unit after completing n rounds of matrix multiplication of sub-matrices. After setting the flag bit to 1, the vector computation unit of the first computing node queries the value of the flag bit of other computing nodes in the communication domain at a first period, to determine whether the other computing nodes in the communication domain have completed the current round of computation. Similarly, any computing node in the communication domain also sets the flag bit to 1 after completing n rounds of computation, and then queries the flag bit of other computing nodes at a first period, to determine whether the other computing nodes have completed the current round of computation. In the case that each computing node in the same communication domain determines that the other computing nodes have completed the current round of computation, the computing nodes in the communication domain start to perform a communication task through the vector computation unit, to synchronize the respective computation results, which can avoid the case that the communication is performed before the other computing nodes in the communication domain have completed m rounds of computation, resulting in a computation error.

[0022] In a possible implementation, the above matrix multiplication operation is a computation in an artificial intelligence (AI) model training process; and the computing node can be a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or other special AI processing chips.

[0023] In a second aspect, the present application provides a distributed computing system, which comprises a management node and a plurality of computing nodes, the management node is configured to perform the operations performed by the management node in the first aspect or any possible implementation manner of the first aspect, and any one of the plurality of computing nodes is configured to perform the operations performed by the first computing node in the first aspect or any possible implementation manner of the first aspect.

[0024] In a third aspect, the present application provides a distributed computing method applied to a computing node in a distributed computing system, the distributed computing system comprises a management node and a plurality of computing nodes, and the method comprises the following steps:

[0025] The first computing node obtains a first fusion operator issued by the management node, the first fusion operator comprises a first matrix and a second matrix, and an instruction for instructing the first computing node to implement a matrix multiplication operation of the first matrix and the second matrix; after the first computing node receives the first fusion operator, a plurality of matrix computing units in the first computing node respectively perform part of the matrix multiplication operation of the first matrix and the second matrix according to the first fusion operator to obtain a first calculation result; then, a communication task between the first computing node and other computing nodes in the distributed computing system is performed, the communication task is used for synchronizing the first calculation result with the other computing nodes, and the first computing node also obtains the calculation results of the other computing nodes by performing the communication task.

[0026] When the distributed computing system needs to perform distributed computing, and the computing node needs to perform a matrix multiplication operation, the management node only needs to issue a fusion operator to the computing node, the fusion operator comprises an instruction for implementing the matrix multiplication operation, that is, the fusion operator comprises complete computing logic and communication logic for performing the matrix multiplication operation, and the computing node can complete a matrix multiplication operation based on the fusion operator. The management node does not need to split the matrix multiplication operation into a plurality of sub-tasks for calculation and communication, and then send the sub-tasks to the computing node one by one, which can avoid the phenomenon of idle computing node resources caused by the task splitting and issuing of the management node, improve the utilization rate of resources and the efficiency of the distributed computing system.

[0027] In a possible implementation, each matrix calculation unit in the first computing node needs to perform m rounds of partial operations of the first matrix and the second matrix, that is, each matrix calculation unit completes partial calculation in the matrix multiplication operation of the first matrix and the second matrix in each round of calculation. The first fusion operator further includes sub-matrix information and a communication trigger parameter n. The sub-matrix information is used to indicate the size of the first sub-matrix and the size of the second sub-matrix participating in calculation when each round of operation is performed by one matrix calculation unit. The communication trigger parameter n is less than or equal to m, and m is a positive integer greater than 1. The plurality of matrix calculation units in the first computing node respectively perform partial operations in the matrix multiplication operation of the first matrix and the second matrix according to the first fusion operator, to obtain a first calculation result, including: each matrix calculation unit in the first computing node acquires two sub-matrices from the first matrix and the second matrix according to the sub-matrix information in each round of calculation, performs partial calculation of the matrix multiplication operation of the first matrix and the second matrix, performs partial operations of the matrix multiplication operation in parallel through the plurality of matrix calculation units, until n rounds of calculation are performed, the first calculation result is obtained, and then the first computing node starts to perform a communication task with other computing nodes, and synchronizes the results obtained after n rounds of calculation by each computing node.

[0028] In a possible implementation, the method further includes: when the first computing node performs the communication task with other computing nodes in the plurality of computing nodes, the plurality of matrix calculation units start to perform the n+1th round of calculation, that is, the first computing node can continue to perform calculation through the matrix calculation units while performing the communication task, parallel execution of calculation and communication is realized, and resource utilization and efficiency of the distributed computing system are improved.

[0029] In a possible implementation, the first fusion operator further includes an output address, and the output address is used to indicate an address of a shared memory space of the first computing node. The method further includes: the first computing node stores the first calculation result in the shared memory space according to the output address, and the shared memory space can be read and written by other computing nodes.

[0030] In a possible implementation, the first computing node stores the first calculation result in the shared memory space according to the output address, including: when each matrix calculation unit in the first computing node stores multiple rows of data included in a result matrix obtained by performing one round of calculation in the shared memory space in sequence, an i+1th row of the result matrix and an ith row are stored continuously in the shared memory space.

[0031] In a possible implementation, the first fusion operator further includes a carry granularity, the carry granularity being used to indicate an amount of data fetched from the shared memory space of another computing node by the first computing node executing the fetch instruction once; the first computing node performs a communication task with other computing nodes in the plurality of computing nodes, including that the first computing node executes the fetch instruction multiple times to fetch the computing result of each computing node from the shared memory space of the other computing node according to the carry granularity.

[0032] In a possible implementation, the first computing node performs the communication task with other computing nodes based on the memory access instruction.

[0033] In a possible implementation, the first computing node performs the communication task with other computing nodes in the plurality of computing nodes, including that the first computing node queries a flag bit of each computing node in the other computing nodes, the flag bit being used to indicate whether the computing node completes n rounds of computation; the first computing node determines that the flag bit of each computing node is the first value, and performs the communication task with other computing nodes in the plurality of computing nodes; and the first value of the flag bit is used to indicate that the computing node has completed n rounds of computation.

[0034] In a possible implementation, the above matrix multiplication operation is a calculation in an artificial intelligence (AI) model training process; and the computing node can be a GPU, NPU, TPU or other special AI processing chip.

[0035] In a fourth aspect, the present application provides a computing node applied to a distributed computing system, the distributed computing system including a management node and a plurality of computing nodes, for any one of the plurality of computing nodes, for example, a first computing node, the first computing node including a plurality of matrix computing units and a plurality of vector computing units, wherein the plurality of matrix computing units are used to obtain a first fusion operator issued by the management node; the first fusion operator including a first matrix and a second matrix, and an instruction used to indicate that the first computing node implements a matrix multiplication operation of the first matrix and the second matrix; according to the first fusion operator, partial operations in the matrix multiplication operation of the first matrix and the second matrix are respectively performed to obtain a first computing result; and the plurality of vector computing units are used to perform a communication task with other computing nodes in the plurality of computing nodes, the communication task being used to synchronize the first computing result with the other computing nodes.

[0036] In a possible implementation, each matrix calculation unit in the first computing node needs to perform m rounds of partial operations of the first matrix and the second matrix, that is, each matrix calculation unit completes partial calculation in the matrix multiplication operation of the first matrix and the second matrix in each round of calculation. The first fusion operator further includes sub-matrix information and a communication trigger parameter n. The sub-matrix information is used to indicate the size of the first sub-matrix and the size of the second sub-matrix participating in calculation when each round of operation is performed by one matrix calculation unit. The communication trigger parameter n is less than or equal to m, and m is a positive integer greater than 1. The plurality of matrix calculation units are configured to perform partial operations of the matrix multiplication operation in parallel according to the sub-matrix information, until n rounds of calculation are completed, to obtain the first calculation result, and then start to perform a communication task between the vector calculation unit and other computing nodes, and synchronize the results obtained after n rounds of calculation by each computing node.

[0037] In a possible implementation, the plurality of matrix calculation units are further configured to start to perform the (n+1)th round of calculation when the plurality of vector calculation units perform the communication task with other computing nodes in the plurality of computing nodes, that is, the first computing node can continue to perform calculation through the matrix calculation unit while performing the communication task.

[0038] In a possible implementation, the first computing node further includes a memory, and the memory includes a shared memory space. The first fusion operator further includes an output address, and the output address is used to indicate an address of the shared memory space of the memory.

[0039] The plurality of matrix calculation units are further configured to store the first calculation result in the shared memory space according to the output address, where the shared memory space can be read and written by other computing nodes.

[0040] In a possible implementation, the plurality of matrix calculation units are specifically configured to sequentially store a plurality of rows of data included in a result matrix obtained by performing one round of calculation in the shared memory space, where an (i+1)th row of the result matrix is stored continuously with an ith row in the shared memory space.

[0041] In a possible implementation, the first fusion operator further includes a carrying granularity, and the carrying granularity is used to indicate an amount of data acquired from the shared memory space of another computing node by the first computing node once according to an acquisition instruction. The plurality of vector calculation units are specifically configured to execute the acquisition instruction multiple times, and acquire the calculation result of each computing node from the shared memory space of another computing node according to the carrying granularity.

[0042] In a possible implementation, the plurality of vector calculation units perform the communication task with other computing nodes based on a memory access instruction.

[0043] In a possible implementation, the plurality of vector calculation units are configured to: query a flag bit of each of the other calculation nodes, the flag bit being used to indicate whether the calculation node has completed the n rounds of calculation; and determine, by the first calculation node, that the flag bit of each of the other calculation nodes is a first value, and perform a communication task with the other calculation nodes in the plurality of calculation nodes; where the first value of the flag bit is used to indicate that the calculation node has completed the n rounds of calculation.

[0044] In a possible implementation, the matrix multiplication operation is a calculation in an artificial intelligence (AI) model training process, and the calculation node can be a GPU, an NPU, a TPU, or another special AI processing chip.

[0045] In a fifth aspect, the present application provides a computing device, which includes a processor, a plurality of calculation nodes, and a memory, the memory being configured to store instructions, and the processor being configured to execute the instructions stored in the memory to implement the method performed by the management node in the first aspect or any possible implementation of the first aspect, and any one of the plurality of calculation nodes is configured to perform the operation performed by the first calculation node in the first aspect or any possible implementation of the first aspect. The calculation node can be a GPU, an NPU, a TPU, or another special AI processing chip.

[0046] In a sixth aspect, the present application provides a computer-readable storage medium, which includes computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster implements the method performed by the calculation node in the first aspect or any possible implementation of the first aspect.

[0047] In a seventh aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is run by a computing device cluster, the computer program implements the method performed by the calculation node in the first aspect or any possible implementation of the first aspect.

[0048] On the basis of the implementation manners of the aspects described above, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0049] FIG. 1 is a schematic diagram of an implementation of a matrix multiplication operation;

[0050] FIG. 2 is a schematic diagram of a distributed computing system provided by the present application;

[0051] FIG. 3 is a schematic diagram of a calculation node provided by the present application;

[0052] FIG. 4 is a schematic diagram of a distributed computing method provided by the present application;

[0053] Fig. 5 is a schematic diagram of data flow of a calculation result in a storage area according to an embodiment of the present application;

[0054] Fig. 6 is a schematic diagram of data flow of another calculation result in a storage area according to an embodiment of the present application;

[0055] Fig. 7 is a schematic diagram of storage of a calculation result according to an embodiment of the present application;

[0056] Fig. 8 is a schematic diagram of a communication process according to an embodiment of the present application;

[0057] Fig. 9 is a schematic diagram of storage of another calculation result according to an embodiment of the present application;

[0058] Fig. 10 is a schematic diagram of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0059] The distributed computing method according to an embodiment of the present application will be described below with reference to the accompanying drawings.

[0060] With the development of information technology, processing of large-scale data has become a focus in various fields. The processing of large-scale data usually requires a distributed computing system to perform distributed computing. For example, in the field of artificial intelligence (AI), distributed training is used to improve the training speed of an AI model. The strategies of distributed training include data parallelism (SP), model parallelism (MP), sequence parallelism (SP), and mixture of experts (MoE) parallelism. Model parallelism includes tensor parallelism (TP) and pipeline parallelism (PP).

[0061] When training a model by multiple computing nodes of a distributed computing system, for one computing node, the tasks to be completed by the computing node include multiple computing tasks and multiple communication tasks, each computing node performs a computing task and then performs a communication task after completing the computing task, the communication task includes synchronizing data calculated by different computing nodes through collective communication, and after completing data synchronization, each computing node performs a subsequent computing task. The data sent by the communication task is the calculation result obtained by each computing node before performing the computing task, and the computing task of each computing node after the communication task depends on the data obtained in the communication task, so the computing task and the communication task depend on each other, which are dependent computing and communication tasks. The above dependent computing and communication tasks must be executed in series, so that the computing task cannot be executed during the execution of the communication task, and the communication task cannot be executed during the execution of the computing task, and the computing resources and communication resources of the distributed computing system are not fully utilized, which affects the training efficiency. The above collective communication includes allreduce communication, ring-allreduce communication, all2all communication, reduce_scatter communication, allgather communication, etc. For example, allreduce communication or ring-allreduce communication is used in data parallelism and tensor parallelism, and all2all communication is used in MoE parallelism.

[0062] In the process of AI model training, model parallelism is mainly for matrix multiplication in model training, and matrix multiplication is the main operation in the process of AI model training. For example, in the forward propagation training phase of the AI model, matrix multiplication between two matrices is usually involved, such as matrix multiplication between activation value data and weight data in convolutional neural networks, recurrent neural networks or feedforward neural networks. For another example, in a transformer model, multiple self-attention structures and multiple feed-forward networks (FFN) are included, each self-attention structure and each FFN involves two matrix multiplications. Model parallelism mainly uses the principle of matrix block multiplication to divide matrix multiplication into multiple computing nodes for calculation to improve computing efficiency.

[0063] As shown in FIG. 1, which is a schematic diagram of implementing matrix multiplication. If a matrix multiplication of Y = X x Z is to be implemented, where X is an M x K matrix, Z is a K x N matrix, and Y is the result matrix of the matrix multiplication, when the above matrix multiplication is implemented by two computing nodes, the matrix X can be divided into an M x K / 2 matrix X1 and an X2 matrix, and the matrix Y can be divided into two K / 2 x N matrices Y1 and Y2. Then the computing task completed by the first computing node is a matrix multiplication of Y1 = X1 x Z1, and the computing task completed by the second computing node is a matrix multiplication of Y2 = X2 x Z2; after both computing nodes complete the computing tasks, the computing results in the two computing nodes are synchronized by means of collective communication, so that each computing node includes Y1 and Y2, then Y1 and Y2 are added to obtain the above matrix Y, and the next computing task is continued according to the matrix Y, and after the next computing task is completed, the next communication task is executed.

[0064] However, the computing tasks and the communication tasks performed between the above computing nodes are in a serial execution mode, the computing nodes must perform the communication tasks after completing the computing tasks, data synchronization is performed, and after the communication tasks are performed and the data synchronization is implemented, the next computing task is executed, so that the distributed computing system is affected by the communication delay of the communication tasks, resulting in a relatively low overall computing efficiency. In view of the above problem, the current main method for improving the overall computing efficiency is a parallel masking strategy of computing and communication. A computing task that needs to be executed by a computing node is divided into multiple sub-computing tasks, after each sub-computing task is executed to obtain a partial computing result, the partial computing result is synchronized, and the next sub-computing task is continued to be executed at the same time, so as to realize the parallel of computing and communication. In this method, each time a sub-computing task is executed, only part of the data in the computing result can be obtained, and the computing result originally synchronized by one communication task also needs to be synchronized multiple times, that is, one communication task is divided into multiple sub-communication tasks. For ease of description, the sub-computing tasks and the sub-communication tasks are collectively referred to as sub-tasks. By dividing a computing task and a communication task into more fine-grained sub-tasks, the parallel of computing and communication is realized to improve the efficiency of the distributed computing system.

[0065] However, in the above calculation and communication parallel masking method, the management node needs to split a calculation task and a communication task that a calculation node needs to perform into multiple subtasks, and then schedules the calculation node to perform each subtask by issuing the subtasks to the calculation node. However, no matter how much data needs to be calculated or how much data needs to be communicated for a subtask, the time for the management node to perform data preprocessing, loading, and issuing of the subtask before issuing the subtask is constant. If the management node splits the task of a calculation node into Q subtasks, the time overhead of the management node will be Q times without splitting, and the amount of task of each subtask of the calculation node will decrease, and the required calculation time or communication time will decrease. With the increase of Q, the time for the calculation node to perform a subtask will decrease, and the time for the management node to issue a subtask is constant, so this will make the time for the management node to issue a task to the calculation node greater than the time for the calculation node to perform a subtask, resulting in the situation that the calculation node resource is idle and waits for the management node to issue a task.

[0066] To solve the above problem, the present application provides a distributed calculation method. For a matrix multiplication operation that a calculation node needs to perform, the management node only needs to issue a fusion operator to the calculation node, and the fusion operator includes instructions for implementing matrix multiplication. That is, the fusion operator includes complete calculation logic and communication logic for performing matrix multiplication calculation, and the calculation node can complete a matrix multiplication calculation based on the fusion operator. The management node does not need to split a matrix multiplication operation into multiple subtasks for calculation and communication, and then send the subtasks to a calculation node one by one, which can avoid the phenomenon of idle calculation node resources caused by task splitting and issuing by the management node, improve the utilization of resources and the efficiency of the distributed calculation system.

[0067] The distributed calculation system provided by the present application is introduced below. Referring to FIG. 2, FIG. 2 is a schematic diagram of a distributed calculation system provided by the present application. The distributed calculation system includes a management node and multiple calculation nodes. The management node and the multiple calculation nodes are connected through an interconnection link, and the multiple calculation nodes are connected through an interconnection link. The management node is used to issue a fusion operator to each calculation node in the distributed calculation system, and the fusion operator is used to instruct the calculation node to implement matrix multiplication of two matrices. Each calculation node in the multiple calculation nodes performs matrix multiplication between two matrices and communication with other calculation nodes according to the received fusion operator. The method for the calculation node to implement matrix multiplication and communication with other calculation nodes according to the received fusion operator is described in detail below.

[0068] Referring to FIG. 3, FIG. 3 is a schematic diagram of a computing node provided by the present application. The distributed computing system includes a plurality of computing nodes as shown in FIG. 3, each of which includes a plurality of matrix computing units for performing matrix operations, such as matrix multiplication, and at least one vector computing unit for performing vector operations, such as vector addition and vector multiplication, and can also be used to obtain data from other nodes, such as reading data from the memory of other computing nodes to the local computing node. Each computing node also includes a memory and a cache. The memory is used to store data required by the computing unit during calculation, as well as to store the calculation results obtained after the computing unit performs calculation. The cache is used to exchange data between the computing unit (matrix computing unit and vector computing unit) and the memory. The computing unit reads data from the memory, first reads the data into the cache, and then reads from the cache to the computing unit. The computing result obtained by the computing unit is first written to the cache, and then written from the cache to the memory.

[0069] The distributed computing system described above also includes a management node, which can be a host or a central processing unit (CPU), and the computing nodes described above can be graphics processing units (GPU), neural network processing units (NPU), tensor processing units (TPU) or other special AI processing chips. The plurality of computing nodes can be located in the same computing device, or can be located in different computing devices. If the plurality of computing nodes are located in the same computing device, the plurality of computing nodes can be connected to each other through an interconnection link, which can be a bus, and the bus can be a peripheral component interconnect express (PCIE) bus, an NVLink bus or other bus; the interconnection link can also be a switch chip, such as an NVSwitch chip or other chip capable of connecting multiple computing nodes and having data communication function, and the present application does not make specific limitation on the topology between the plurality of computing nodes. If the plurality of computing nodes are located in a plurality of different computing devices, each computing device includes at least one computing node, and the plurality of computing devices are connected through an interconnection link, and the computing nodes in the plurality of computing devices can communicate through the interconnection link, and the present application does not make specific limitation on the topology between the plurality of computing devices.

[0070] When the plurality of computing nodes are located in the same computing device, the device management node can be located in the same computing device as the plurality of computing nodes, and the management node and the plurality of computing nodes are connected through a bus; or the management node and the plurality of computing nodes can be located in different computing devices. When the plurality of computing nodes are located in a plurality of computing devices, the management node can be located in one of the computing devices, or can be located in other computing devices and communicate with the plurality of computing devices through an interconnection link.

[0071] The memory can be a random access memory (RAM), a dynamic random access memory (DRAM), such as a high bandwidth memory (HBM), or a static random access memory (SRAM), a synchronous dynamic random access memory (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a synchlink DRAM (SLDRAM), or a direct rambus RAM (DR RAM).

[0072] In the present application, the memory includes a standard address space and a shared memory space, the standard address space is used to store data required by the computing nodes to perform matrix multiplication operations, and the shared memory space is used to store results obtained by the computing nodes in the process of performing matrix multiplication operations, and the shared memory space of each computing node can be accessed and read and written by other computing nodes. When the computing node writes the result obtained in the process of performing matrix multiplication operations to the shared memory space, the computing result is first written to a cache, and then the computing result is written from the cache to the shared memory space.

[0073] The distributed computing method provided in the present application will be introduced below. As shown in FIG. 4, FIG. 4 is a schematic diagram of a distributed computing method provided in the present application. The method includes the following S401-S403.

[0074] S401. The management node issues a first fusion operator to a first computing node.

[0075] The distributed computing system includes a management node and a plurality of computing nodes. When a large-scale computation needs to be performed by the distributed computing system, the management node respectively issues a fusion operator to the plurality of computing nodes. Each fusion operator includes two matrices that need to be computed by a corresponding computing node and an instruction for implementing matrix multiplication of the two matrices. For example, the management node issues a first fusion operator to a first computing node in the plurality of computing nodes. The first fusion operator includes a first matrix and a second matrix and an instruction for implementing matrix multiplication of the first matrix and the second matrix.

[0076] It should be noted that each computing node in the distributed computing system receives a fusion operator for performing matrix multiplication of two matrices. The method for performing matrix multiplication by each computing node and the method for communicating with other computing nodes are the same as those of the first computing node. The following describes the distributed computing method provided by the present application by taking the first computing node as an example.

[0077] The large-scale computation described above can be training of an AI model. The first matrix and the second matrix can be part of data in two parameter matrices that need to perform matrix multiplication in the AI model training process. For example, the first matrix is part of data in an activation matrix, and the second matrix is part of data in a weight matrix. The large-scale computation described above can also be a computation of other applications. The matrix multiplication that needs to be performed by each computing node is the computation logic included in the large-scale computation.

[0078] S402. The first computing node receives the first fusion operator issued by the management node and performs part of the matrix multiplication of the first matrix and the second matrix by the plurality of matrix calculation units to obtain a first calculation result.

[0079] The first computing node receives the first fusion operator issued by the management node. The plurality of matrix calculation units included in the first computing node respectively perform part of the matrix multiplication of the first matrix and the second matrix according to the first fusion operator to obtain a first calculation result.

[0080] The following specifically describes the process in which the plurality of matrix calculation units in the first computing node perform part of the matrix multiplication of the first matrix and the second matrix to obtain a first calculation result.

[0081] In the present application, the first fusion operator includes a first matrix A and a second matrix W and matrix information of the first matrix A and the second matrix W. The matrix information indicates the size of the first matrix A and the second matrix W. For example, the first matrix A is an MxK matrix, and the second matrix W is a KxN matrix.

[0082] In the present application, the first computing node performs the matrix multiplication operation of the first matrix A and the second matrix W through the plurality of matrix calculation units included in the first computing node, and thus each matrix calculation unit is used to calculate a partial operation in the matrix multiplication operation of the first matrix A and the second matrix W. For example, the matrix multiplication operation of the first matrix A and the second matrix W is decomposed into a plurality of groups of sub-matrix multiplication operations, each group of sub-matrix includes one sub-matrix obtained from the first matrix A and one sub-matrix obtained from the second matrix W, and each matrix calculation unit performs multiplication operation of partial sub-matrices, i.e., each matrix calculation unit respectively performs a partial operation in the matrix multiplication operation of the first matrix A and the second matrix W.

[0083] For example, when the first computing node performs the matrix multiplication operation of the first matrix A and the second matrix W, each matrix calculation unit obtains one row of data from the first matrix A as a first sub-matrix and obtains one column of data from the second matrix W as a second sub-matrix each time, and one round of calculation of each matrix calculation unit realizes the matrix multiplication operation of the first sub-matrix and the second sub-matrix, which is equivalent to dividing the first matrix A and the second matrix W into a plurality of sub-matrices respectively, taking one sub-matrix in the first matrix A and one sub-matrix in the second matrix W as a group of sub-matrices, and calculating the matrix multiplication operation of each group of sub-matrices respectively, and each matrix calculation unit performs the matrix multiplication operation of a plurality of groups of sub-matrices.

[0084] For example, the first matrix A is divided into 8 sub-matrices according to rows, which are A1, A2…A8 respectively, and the size of each sub-matrix is M / 8×K; the second matrix W is divided into 10 sub-matrices according to columns, which are W1, W2…W10 respectively, and the size of each sub-matrix is K×(N / 10), and then the matrix multiplication between the first matrix A and the second matrix W can be expressed as the following formula 1:

[0085] Wherein, the matrix O1 is the result of the matrix multiplication operation of the first matrix A and the second matrix W. As can be seen from the above formula, the matrix multiplication operation between the first matrix A and the second matrix W is converted into 80 groups of sub-matrix multiplication operations. The matrix multiplication operation between a group of sub-matrices refers to the matrix multiplication between one sub-matrix in the first matrix A and one sub-matrix in the second matrix W, such as A1×W1 in the formula 1. If the first computing node includes 5 matrix calculation units, and each matrix calculation unit completes the matrix multiplication operation of 16 groups of sub-matrices, the matrix multiplication operation of the first matrix A and the second matrix W can be realized.

[0086] In the present application, the first fusion operator further comprises sub-matrix information, the sub-matrix information being used to indicate the size of the first sub-matrix and the size of the second sub-matrix participating in one round of operation when one matrix calculation unit performs one round of operation. For example, the sub-matrix information comprises three parameters M1, K1 and N1, and the sub-matrix information is used to indicate that one matrix calculation unit obtains M1×K1 first sub-matrix from the first matrix A and K1×N1 second sub-matrix from the second matrix W each time, wherein M is an integer multiple of M1 and N is an integer multiple of N1. Wherein, one round of operation performed by one matrix calculation unit refers to that the matrix calculation unit obtains the first sub-matrix from the first matrix A, obtains the second sub-matrix from the second matrix W, and performs matrix multiplication of the first sub-matrix and the second sub-matrix to obtain a result matrix.

[0087] For example, if the plurality of matrix calculation units of the first calculation node perform matrix multiplication of the first matrix A and the second matrix W in the manner of the above (formula 1), the sub-matrix information indicates that one matrix calculation unit obtains M / 8×K first sub-matrix from the first matrix A and K×(N / 10) second sub-matrix from the second matrix W each time, one matrix calculation unit obtains M / 8×K first sub-matrix from the first matrix A and K×(N / 10) second sub-matrix from the second matrix W each time, and performs matrix multiplication of the first sub-matrix and the second sub-matrix to obtain M / 8×(N / 10) result matrix.

[0088] When the plurality of matrix calculation units of the first calculation node perform matrix multiplication of the first matrix A and the second matrix W according to the first fusion operator, one matrix calculation unit obtains one first sub-matrix from the first matrix A and one second sub-matrix from the second matrix W according to the sub-matrix information, and then calculates matrix multiplication of the first sub-matrix and the second sub-matrix to obtain a result matrix.

[0089] If the matrix multiplication of the first matrix A and the second matrix W is implemented in the manner corresponding to the above formula 1 and the related description, the first calculation node needs to perform matrix multiplication of 80 groups of sub-matrices in total, and if the first calculation node comprises 5 matrix calculation units, each matrix calculation unit needs to perform matrix multiplication of 16 groups of sub-matrices, i.e. each matrix calculation unit needs to perform 16 rounds of calculation to implement the matrix multiplication of the first matrix A and the second matrix W and obtain the calculation result of the first matrix A and the second matrix W.

[0090] It should be noted that the application does not limit the parameters in the sub-matrix information, that is, the application does not limit the division manner of the first matrix A and the second matrix W. For example, the first computing node can also divide the sub-matrix for only one of the matrices, such as dividing the first matrix A by rows, and not dividing the second matrix W; or not dividing the first matrix A, and dividing the second matrix W by columns. Or divide the first matrix A into M1xK1 sub-matrices, and divide the second matrix W into K1xN1 sub-matrices.

[0091] In the application, the first fusion operator further includes a communication trigger parameter n, each matrix calculation unit of the first computing node respectively acquires a sub-matrix from the first matrix A and the second matrix W according to the sub-matrix information, completes one round of calculation, and after each matrix calculation unit in the first computing node performs n rounds of calculation, completes part of the operation in the matrix multiplication operation between the first matrix and the second matrix, obtains the first calculation result, and then triggers a communication task. The communication task is used to synchronize the first calculation result between the first computing node and other computing nodes in the plurality of computing nodes in the distributed computing system. For example, the first computing node includes 5 matrix calculation units, each matrix calculation unit needs to perform 16 rounds of calculation, and if the value of the above n is 4, each matrix calculation unit in the first computing node triggers the communication task after completing 4 rounds of calculation.

[0092] If each matrix calculation unit in the first computing node needs to complete m rounds of calculation to complete the matrix multiplication operation between the first matrix A and the second matrix W, the above communication trigger parameter n is less than or equal to m. For example, the first computing node needs to perform matrix multiplication operation of 80 groups of sub-matrices in total, if the first computing node includes 5 matrix calculation units, each matrix calculation unit needs to perform matrix multiplication operation of 16 groups of sub-matrices, the value of m is 16, and n is a positive integer less than or equal to 16.

[0093] When the communication trigger parameter n is less than m, it indicates that the plurality of matrix calculation units of the first computing node only need to complete part of the matrix multiplication operation of the first matrix A and the second matrix W to trigger the communication task and perform the communication task with other computing nodes to synchronize the first calculation result. In the application, when each matrix calculation unit in the first computing node performs n rounds of calculation, the first computing node performs the communication task with other computing nodes, each matrix calculation unit of the first computing node starts the (n+1)th round of calculation, and each calculation unit in the first computing node synchronously performs calculation in the process of the communication task of the first computing node, so that the calculation and the communication are parallel, the utilization rate of the calculation resource and the communication resource of the first computing node is improved, and the calculation efficiency is also improved.

[0094] In the present application, the first fusion operator further includes an output address, which is used to indicate the storage location of the first calculation result of the first calculation node in the memory, wherein the output address is the address of the shared memory space of the first calculation node. Each matrix calculation unit directly writes the result matrix into the shared memory space of the first calculation node, wherein the shared memory space refers to an area that can be accessed and read and written by other calculation nodes, so that other calculation nodes can obtain the first calculation result of the first calculation node.

[0095] Referring to FIG. 5, FIG. 5 is a schematic diagram of data flow of a calculation result in a storage area according to the present application. FIG. 5 includes two calculation nodes: a first calculation node and a second calculation node, and the memory of each calculation node includes a standard address space and a shared memory space. In the method for implementing matrix multiplication of two matrices by issuing sub-tasks to the calculation nodes by the management node, after each matrix calculation unit of the first calculation node obtains a calculation result by executing a sub-task, the calculation result is written into the standard address space through the cache. When executing a communication task, the first calculation node needs to copy the to-be-transmitted data obtained by the plurality of matrix calculation units from the standard address space to the shared memory space, so that the to-be-transmitted data can be read by other nodes to complete the communication task. The to-be-transmitted data includes the calculation results obtained by the plurality of matrix calculation units. The second calculation node obtains the to-be-transmitted data from the shared memory space of the first calculation node, writes the to-be-transmitted data obtained from the first calculation node into the shared memory space of the second calculation node, and then the vector calculation unit of the second calculation node reads the to-be-transmitted data from the shared memory space of the second calculation node and the data obtained by the second calculation node by executing a sub-task, aggregates the data to obtain an aggregated result, and writes the aggregated result into the standard address space of the second calculation node through the cache.

[0096] Referring to FIG. 6, FIG. 6 is a schematic diagram of data flow of a calculation result in a storage area according to the present application. In the present application, the fusion operator includes instructions for implementing multiplication of two matrices, that is, the present application instructs the calculation node to complete calculation and communication at the instruction level, and therefore the matrix calculation unit and the vector calculation unit in the present application can read and write the memory through a memory tagging extension (MTE) instruction, such as a store / load memory access instruction. In the fusion operator, the address to which the matrix calculation unit writes the data after calculation can be specified, and therefore the calculation result of each execution of the matrix calculation unit can be directly written into the shared memory space.

[0097] The first fusion operator issued by the management node includes an output address. After each matrix calculation unit of the first calculation node performs matrix multiplication operation on a round of sub-matrices to obtain a result matrix, the result matrix is written into the shared memory space through the cache by a store instruction. When the second calculation node needs to obtain the first calculation result of the first calculation node, the vector calculation unit of the second calculation node obtains the first calculation result from the shared memory space of the first calculation node through a load instruction. Then, after the vector calculation unit of the second calculation node aggregates the first calculation result and data obtained by the second calculation node performing n rounds to obtain an aggregated result, the aggregated result is written into the standard address space of the second calculation node through the cache.

[0098] Compared with the method in which the calculation result of the current calculation node is first written into the standard address space and then copied from the standard address space to the shared memory space, so as to be obtained by other calculation nodes from the shared memory space, the method of the present application can reduce the data copying operation and improve the communication efficiency.

[0099] It can be understood that when the matrix calculation unit writes the result matrix into the shared memory space, the result matrix is first written into the cache, and then the result matrix is written from the cache into the shared memory space. If the result matrix is still in the cache when the calculation node completes n rounds of calculation and performs the communication task, other nodes can obtain the result matrix from the cache, that is, other calculation nodes can obtain the first calculation result from the cache of the first calculation node.

[0100] In the present application, when each matrix calculation unit performs matrix multiplication operation on a group of sub-matrices to obtain a result matrix, the result matrix of the matrix multiplication of the same group of sub-matrices is written into the continuous memory space of the shared memory space. For example, if the first calculation node completes the matrix multiplication between the first matrix A and the second matrix W in the manner corresponding to (Formula 1), the result matrix of the matrix multiplication of each group of sub-matrices is an M / 8×(N / 10) matrix. When a matrix calculation unit performs matrix multiplication operation and writes the calculated result matrix into the shared memory space, the first row data of the result matrix is first written into the shared memory space, and then the second row data is continuously stored with the first row data, that is, the i+1th row data of the result matrix is stored in the continuous memory space with the ith row data. Wherein, i+1 is less than or equal to M / 8 and i is a positive integer. In this way, in the process of performing the communication task, when other calculation nodes obtain the result matrix of the matrix multiplication of a group of sub-matrices, the result matrix can be obtained from the continuous memory space, without the need to obtain from different memory spaces, so as to improve the communication efficiency.

[0101] For example, as shown in FIG. 7, FIG. 7 is a schematic diagram of storing the calculation result provided by the present application. If one matrix calculation unit of the first calculation node completes the matrix multiplication of the above-mentioned sub-matrix A1 and W1 in one round of calculation, and obtains matrix O1 11 , and completes the matrix multiplication of A1 and W2 in another round of calculation, and obtains matrix O2 12 , although the first row data of matrix O1 11 and the first row data of matrix O2 12 belong to the same row data in matrix O1, and the last value of the first row data of matrix O1 11 and the first value of the first row data of matrix O2 12 are adjacent two elements in matrix O1, as shown in FIG. 7, value a1 14 and value a2 15 are adjacent two values in matrix O1. However, when writing matrix O1 11 and matrix O2 12 into the shared memory space, matrix O1 11 is written into the continuous storage space in the shared memory space, that is, the first value of the i+1th row of matrix O1 11 is adjacent to the last value of the ith row, and then matrix O2 12 is written into the continuous storage space in the shared memory space. As shown in FIG. 7, value a1 14 and value a2 21 are adjacent stored, value a3 24 and value a4 31 are adjacent stored, value a5 34 and value a6 41 are adjacent stored, and value a7 44 and value a8 15 are adjacent stored.

[0102] It should be noted that the present application does not limit the matrix multiplication of the multiple groups of sub-matrices performed by each matrix calculation unit in the calculation node, that is, each matrix calculation unit randomly performs the matrix multiplication of the multiple groups of sub-matrices, and the order of performing the matrix multiplication of the multiple groups of sub-matrices by each matrix calculation unit is not limited. For example, the five matrix calculation units in the above-mentioned first calculation node need to complete the matrix multiplication of 80 groups of sub-matrices, each matrix calculation unit needs to perform 16 rounds of calculation, that is, each matrix calculation unit needs to perform the matrix multiplication of 16 groups of sub-matrices, the present application does not limit which 16 groups of sub-matrices are performed by each matrix calculation unit, and the order of performing the matrix multiplication of 16 groups of sub-matrices by one matrix calculation unit is not limited.

[0103] S403. The first calculation node performs a communication task with other calculation nodes, and synchronizes the first calculation result.

[0104] In the present application, the fusion operator further comprises a communication domain, the communication domain comprising the identification of the plurality of computing nodes that need to exchange data with each other, and the first computing node being able to determine the other computing nodes that need to exchange data when performing the communication task according to the communication domain.

[0105] After completing n rounds of computation, the computing node initiates the communication task, and the vector computation unit of the computing node first queries whether the other computing nodes in the communication domain have also completed n rounds of computation according to the communication domain in the fusion operator. If the other computing nodes in the communication domain have all completed n rounds of computation, the plurality of computing nodes in the communication domain start to perform the communication task to exchange data. For example, after completing a round of computation, the first computing node initiates the communication task, and queries whether the other computing nodes in the communication domain have completed the round of computation. If the first computing node determines that the other computing nodes in the communication domain have all completed the round of computation, the first computing node starts to perform the communication task.

[0106] In the present application, each computing node is provided with a flag bit, which is used to indicate whether a computing node has completed N rounds of computation. The flag bit has two values, the first value indicating that the computing node has completed a round of computation, and the second value indicating that the computing node has not completed a round of computation. After receiving the fusion operator, the first computing node first sets the flag bit to the second value by the vector computation unit, and then performs the matrix multiplication of the first matrix A and the second matrix W according to the above method. After completing n rounds of matrix multiplication of the sub-matrices, the first computing node sets the flag bit to the first value by the vector computation unit. In the present application, the flag bit can be represented by a bit, the first value being 1 and the second value being 0. After completing n rounds of matrix multiplication of the sub-matrices, each matrix computation unit in the first computing node sets the flag bit to 1 by the vector computation unit.

[0107] After setting the flag bit to 1, the vector computation unit of the first computing node queries the value of the flag bit of the other computing nodes in the communication domain at a first period to determine whether the other computing nodes in the communication domain have completed the round of computation. Similarly, any computing node in the communication domain also sets the flag bit to 1 after completing n rounds of computation, and then queries the flag bit of the other computing nodes at a first period to determine whether the other computing nodes have completed the round of computation. In the case that each computing node in the same communication domain determines that the other computing nodes have completed the round of computation, the computing nodes in the communication domain start to perform the communication task by the vector computation unit. For example, data aggregation is performed by means of set communication to synchronize the respective computation results, which can avoid the case that the other computing nodes in the communication domain have not completed n rounds of computation, and thus the communication leads to computation errors.

[0108] In the present application, when each computing node in the same communication domain performs a communication task through the vector computing unit, each computing node also starts the next round of calculation through the matrix computing unit, that is, starts to perform the n+1th round of calculation. It should be understood that when each computing node performs the next round of calculation, the shared memory space of each computing node also saves the calculation results of the previous n rounds, so the computing node needs to write the calculation results obtained by the next round of calculation to other shared memory spaces.

[0109] In the present application, the above-mentioned first fusion operator also includes a carrying granularity, which refers to the amount of data that can be obtained from the shared memory space of another computing node by the computing node executing an acquisition instruction. When the computing node performs a communication task, it mainly executes an acquisition instruction to obtain data from the shared memory space of another computing node. In the present application, when the first computing node performs a communication task, the first computing node reads data from the shared memory space of another computing node through the vector computing unit. The first computing node cannot obtain the complete calculation results obtained by another computing node executing n rounds of calculation by executing an acquisition instruction once, so the first computing node executes multiple acquisition instructions to obtain the complete calculation results in another computing node according to the carrying granularity.

[0110] For ease of description, the process of executing n rounds of calculation by the matrix computing unit in the above-mentioned computing node in the present application is referred to as executing a calculation task once. The present application masks the task with shorter execution time by the task with longer execution time to realize parallel execution of the calculation task and the communication task.

[0111] When the computing node performs a communication task, the computing node obtains data from the shared memory space of another computing node by executing an acquisition instruction. One computing node cannot obtain the complete calculation results in another computing node by executing an acquisition instruction once, so one computing node needs to execute multiple acquisition instructions to obtain the complete calculation results in another computing node. Since the computing node performs a calculation task when performing a communication task, both the communication task and the calculation task need to access memory, and both the communication task and the calculation task need to occupy resources such as outstanding resources when accessing memory. When the carrying granularity corresponding to the acquisition instruction is smaller, the calculation task is more likely to obtain memory access resources. Therefore, by pre-determining the carrying granularity in the communication task, the memory access resources occupied when executing an acquisition instruction can be controlled, so that the resources required by the calculation task and the communication task can be reasonably allocated, the calculation task and the communication task can be reasonably scheduled, efficient parallel execution of the calculation task and the communication task can be ensured, and the efficiency of the distributed computing system can be improved.

[0112] In the case of the above sub-matrix information and communication trigger parameter determination, the calculation amount of a calculation task and the communication amount required by a communication task for one transmission can be determined. According to the computing power of the computing node and the calculation amount, the theoretical calculation time length of the computing node to complete a calculation task can be determined, and the theoretical communication time length of the computing node can be determined according to the communication amount and the communication bandwidth. For the case where the theoretical communication time length is greater than the theoretical calculation time length, the carrying granularity needs to be increased to improve the communication efficiency; for the case where the theoretical communication time length is less than the theoretical calculation time length, the carrying granularity needs to be reduced, thereby realizing effective optimization of the resources occupied by the calculation task and the communication task, making the actual calculation time length and the actual communication time length more close, and avoiding the waste of bandwidth resources caused by the completion of the communication task during the calculation time length process.

[0113] In the present application, the management node can determine the carrying granularity of a computing node according to the calculation parameters and the communication parameters, wherein the calculation parameters include the calculation amount and the theoretical calculation time length of the calculation task, and the communication parameters include the communication amount and the theoretical communication time length of the communication task. The theoretical calculation time is the calculation time length predicted according to the calculation amount, the computing power and the memory bandwidth of the computing node, and the theoretical communication time length is the communication time length predicted according to the communication amount and the communication bandwidth. The carrying granularity S and the calculation parameters and the communication parameters can be represented as S=f(calculation parameters, communication parameters). f is a function fitted by a large number of sample data including the calculation parameters and the communication parameters, which can be a linear function or a nonlinear function, and the present application does not make specific limitations.

[0114] The following describes the communication method provided by the present application, taking the full-mesh networking between multiple computing nodes in the same communication domain as an example, and taking the allreduce performed by the multiple computing nodes in the same communication domain to interact data as an example, such as the allreduce implemented by the reduce_scatter and all_gather methods to interact data. If the communication domain includes four computing nodes, i.e., a first computing node R1, a second computing node R2, a third computing node R3, and a fourth computing node R4, the second computing node R2, the third computing node R3, and the fourth computing node R4 perform the matrix multiplication of two matrices of the same dimension as the first computing node R1, and the method of performing the matrix multiplication is the same as that of the first computing node R1. Each computing node divides the calculation result of the first n rounds of calculation into four parts, as shown in FIG. 8, which is a schematic diagram of a communication process provided by the present application. The four calculation results of the first computing node R1 are a1-a4, the four calculation results of the second computing node R2 are b1-b4, the four calculation results of the third computing node R3 are c1-c4, and the four calculation results of the fourth computing node R4 are d1-d4. It should be noted that in the above example, the first computing node R1 needs to complete the matrix multiplication of the first matrix A and the second matrix W, and the matrix multiplication of the first matrix A and the second matrix W needs to complete 80 groups of matrix multiplication of sub-matrices, and the calculation result of each group of matrix multiplication of sub-matrices is a matrix block, such as the matrix O in the formula 1 11 If the value of n is 4, the first computing node R1 completes 4 rounds of calculation, and completes 20 groups of matrix multiplication of sub-matrices, i.e., 20 matrix blocks. If the first calculation result obtained by the first computing node R1 performing 4 rounds of calculation is divided into four parts, each part includes 5 matrix blocks, i.e., a1-a4 each include 5 matrix blocks.

[0115] Similarly, the second computing node R2, the third computing node R3, and the fourth computing node R4 include the same number of matrix calculation units as the first computing node R1, and perform the matrix multiplication of two matrices of the same dimension, and the method of performing the matrix multiplication is the same as that of the first computing node R1.

[0116] The process of the four computing nodes performing the collective communication is divided into two steps: first, each computing node performs reduce_scatter, and each computing node obtains a share of data of other computing nodes through the vector computing unit, for example, the first computing node obtains b1, c1 and d1 to obtain a share of data e1=a1+b1+c1+d1. Wherein, the five matrix blocks included in a1, b1, c1 and d1 are in the same position in the result obtained by each computing node performing the matrix multiplication operation, for example, the five matrix blocks included in a1 are O 11 , O 12 , O 13 , O 14 and O 15 , then the five matrix blocks included in b1 are in the same position in the matrix finally obtained by the second computing node. The second computing node obtains a2, c2 and d2 to obtain data e2=a2+b2+c2+d2; the third computing node obtains a3, b3 and d3 to obtain data e3=a3+b3+c3+d3; the third computing node obtains a4, b4 and c4 to obtain data e4=a4+b4+c4+d4. It should be understood that each of the four shares of data e1, e2, e3 and e4 includes five matrix blocks.

[0117] After each computing node performs reduce_scatter, each computing node then synchronizes the share of data obtained by each computing node through reduce_scatter by means of all_gather, so that the data in each computing node is the same, and after the four computing nodes perform all_gather, each computing node includes data composed of the four shares of data e1, e2, e3 and e4, i.e., each computing node includes 20 matrix blocks.

[0118] It should be noted that when the vector computing unit obtains data from the shared memory space of each computing node, the data is directly read from the shared memory space of other computing nodes to the cache, and then the vector computing unit obtains the data from the cache and performs data aggregation. For example, the vector computing unit of the first computing node obtains data b1 from the shared memory space of the second computing node, reads b1 to the cache of the first computing node, and then the vector computing unit performs addition operation on a1 and b1. After the vector computing unit of the first computing node completes data aggregation to obtain aggregated data e1, the vector computing unit writes the aggregated data e1 to the standard address space of the first computing node, and sends the data e1 to the other three computing nodes. At the same time, the aggregated data e2, e3 and e4 sent by the other three computing nodes are received and written to the standard address space.

[0119] Figure 9 is a diagram illustrating storage of a calculation result according to the present application. The first calculation node performs a matrix multiplication operation on the first matrix A and the second matrix W, and the calculation result is the matrix O1. If the second calculation node, the third calculation node and the fourth calculation node perform the matrix multiplication operation, the calculation results are O2, O3 and O4 respectively. After data aggregation of the four calculation nodes, the calculation result is the matrix O. The matrix O includes 80 matrix blocks, and e1, e2, e3 and e4 include 20 data blocks in the matrix O. When the vector calculation unit writes the aggregated data to the standard address space, the data in the 20 data blocks included by e1, e2, e3 and e4 are written to adjacent storage areas, if the data belong to the same row and are adjacent. For example, e1 includes one data block E 11 and E 12 , E 11 According to the matrix O 11 in Figure 7, E 12 is obtained from O 12 in Figure 7, and E 11 and E 12 have the same position relationship in the matrix O 11 as in the matrix O 12 1. The values included by E 11 and E 12 are stored in the standard address space as shown in Figure 9. The first row data of E 11 and the first row data of E 12 are stored continuously; the second row data of E 11 and the second row data of E 12 are stored continuously.

[0120] After the first calculation node completes the communication task, the value of the above-mentioned flag bit is reset to the second value by the vector calculation unit, so that the matrix calculation unit can use the shared memory space occupied by the first calculation result again.

[0121] The embodiments corresponding to Figures 1-9 above describe the process of implementing a calculation task and a communication task by a calculation node. Each calculation node performs multiple calculation and communication tasks according to the above-mentioned method, and the tasks indicated by the fusion operator of each calculation node can be completed.

[0122] The present application carries instructions for performing computing tasks and communication tasks through only one operator, can realize decoupling with an AI model learning framework, can be conveniently called by different AI learning frameworks and acceleration libraries, and can be applied to different AI model learning frameworks and different scenarios. For example, the present application can be applied to a scenario of model training based on model parallelism in large model training, which includes a process of computing first and then communicating, such as performing matrix multiplication first and then performing allreduce communication; the present application can also be applied to a scenario of model training based on model parallelism and sequence parallelism in large model training, which includes a process of computing first and then communicating, such as performing matrix multiplication first and then performing reduce_scatter communication, and also includes a process of communicating first and then computing, such as performing allgather communication first and then performing matrix multiplication; and the present application can also be applied to a scenario of model training based on MoE parallelism in large model training, which includes a process of computing first and then communicating, such as performing matrix multiplication first and then performing all2all communication, and also includes a process of communicating first and then computing, such as performing all2all communication first and then performing matrix multiplication. In each of the above scenarios, whether it is a process of computing first and then communicating or a process of communicating first and then computing, the present application can be realized by the way of the management node issuing the fusion operator to the computing node, and thus will not be described in detail here.

[0123] For the above method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the action sequence described, and secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the present application. Therefore, other reasonable step combinations that those skilled in the art can think of based on the above description are also within the protection scope of the present application.

[0124] The present application also provides a computing node as shown in FIG. 3, which includes a plurality of matrix computing units for performing the operations performed by the matrix computing units in the embodiments corresponding to FIGS. 2-9 described above, and a plurality of vector computing units for performing the operations performed by the vector computing units in the embodiments corresponding to FIGS. 2-9 described above, and thus will not be described in detail here.

[0125] Referring to FIG. 10, FIG. 10 is a schematic diagram of a computing device provided by the present application. As shown in FIG. 10, the computing device 100 includes a bus 101, a processor 102, a plurality of computing nodes 103, a communication interface 104, and a memory 105. The processor 102, the plurality of computing nodes 103, the communication interface 104, and the memory 105 communicate through the bus 101. It should be understood that the present application does not limit the number of the processor 102, the computing nodes 103, and the memory 105 in the computing device 100. The computing device can be a server, for example, a server of a cloud data center, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a desktop computer, a notebook computer, or the like terminal device.

[0126] The bus 101 can be a peripheral component interconnect express (PCIe) bus or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one line is shown in FIG. 10, but it does not mean that there is only one bus or only one type of bus. The bus 101 can include a path for transmitting information between various components of the computing device 100. Among them, the unified bus is, for example, a bus.

[0127] The processor 102 can be a central processing unit (CPU), and can also include a CPU and other hardware chips. The hardware chips can be of various types, for example, can be any one of programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or digital signal processors (DSPs), etc. The computing device 100 can include one or more hardware chips of any of the above types, or can include multiple types of the above hardware chips, and the present application does not make specific limitations.

[0128] The computing node 103 can be a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or other special AI processing chip, etc., and the present application does not make specific limitations.

[0129] The memory 105 can be a non-volatile memory, for example, a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), or a flash memory. The memory 105 can also be a volatile memory, which can be a random access memory (RAM) used as an external cache. By way of example and not limitation, many forms of RAM can be used, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). In addition, the memory 105 can also be implemented by a storage class memory (SCM), a phase change memory (PCM), or other types of storage media.

[0130] It should be noted that the same type of storage medium can be configured to implement the function of the memory 105 in the same computing device, or two or more types of storage media can be configured to implement the function of the memory 105, and the present application does not make limitations thereto.

[0131] The memory 105 stores executable program code, and the processor 102 executes the executable program code to issue a fusion operator to the computing node 103, so that the computing node implements the distributed computing method introduced in the method embodiments corresponding to FIGS. 2 to 9 according to the fusion operator.

[0132] The communication interface 104 uses a transceiving module such as, but not limited to, a network interface card, a transceiver, to enable communication between the computing device 100 and other devices or communication networks.

[0133] The present application also provides a computer program product containing instructions, which can be a software or program product containing instructions capable of running on a computing device or stored in any available medium. When the computer program product runs on the computing device, it enables the computing device to implement the distributed computing method described in the method embodiment corresponding to FIG. 2 to FIG. 9.

[0134] The present application also provides a computer readable storage medium, which can be any available medium capable of being stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium contains instructions, which instruct the computing device to implement the distributed computing method described in the method embodiment corresponding to FIG. 2 to FIG. 9.

[0135] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of distributed computing, the method comprising: The method is applied to a distributed computing system including a management node and a plurality of computing nodes, and the method comprises: The management node issues a first fusion operator to a first computing node in the plurality of computing nodes; wherein the first fusion operator comprises a first matrix and a second matrix, and the first fusion operator further comprises instructions for instructing the first computing node to implement matrix multiplication of the first matrix and the second matrix; The first computing node performs part of the matrix multiplication of the first matrix and the second matrix by using a plurality of matrix calculation units in the first computing node according to the first fusion operator, to obtain a first calculation result; The first computing node performs a communication task with other computing nodes in the plurality of computing nodes, and the communication task is used to synchronize the first calculation result with the other computing nodes.

2. The method of claim 1, wherein, Each matrix calculation unit in the first computing node needs to perform m rounds of part of the matrix multiplication of the first matrix and the second matrix, and the first fusion operator further comprises sub-matrix information and a communication trigger parameter n; the sub-matrix information is used to indicate the size of a first sub-matrix and a second sub-matrix when each round of operation is performed by one matrix calculation unit, and the communication trigger parameter n is less than or equal to m, and m is a positive integer greater than 1; The first computing node performs part of the matrix multiplication of the first matrix and the second matrix by using a plurality of matrix calculation units in the first computing node according to the first fusion operator, to obtain a first calculation result, comprising: The first computing node performs part of the matrix multiplication of the first matrix and the second matrix by using a plurality of matrix calculation units in the first computing node according to the first fusion operator, to obtain a first calculation result, comprising:

3. The method of claim 2, wherein, The method further comprises: When the first computing node performs the communication task with the other computing nodes in the plurality of computing nodes, the plurality of matrix calculation units start to perform the n+1th round of calculation.

4. The method according to claim 2 or 3, characterized in that, The first fusion operator further comprises an output address, and the output address is used to indicate the address of a shared memory space of the first computing node; The method further comprises: The first computing node stores the first calculation result into the shared memory space according to the output address, wherein the shared memory space can be read and written by the other computing nodes.

5. The method of claim 4, wherein The first computing node stores the first calculation result into the shared memory space according to the output address, comprising: Each matrix calculation unit in the first computing node stores a plurality of rows of data included in a result matrix obtained by performing one round of calculation into the shared memory space in sequence, wherein the i+1th row of the result matrix is stored continuously with the ith row in the shared memory space.

6. The method according to any one of claims 2-5, characterized in that, The first fusion operator further comprises a carrying granularity, and the carrying granularity is used to indicate the amount of data acquired from the shared memory space of another computing node by the first computing node once. Before the management node issues the first fusion operator to the first computing node in the plurality of computing nodes, the method further comprises: The management node determines the carrying granularity according to the calculation amount and the calculation time length of n rounds of calculation and the communication amount and the communication time length of the communication task.

7. The method of claim 6, wherein, The first computing node performs a communication task with other computing nodes in the plurality of computing nodes, including: The first computing node executes a plurality of acquisition instructions to acquire the calculation result of each computing node from the shared memory space of the other computing nodes according to the carrying granularity.

8. The method according to any one of claims 1 to 7, characterized in that, The first computing node performs a communication task with other computing nodes in the plurality of computing nodes, including:

9. The method according to any one of claims 1 to 8, characterized in that, The first computing node queries a flag bit of each computing node in the other computing nodes, the flag bit being used to indicate whether the computing node has completed n rounds of calculation; The first computing node determines that the flag bit of each computing node is a first value, and performs a communication task with other computing nodes in the plurality of computing nodes; the first value of the flag bit is used to indicate that the computing node has completed n rounds of calculation. The matrix multiplication operation is a calculation in an artificial intelligence AI model training process; the computing node includes a graphics processing unit GPU, a neural network processing unit NPU, a tensor processing unit TPU, or other special AI processing chips.

10. The method according to any one of claims 1 to 9, characterized in that, The distributed computing system includes a management node and a plurality of computing nodes, the management node is used to perform the operations performed by the management node in any one of claims 1-10, and any one of the plurality of computing nodes is used to perform the operations performed by the first computing node in any one of claims 1-10.

11. A distributed computing system, characterized in that, The method applied to a computing node in a distributed computing system, the distributed computing system including a management node and a plurality of computing nodes, the method including:

12. A method of distributed computing, the method comprising: The first computing node acquires a first fusion operator issued by the management node; wherein the first fusion operator includes a first matrix and a second matrix, and the first fusion operator further includes an instruction for instructing the first computing node to implement a matrix multiplication operation of the first matrix and the second matrix; the first computing node is one of the plurality of computing nodes; The first computing node performs part of the matrix multiplication operation of the first matrix and the second matrix by using a plurality of matrix calculation units in the first computing node according to the first fusion operator, to obtain a first calculation result; The first computing node performs a communication task with other computing nodes in the plurality of computing nodes, the communication task being used to synchronize the first calculation result with the other computing nodes. Each matrix calculation unit in the first computing node needs to perform m rounds of part of the matrix multiplication operation of the first matrix and the second matrix, the first fusion operator further includes sub-matrix information and a communication trigger parameter n; the sub-matrix information is used to indicate the size of a first sub-matrix and a second sub-matrix when each round of operation is performed by a matrix calculation unit, and the communication trigger parameter n is less than or equal to m, m being a positive integer greater than 1; 13. The method of claim 12, wherein, ​ The first computing node performs, according to the first fusion operator, partial operations in matrix multiplication of the first matrix and the second matrix by using a plurality of matrix calculation units in the first computing node respectively, to obtain a first calculation result, including: The first computing node performs, according to the first fusion operator, partial operations in matrix multiplication of the first matrix and the second matrix by using a plurality of matrix calculation units in the first computing node respectively, to obtain a first calculation result, including:

14. The method of claim 13, wherein, The method further includes: When the first computing node performs a communication task with other computing nodes in the plurality of computing nodes, the plurality of matrix calculation units start to perform the (n+1)th round of calculation.

15. The method according to claim 13 or 14, characterized in that, The first fusion operator further includes an output address, which is used to indicate an address of a shared memory space of the first computing node. The method further includes: The first computing node stores the first calculation result into the shared memory space according to the output address, wherein the shared memory space can be read and written by the other computing nodes.

16. The method of claim 15, wherein The first computing node stores the first calculation result into the shared memory space according to the output address, including: Each matrix calculation unit in the first computing node stores, in sequence, a plurality of rows of data included in a result matrix obtained by performing one round of calculation, into the shared memory space, wherein an (i+1)th row of the result matrix is stored continuously with an ith row in the shared memory space.

17. The method according to any one of claims 12-16, characterized by, The first fusion operator further includes a carrying granularity, which is used to indicate an amount of data acquired from a shared memory space of another computing node by the first computing node in one acquisition instruction; The first computing node performs a communication task with other computing nodes in the plurality of computing nodes, including: The first computing node performs a plurality of acquisition instructions to acquire the calculation result of each computing node from the shared memory space of the other computing nodes according to the carrying granularity.

18. The method according to any one of claims 12-17, characterized by, The first computing node performs a communication task with the other computing nodes based on a memory access instruction.

19. The method according to any one of claims 12-18, characterized in that, The first computing node performs a communication task with other computing nodes in the plurality of computing nodes, including: The first computing node queries a flag bit of each computing node in the other computing nodes, the flag bit being used to indicate whether the computing node has completed n rounds of calculation; The first computing node determines that the flag bit of each computing node is a first value, and performs a communication task with other computing nodes in the plurality of computing nodes; the first value of the flag bit is used to indicate that the computing node has completed n rounds of calculation.

20. The method according to any one of claims 12-19, characterized by, The matrix multiplication operation is a calculation in an artificial intelligence AI model training process; the computing node includes a graphics processing unit GPU, a neural network processing unit NPU, a tensor processing unit TPU or other special AI processing chips.

21. A computing node, characterized in that, The application is applied to a distributed computing system, the distributed computing system including a management node and a plurality of computing nodes, and a first computing node including a plurality of matrix calculation units and a plurality of vector calculation units, wherein The plurality of matrix calculation units are configured to receive a first fusion operator issued by a management node, the first fusion operator comprising a first matrix and a second matrix, and the first fusion operator further comprising an instruction for instructing the first calculation node to implement a matrix multiplication operation of the first matrix and the second matrix; the first calculation node is one of the plurality of calculation nodes; The plurality of matrix calculation units are further configured to respectively perform partial operations in the matrix multiplication operation of the first matrix and the second matrix according to the first fusion operator, to obtain a first calculation result. The plurality of vector calculation units are configured to perform a communication task between the first calculation node and other calculation nodes in the plurality of calculation nodes, and the communication task is configured to synchronize the first calculation result with the other calculation nodes.

22. The node of claim 21, wherein, Each matrix calculation unit in the first calculation node needs to perform m rounds of partial operations of the first matrix and the second matrix, the first fusion operator further comprises sub-matrix information and a communication trigger parameter n; the sub-matrix information is configured to indicate sizes of a first sub-matrix and a second sub-matrix when each matrix calculation unit performs each round of operation, and the communication trigger parameter n is less than or equal to m, and m is a positive integer greater than 1. The plurality of matrix calculation units are configured to perform the partial operations of the matrix multiplication operation in parallel according to the sub-matrix information, until n rounds of calculation are performed, to obtain the first calculation result.

23. The node of claim 22, wherein, The plurality of matrix calculation units are further configured to: start to perform an (n+1)th round of calculation when the plurality of vector calculation units perform the communication task between the first calculation node and the other calculation nodes in the plurality of calculation nodes.

24. The node of claim 22 or 23, characterized by The first calculation node further comprises a memory, the memory comprises a shared memory space, the first fusion operator further comprises an output address, and the output address is configured to indicate an address of the shared memory space of the memory. The plurality of matrix calculation units are further configured to: store the first calculation result into the shared memory space according to the output address, and the shared memory space can be read and written by the other calculation nodes.

25. The node of claim 24, wherein: the plurality of matrix calculation units are specifically configured to sequentially store a plurality of rows of data included in a result matrix obtained by performing one round of calculation into the shared memory space, and an (i+1)th row of the result matrix is stored continuously with an ith row in the shared memory space.

26. The node of any of claims 21-25, wherein, The first fusion operator further comprises a carrying granularity, and the carrying granularity is configured to indicate an amount of data acquired from a shared memory space of another calculation node by the first calculation node once according to an acquisition instruction; the plurality of vector calculation units are specifically configured to perform the acquisition instruction multiple times to acquire the calculation result of each calculation node from the shared memory space of the other calculation node according to the carrying granularity.

27. The node of any of claims 21-26, wherein, The plurality of vector calculation units perform the communication task between the first calculation node and the other calculation nodes based on a memory access instruction.

28. The node of any of claims 21-27, wherein, The plurality of vector calculation units are specifically configured to: query a flag bit of each of the other computing nodes, the flag bit being used to indicate whether the computing node has completed a round of computation; the second vector computation unit is one of the plurality of vector computation units; after determining that the flag bit of each of the computing nodes is a first value, perform a communication task with other computing nodes of the plurality of computing nodes; the first value of the flag bit is used to indicate that the computing node has completed n rounds of computation.

29. The node of any of claims 21-28, wherein, The matrix multiplication operation is a calculation in an artificial intelligence (AI) model training process. The computing node includes a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), or other special-purpose AI processing chip.

30. A computer-readable storage medium, characterized in that, computer program instructions that, when executed by a computing device, cause the computing device to perform the method of any one of claims 12-20.

Images