Distributed computing method and system and related equipment

Abstract

The invention provides a distributed computing method and system and related equipment. The method comprises the steps that a management node issues a fusion operator to a computing node; the fusion operator comprises two matrixes and an instruction for realizing matrix multiplication operation of the two matrixes; a plurality of matrix calculation units in the calculation node respectively execute part of the two matrix multiplication operations according to the fusion operator to obtain part of calculation results of the matrix multiplication operations; and executing a communication task to synchronize the partial calculation result. A management node only needs to send a fusion operator to a calculation node, the calculation node can complete calculation of a matrix multiplication according to the fusion operator, and the management node does not need to split the matrix multiplication operation into a plurality of subtasks for calculation and communication and then send the subtasks to the calculation node in sequence. The phenomenon that computing node resources are idle due to management node splitting and task issuing can be avoided, and the utilization rate of the resources and the efficiency of the distributed computing system are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a distributed computing method, system and related equipment. BACKGROUND

[0002] When performing large-scale computing through multiple computing nodes of a distributed computing system, each computing node is used to perform part of the tasks in the large-scale computing, a management node splits the task to be performed by a computing node into multiple computing tasks and multiple communication tasks, and then sequentially issues the multiple computing tasks and the multiple communication tasks to the computing node, and the computing node performs the computing tasks and the communication tasks in parallel according to the computing tasks and the communication tasks issued by the management node through a reasonable scheduling strategy.

[0003] Generally, the more the computing tasks and the communication tasks that a computing node needs to perform are split, the more the computing tasks and the communication tasks overlap when performing in parallel, and the more the utilization of computing resources and communication resources can be improved, thereby improving the efficiency of the distributed system. However, no matter how much data a computing task needs to calculate or how much data a communication task needs to transmit, the time for the management node to issue a computing task or a communication task is constant, and the more the number of computing tasks that the management node splits from the task to be performed by a computing node, the less the amount of a single computing task and a single communication task, and the less the time for the computing node to perform a single computing task and a single communication task. However, the time for the management node to issue a computing task or a communication task is constant, which may cause the time for the management node to issue a computing or communication task to a computing node to be greater than the time for the computing node to perform a computing task or a communication task, resulting in the phenomenon that the computing node waits for the management node to issue a computing task or a communication task, which may cause waste of computing node resources. SUMMARY

[0004] The present application provides a distributed computing method, system and related equipment, which can perform parallel computing and communication, and improve the resource utilization and efficiency of the distributed computing system.

[0005] In a first aspect, the present application provides a distributed computing method applied to a distributed computing system including a management node and a plurality of computing nodes, the method comprising: the management node sending a fusion operator to each of the plurality of computing nodes, wherein the management node sends a first fusion operator to a first computing node; the first fusion operator including a first matrix and a second matrix, and an instruction for instructing the first computing node to perform a matrix multiplication operation of the first matrix and the second matrix; after receiving the first fusion operator, a plurality of matrix computing units in the first computing node perform part of the matrix multiplication operation of the first matrix and the second matrix according to the first fusion operator, to obtain a first calculation result; and then perform a communication task with other computing nodes in the distributed computing system, the communication task being used for synchronizing the first calculation result with the other computing nodes, and the first computing node also obtains calculation results of the other computing nodes by performing the communication task.

[0006] When the distributed computing system needs to perform distributed computing, and the computing nodes need to perform matrix multiplication, the management node only needs to send a fusion operator to the computing node, the fusion operator including an instruction for performing the matrix multiplication, i.e., the fusion operator including complete computing logic and communication logic for performing the matrix multiplication, and the computing node can complete one matrix multiplication based on the fusion operator. The management node does not need to split one matrix multiplication into a plurality of sub-tasks for computing and for communication, and then send the sub-tasks to the computing node one by one, which can avoid the phenomenon of idle computing node resources caused by the management node splitting and sending tasks, and improve the utilization of resources and the efficiency of the distributed computing system.

[0007] In a possible implementation, each matrix computing unit in the first computing node needs to perform m rounds of part of the matrix multiplication of the first matrix and the second matrix, i.e., each matrix computing unit performs part of the matrix multiplication of the first matrix and the second matrix in each round. The first fusion operator further includes sub-matrix information and a communication trigger parameter n, the sub-matrix information being used for indicating the size of a first sub-matrix and the size of a second sub-matrix participating in the calculation of each round of operation of one matrix computing unit, and the communication trigger parameter n being less than or equal to m, m being a positive integer greater than 1. The plurality of matrix computing units in the first computing node perform part of the matrix multiplication of the first matrix and the second matrix according to the first fusion operator, to obtain the first calculation result, including: each matrix computing unit in the first computing node acquires two sub-matrices from the first matrix and the second matrix according to the sub-matrix information in each round of calculation, performs part of the matrix multiplication of the first matrix and the second matrix, performs part of the matrix multiplication of the matrix multiplication in parallel through the plurality of matrix computing units, until n rounds of calculation are performed, to obtain the first calculation result, and then the first computing node starts to perform the communication task with other computing nodes, to synchronize the results obtained by each computing node after n rounds of calculation.

[0008] By setting appropriate submatrix information and communication trigger parameter n, each matrix calculation unit obtains partial data from the first and second matrices for calculation in each round. After each matrix calculation unit performs n rounds of calculation and obtains the first calculation result, the calculation is triggered. This allows the first calculation results obtained by multiple matrix calculation units to be communicated centrally, which can reduce the number of times data is synchronized between computing nodes. It can also avoid the problem that the amount of data of the first calculation result obtained before the communication task is executed is too large, resulting in insufficient memory of the computing node to store the calculation results obtained in subsequent rounds of calculation before the communication task is completed.

[0009] In one possible implementation, the above method further includes: when the first computing node performs a communication task with other computing nodes among the multiple computing nodes, multiple matrix computing units start to execute the calculation in the (n+1)th round. That is, while the first computing node is performing the communication task, it can continue to perform calculations through the matrix computing units, thereby realizing the parallel execution of calculation and communication and improving the resource utilization and efficiency of the distributed computing system.

[0010] In one possible implementation, the first fusion operator further includes an output address, which indicates the address of the shared memory space of the first computing node; the method further includes: the first computing node stores the first calculation result into the shared memory space according to the output address, wherein the shared memory space can be read and written by other computing nodes.

[0011] The first fusion operator issued by the management node includes an output address. After a matrix calculation unit of the first computing node performs a round of matrix multiplication of submatrices to obtain the result matrix, it can write the result matrix directly into the shared memory space through instructions. When other computing nodes need to obtain the first calculation result of the first computing node from the first computing node, the other computing nodes can obtain the first calculation result from the shared memory space of the first computing node through instructions. This method can reduce data copying operations in different memory areas and improve communication efficiency.

[0012] In one possible implementation, the first computing node stores the first calculation result into the shared memory space according to the output address, including: when each matrix calculation unit in the first computing node stores multiple rows of data including the result matrix obtained from one round of calculation into the shared memory space in sequence, the (i+1)th row and the i-th row of the result matrix are stored consecutively in the shared memory space.

[0013] By storing the (i+1)th row and the ith row of the result matrix of a set of submatrices in contiguous memory, other computing nodes can obtain the result matrix from the contiguous memory space during communication tasks, instead of from different memory spaces, thus improving communication efficiency.

[0014] In one possible implementation, the first fusion operator further includes a transport granularity, which is used to indicate the amount of data that the first computing node obtains from the shared memory space of another computing node in one fetch instruction. Before the management node issues the first fusion operator to the first computing node among multiple computing nodes, the management node determines the above-mentioned transport granularity based on the computational amount and computational duration of n rounds of computation, as well as the communication amount and communication duration of the communication task.

[0015] When a compute node executes a communication task, it retrieves data from the shared memory space of other compute nodes by executing fetch instructions. A single fetch instruction from one compute node is insufficient to obtain the complete computation result from another node; therefore, multiple fetch instructions are required. Since compute nodes also perform computations while executing communication tasks, both communication and computation tasks require memory access, which consumes resources, such as outstanding resources. The finer the granularity of the fetch instruction, the easier it is for the computation task to access memory resources. Therefore, by pre-determining the granularity of the fetch instruction, the memory resources consumed by executing a single fetch instruction can be controlled. This allows for the rational allocation and scheduling of resources required by computation and communication tasks, ensuring efficient parallel execution and improving the efficiency of the distributed computing system.

[0016] In one possible implementation, the first computing node performs communication tasks with other computing nodes among a plurality of computing nodes, including: the first computing node executes multiple fetch instructions to fetch the computation results of each computing node from the shared memory space of the other computing nodes according to the granularity of the fetch.

[0017] In one possible implementation, the first computing node performs communication tasks with other computing nodes based on memory access instructions.

[0018] The fusion operator includes instructions to perform two matrix multiplication operations. This application instructs computation and communication at the instruction level, enabling the matrix and vector computation units to read and write memory using Memory Tagging Extension (MTE) instructions, such as store / load instructions. The fusion operator allows specifying the address where the matrix computation unit will write data after computation; therefore, it allows directly specifying that the computation results obtained from each round of matrix computation be written to the shared memory space.

[0019] In one possible implementation, the first computing node performs a communication task with other computing nodes among the multiple computing nodes, including: the first computing node queries the flag bit of each computing node among the other computing nodes, the flag bit being used to indicate whether the computing node has completed n rounds of computation; the first computing node determines that the flag bit of each computing node is a first value, and performs a communication task with other computing nodes among the multiple computing nodes; wherein, the first value of the flag bit is used to indicate that the computing node has completed n rounds of computation.

[0020] After each matrix computation unit in the first computation node completes n rounds of submatrix multiplication, it sets its flag to 1 via the vector computation unit. Following this, the vector computation unit queries the flag values ​​of other computation nodes within the communication domain in a first-cycle manner to determine whether those nodes have completed their current round of computation. Similarly, any computation node within the communication domain, after completing n rounds of computation, sets its flag to 1 and then queries the flag values ​​of other computation nodes in a first-cycle manner to determine whether they have completed their current round of computation. When each computation node in the same communication domain confirms that other computation nodes have completed their current round of computation, the computation nodes in that domain begin executing communication tasks via the vector computation unit to synchronize their computation results. This avoids computational errors caused by communication occurring before other computation nodes within the communication domain have completed m rounds of computation.

[0021] In one possible implementation, the matrix multiplication operation described above is a computation during the training process of an artificial intelligence (AI) model; the computing node can be a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or other dedicated AI processing chips.

[0022] Secondly, this application provides a distributed computing system, which includes a management node and multiple computing nodes. The management node is used to perform the operations performed by the management node as described in the first aspect or any possible implementation of the first aspect, and any one of the multiple computing nodes is used to perform the operations performed by the first computing node as described in the first aspect or any possible implementation of the first aspect.

[0023] Thirdly, this application provides a distributed computing method applied to computing nodes in a distributed computing system, the distributed computing system including a management node and multiple computing nodes, the method comprising:

[0024] The first computing node receives a first fusion operator issued by the management node. The first fusion operator includes a first matrix and a second matrix, as well as instructions for the first computing node to perform matrix multiplication of the first matrix and the second matrix. After receiving the first fusion operator, multiple matrix calculation units in the first computing node perform partial operations of the matrix multiplication of the first matrix and the second matrix according to the first fusion operator to obtain a first calculation result. Then, it executes a communication task with other computing nodes in the distributed computing system. This communication task is used to synchronize the first calculation result with other computing nodes. The first computing node also obtains the calculation results of other computing nodes by executing the communication task.

[0025] When a distributed computing system needs to perform distributed computations and computing nodes need to execute matrix multiplication operations, the management node only needs to issue a fusion operator to the computing node. This fusion operator includes instructions for implementing the matrix multiplication operation; that is, it includes the computational logic and communication logic for performing the matrix multiplication calculation. The computing node can then complete a matrix multiplication calculation based on this fusion operator. This eliminates the need for the management node to break down a matrix multiplication operation into multiple subtasks for computation and communication, and then send these subtasks sequentially to each computing node. This avoids the idle computing node resources caused by the management node splitting and issuing tasks, improving resource utilization and the efficiency of the distributed computing system.

[0026] In one possible implementation, each matrix computation unit in the first computing node needs to perform m rounds of partial operations on the first and second matrices, that is, each matrix computation unit completes a partial calculation of the matrix multiplication operation of the first and second matrices in each round. The aforementioned first fusion operator also includes submatrix information and a communication trigger parameter n; the submatrix information is used to indicate the size of the first and second submatrixes involved in the calculation when a matrix computation unit performs each round of calculation, and the communication trigger parameter n is less than or equal to m, where m is a positive integer greater than 1. Multiple matrix computation units in the first computing node perform partial operations of the matrix multiplication operation of the first and second matrices according to the first fusion operator to obtain the first calculation result, including: each matrix computation unit in the first computing node obtains two submatrices from the first and second matrices according to the submatrix information in each round of calculation, performs a partial calculation of the matrix multiplication operation of the first and second matrices, and performs the partial calculation of the matrix multiplication operation of the first and second matrices in parallel by multiple matrix computation units until n rounds of calculation are completed to obtain the aforementioned first calculation result. Then, the first computing node starts to perform communication tasks with other computing nodes to synchronize the results obtained by each computing node after performing n rounds of calculation.

[0027] In one possible implementation, the above method further includes: when the first computing node performs a communication task with other computing nodes among the multiple computing nodes, multiple matrix computing units start to execute the calculation in the (n+1)th round. That is, while the first computing node is performing the communication task, it can continue to perform calculations through the matrix computing units, thereby realizing the parallel execution of calculation and communication and improving the resource utilization and efficiency of the distributed computing system.

[0028] In one possible implementation, the first fusion operator further includes an output address, which indicates the address of the shared memory space of the first computing node; the method further includes: the first computing node stores the first calculation result into the shared memory space according to the output address, wherein the shared memory space can be read and written by other computing nodes.

[0029] In one possible implementation, the first computing node stores the first calculation result into the shared memory space according to the output address, including: when each matrix calculation unit in the first computing node stores multiple rows of data including the result matrix obtained from one round of calculation into the shared memory space in sequence, the (i+1)th row and the i-th row of the result matrix are stored consecutively in the shared memory space.

[0030] In one possible implementation, the first fusion operator further includes a transport granularity, which is used to indicate the amount of data that the first computing node retrieves from the shared memory space of another computing node in a single fetch instruction; the first computing node performs a communication task with other computing nodes among the multiple computing nodes, including: the first computing node executes multiple fetch instructions to retrieve the computation results of each computing node from the shared memory space of the other computing nodes according to the transport granularity.

[0031] In one possible implementation, the first computing node performs communication tasks with other computing nodes based on memory access instructions.

[0032] In one possible implementation, the first computing node performs a communication task with other computing nodes among the multiple computing nodes, including: the first computing node queries the flag bit of each computing node among the other computing nodes, the flag bit being used to indicate whether the computing node has completed n rounds of computation; the first computing node determines that the flag bit of each computing node is a first value, and performs a communication task with other computing nodes among the multiple computing nodes; wherein, the first value of the flag bit is used to indicate that the computing node has completed n rounds of computation.

[0033] In one possible implementation, the matrix multiplication operation described above is a computation during the training process of an artificial intelligence (AI) model; the computing node can be a GPU, NPU, TPU, or other dedicated AI processing chip.

[0034] Fourthly, this application provides a computing node applied to a distributed computing system. The distributed computing system includes a management node and multiple computing nodes. For any one of these multiple computing nodes, such as a first computing node, the first computing node includes multiple matrix computing units and multiple vector computing units. The multiple matrix computing units are used to obtain a first fusion operator issued by the management node. The first fusion operator includes a first matrix and a second matrix, as well as instructions for the first computing node to perform matrix multiplication of the first matrix and the second matrix. According to the first fusion operator, partial operations in the matrix multiplication of the first matrix and the second matrix are executed respectively to obtain a first calculation result. The multiple vector computing units are used to perform communication tasks with other computing nodes among the multiple computing nodes. The communication tasks are used to synchronize the first calculation result with other computing nodes.

[0035] In one possible implementation, each matrix computation unit in the first computation node needs to perform m rounds of partial operations on the first and second matrices. That is, each matrix computation unit completes a partial calculation of the matrix multiplication operation of the first and second matrices in each round. The aforementioned first fusion operator also includes submatrix information and a communication trigger parameter n. The submatrix information is used to indicate the size of the first and second submatrixes involved in the calculation when a matrix computation unit performs each round of operation. The communication trigger parameter n is less than or equal to m, where m is a positive integer greater than 1. The aforementioned multiple matrix computation units are used to perform partial operations of the matrix multiplication operation in parallel according to the submatrix information until n rounds of calculation are completed and the first calculation result is obtained. Then, the vector computation unit starts communication tasks with other computation nodes to synchronize the results obtained by each computation node after performing n rounds of calculation.

[0036] In one possible implementation, the aforementioned multiple matrix computation units are also used to: when the multiple vector computation units are performing communication tasks with other computation nodes among the multiple computation nodes, the multiple matrix computation units start executing the (n+1)th round of computation, that is, while the first computation node is performing the communication task, it can continue to perform computation through the matrix computation units.

[0037] In one possible implementation, the first computing node further includes a memory, which includes a shared memory space, and the first fusion operator further includes an output address, which is used to indicate the address of the shared memory space of the memory.

[0038] The aforementioned matrix calculation units are also used to: store the first calculation result into a shared memory space according to the output address, wherein the shared memory space can be read and written by other calculation nodes.

[0039] In one possible implementation, the multiple matrix computation units are specifically used to: store the multiple rows of data included in the result matrix obtained from performing one round of computation sequentially into a shared memory space, wherein the (i+1)th row of the result matrix is ​​stored contiguously with the ith row in the shared memory space.

[0040] In one possible implementation, the first fusion operator further includes a transport granularity, which is used to indicate the amount of data that the first computing node obtains from the shared memory space of another computing node when executing a fetch instruction. Specifically, the aforementioned multiple vector computing units are used to execute multiple fetch instructions and obtain the computation results of each computing node from the shared memory space of other computing nodes according to the transport granularity.

[0041] In one possible implementation, the aforementioned multiple vector computing units perform communication tasks with other computing nodes based on memory access instructions.

[0042] In one possible implementation, the aforementioned multiple vector computation units are specifically used to: query the flag bit of each computation node in other computation nodes, the flag bit being used to indicate whether the computation node has completed n rounds of computation; the first computation node determines that the flag bit of each computation node is a first value, and performs a communication task with other computation nodes in the multiple computation nodes; wherein, the first value of the flag bit is used to indicate that the computation node has completed n rounds of computation.

[0043] In one possible implementation, the matrix multiplication operation described above is a computation during the training process of an artificial intelligence (AI) model; the computing node can be a GPU, NPU, TPU, or other dedicated AI processing chip.

[0044] Fifthly, this application provides a computing device comprising a processor, multiple computing nodes, and a memory. The memory stores instructions, and the processor executes the instructions stored in the memory to implement the method implemented by the management node as described in the first aspect or any possible implementation thereof. Any one of the multiple computing nodes performs the operation performed by the first computing node as described in the first aspect or any possible implementation thereof. The computing nodes may be GPUs, NPUs, TPUs, or other dedicated AI processing chips.

[0045] Sixthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, cause the computing devices to implement the method implemented by computing nodes as described in the first aspect or any possible implementation of the first aspect.

[0046] In a seventh aspect, this application provides a computer program product, which includes a computer program that, when run by a cluster of computing devices, implements the method implemented by the computing nodes as described in the first aspect or any possible implementation of the first aspect.

[0047] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0048] Figure 1 This is a schematic diagram illustrating matrix multiplication.

[0049] Figure 2 This is a schematic diagram of a distributed computing system provided in this application;

[0050] Figure 3 This is a schematic diagram of a computing node provided in this application;

[0051] Figure 4 This is a schematic diagram of a distributed computing method provided in this application;

[0052] Figure 5 This is a schematic diagram illustrating the data flow of calculation results in the storage area, as provided in this application.

[0053] Figure 6 This is another schematic diagram illustrating the data flow of calculation results in the storage area provided in this application;

[0054] Figure 7 This is a schematic diagram illustrating the storage of calculation results provided in this application;

[0055] Figure 8 This is a schematic diagram of a communication process provided in this application;

[0056] Figure 9 This is a schematic diagram illustrating the storage of another calculation result provided in this application;

[0057] Figure 10 This is a schematic diagram of a computing device provided in this application. Detailed Implementation

[0058] The distributed computing method provided in this application will be described below with reference to the accompanying drawings.

[0059] With the development of information technology, the processing of large-scale data has become a key focus in various fields. The computational processing of large-scale data typically requires distributed computing systems for distributed computation. For example, in the field of artificial intelligence (AI), distributed training is used to improve the training speed of AI models. Distributed training strategies include data parallelism (SP), model parallelism (MP), sequence parallelism (SP), and mixture of experts (MoE) parallelism. Among these, model parallelism includes tensor parallelism (TP) and pipeline parallelism (PP). All of these different parallel strategies improve training efficiency by deploying the model across multiple computing nodes in a distributed computing system, with each computing node responsible for a portion of the model training process.

[0060] When training a model using multiple computing nodes in a distributed computing system, each computing node needs to complete multiple computational tasks and multiple communication tasks. After completing a computational task, each computing node executes a communication task. This communication task involves synchronizing data obtained from different computing nodes via collective communication. After data synchronization, each computing node then performs subsequent computational tasks. The data sent in the communication task is the computational result obtained from the previous computational tasks on each computing node. Furthermore, subsequent computational tasks on each computing node depend on the data obtained from the previous communication tasks. Therefore, computational and communication tasks are interdependent. These dependent computational and communication tasks must be executed sequentially. This means that communication tasks cannot be executed during computational tasks, and vice versa. Consequently, the computational and communication resources of the distributed computing system are not fully utilized, impacting training efficiency. The collective communication methods mentioned include allreduce communication, ring-allreduce communication, all2all communication, reduce_scatter communication, and allgather communication. For example, allreduce communication or ring-allreduce communication is used in data parallelism and tensor parallelism, while all2all communication is used in MoE parallelism.

[0061] In AI model training, model parallelism primarily targets the matrix multiplication portion, a major operation in AI model training. For example, the forward propagation training phase of an AI model typically involves matrix multiplication between two matrices, such as the matrix multiplication between activation values ​​and weights in convolutional neural networks, recurrent neural networks, or feedforward neural networks. Another example is the transformer model, which includes multiple self-attention structures and multiple feedforward networks (FFNs), each involving two matrix multiplication operations. Model parallelism mainly utilizes the principle of block matrix multiplication, dividing matrix multiplication operations across multiple computational nodes to improve computational efficiency.

[0062] like Figure 1 As shown, Figure 1This is a schematic diagram of matrix multiplication. To perform matrix multiplication Y = X × Z, where X is an M × K matrix, Z is a K × N matrix, and Y is the result of the matrix multiplication, when implementing the matrix multiplication using two computing nodes, matrix X can be divided into two M × K / 2 matrices, X1 and X2, and matrix Y can be divided into two K / 2 × N matrices, Y1 and Y2. The first computing node performs the matrix multiplication Y1 = X1 × Z1, and the second computing node performs the matrix multiplication Y2 = X2 × Z2. After both computing nodes have completed their tasks, the results are synchronized through aggregated communication, ensuring that each computing node includes both Y1 and Y2. Then, Y1 and Y2 are added together to obtain the matrix Y. The next computation task is then performed based on matrix Y, and the next communication task is executed after the next computation task is completed.

[0063] However, the aforementioned computing nodes execute computation and communication tasks sequentially. A computing node must perform a communication task and synchronize data after completing its computation task, and only after data synchronization is achieved can the next computation task be executed. This results in low overall computational efficiency due to communication latency. To address this issue, current methods primarily employ parallel masking strategies for computation and communication to improve overall efficiency. This involves breaking down a computation task performed by a computing node into multiple sub-computation tasks. After each sub-computation task completes and yields partial results, these results are synchronized, and the next sub-computation task continues execution simultaneously, achieving parallelism between computation and communication. In this method, each execution of a sub-computation task only yields a portion of the computational results, and the computational results that would normally require a single communication task for synchronization now require multiple synchronizations; essentially, a communication task is broken down into multiple sub-communication tasks. For ease of description, sub-computation tasks and sub-communication tasks are collectively referred to as subtasks. By breaking down a computation task and a communication task into finer-grained subtasks, parallel masking of computation and communication is achieved, thereby improving the efficiency of the distributed computing system.

[0064] However, in the aforementioned parallel computing and communication masking method, the management node needs to break down a computing task and a communication task that a computing node needs to execute into multiple subtasks, and then schedule the computing nodes to execute each subtask by issuing subtasks. However, regardless of the amount of data that a subtask needs to compute or communicate, the time spent by the management node on data preprocessing, loading, and issuing operations for that subtask before issuing it remains constant. If the management node breaks down a computing node's task into Q subtasks, the management node's time overhead will be Q times that of not breaking it down, while the workload of each subtask of the computing node will decrease, and the required computing or communication time will decrease. As Q increases, the time for the computing node to execute a subtask will decrease accordingly, while the time for the management node to issue a subtask remains constant. Therefore, this will cause the time for the management node to issue a task to the computing node to be greater than the time for the computing node to execute a subtask, leading to situations where computing node resources are idle while waiting for the management node to issue tasks.

[0065] To address the aforementioned issues, this application provides a distributed computing method. For a matrix multiplication operation that a computing node needs to perform, the management node only needs to issue a fusion operator to that computing node. This fusion operator includes instructions for implementing the matrix multiplication operation; that is, it includes both computational and communication logic for performing the matrix multiplication. The computing node can then complete the matrix multiplication calculation based on this fusion operator. This eliminates the need for the management node to break down a matrix multiplication operation into multiple subtasks for computation and communication, and then sequentially send these subtasks to each computing node. This avoids the resource idleness of computing nodes caused by the management node splitting and issuing tasks, thereby improving resource utilization and the efficiency of the distributed computing system.

[0066] The distributed computing system provided in this application is described below. (See also...) Figure 2 , Figure 2 This is a schematic diagram of a distributed computing system provided in this application. The distributed computing system includes a management node and multiple computing nodes. The management node and the multiple computing nodes are connected via interconnection links, and the multiple computing nodes are connected to each other via interconnection links. The management node is used to issue fusion operators to each computing node in the distributed computing system. The fusion operators instruct the computing nodes to perform matrix multiplication of two matrices. Each computing node performs matrix multiplication of two matrices according to the received fusion operator, and also communicates with other computing nodes. The methods for computing nodes to perform matrix multiplication according to the received fusion operator and to communicate with other computing nodes are described in detail below.

[0067] See Figure 3 , Figure 3This is a schematic diagram of a computing node provided in this application. The distributed computing system includes multiple such nodes. Figure 3 The computing nodes shown each include multiple matrix (cube) computing units and at least one vector (vector) computing unit. Matrix computing units perform matrix operations, such as matrix multiplication; vector computing units perform vector operations, such as vector addition and multiplication, and can also be used to retrieve data from other nodes, such as reading data from the memory of other computing nodes into the local computing node. Each computing node also includes memory and a cache. Memory stores the data needed by the computing units during computation, as well as the computation results obtained after the computing units perform the computations. The cache is used for exchanging data between the computing units (matrix and vector computing units) and memory. When a computing unit reads data from memory, it first reads the data into the cache, and then reads it from the cache into the computing unit; the computation results obtained by the computing unit are first written to the cache, and then written from the cache into memory.

[0068] The aforementioned distributed computing system also includes management nodes, which can be hosts or central processing units (CPUs). The computing nodes can be graphics processing units (GPUs), neural network processing units (NPUs), tensor processing units (TPUs), or other dedicated AI processing chips. Multiple computing nodes can reside in the same computing device or in different computing devices. If the multiple computing nodes are located in the same computing device, they can be interconnected via interconnect links. These interconnect links can be buses, such as high-speed serial computer component interconnect express (PCIE) buses, NVLink buses, or other buses. The interconnect links can also be switching chips, such as NVSwitch chips or other chips capable of connecting multiple computing nodes and providing data communication functionality. This application does not specifically limit the topology between multiple computing nodes. If the aforementioned multiple computing nodes are located in multiple different computing devices, then each computing device includes at least one computing node. These multiple computing devices are connected by interconnection links, and the computing nodes in the multiple computing devices can communicate with each other through interconnection links. This application does not specifically limit the topology between the multiple computing devices.

[0069] When the aforementioned multiple computing nodes are located in the same computing device, the device management node can be located in the same computing device as these multiple computing nodes, and the management node and the multiple computing nodes are connected via a bus; alternatively, the management node can be located in different computing devices than these multiple computing nodes. When the aforementioned multiple computing nodes are located in multiple computing devices, the management node can be located in one of the computing devices or in other computing devices, and communicate with these multiple computing devices via interconnect links.

[0070] The aforementioned memory can be random access memory (RAM), dynamic random access memory (DRAM), such as high bandwidth memory (HBM), or static random access memory (SRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), or direct rambus RAM (DRRAM).

[0071] In this application, the aforementioned memory includes a standard address space and a shared memory space. The standard address space is used to store the data needed by the computing node when performing matrix multiplication operations, and the shared memory space is used to store the results obtained by the computing node during the matrix multiplication operation. The shared memory space of each computing node can be accessed and read / written by other computing nodes. When a computing node writes the results obtained during the matrix multiplication operation to the shared memory space, it first writes the calculation result to a cache, and then writes the calculation result from the cache to the shared memory space.

[0072] The distributed computing method provided in this application is described below. For example... Figure 4 As shown, Figure 4 This is a schematic diagram of a distributed computing method provided in this application. The method includes the following steps S401 to S403.

[0073] S401. The management node issues the first fusion operator to the first computing node.

[0074] A distributed computing system comprises a management node and multiple computing nodes. When large-scale computation is required through the distributed computing system, the management node issues fusion operators to these computing nodes. Each fusion operator includes two matrices that the corresponding computing node needs to compute, as well as instructions for performing matrix multiplication on these two matrices. Taking the first computing node among the multiple computing nodes as an example, the management node issues a first fusion operator to the first computing node. The first fusion operator includes a first matrix and a second matrix, as well as instructions for performing matrix multiplication on the first matrix and the second matrix.

[0075] It should be noted that the fusion operator received by each computing node in the distributed computing system is a matrix multiplication operation of two matrices. The method of each computing node performing matrix multiplication operation and communicating with other computing nodes is the same as that of the first computing node. The distributed computing method provided in this application is introduced below with the example of the first computing node performing matrix multiplication operation.

[0076] The aforementioned large-scale computation may be for training an AI model. The first and second matrices may be partial data from two parameter matrices that require matrix multiplication during the AI ​​model training process. For example, the first matrix might be partial data from the activation matrix, and the second matrix might be partial data from the weight matrix. This large-scale computation could also be for other applications, where the matrix multiplication operations performed by each computing node constitute the computational logic within that large-scale computation.

[0077] S402. The first computing node receives the first fusion operator issued by the management node, and performs partial operations in the matrix multiplication of the first matrix and the second matrix through multiple matrix computing units to obtain the first calculation result.

[0078] The first computing node receives the first fusion operator issued by the management node. The multiple matrix computing units included in the first computing node perform partial operations in the matrix multiplication of the first matrix and the second matrix respectively according to the first fusion operator to obtain the first calculation result.

[0079] The following describes in detail the process by which multiple matrix calculation units in the first calculation node perform some operations in the matrix multiplication of the first and second matrices to obtain the first calculation result.

[0080] In this application, the first fusion operator includes a first matrix A and a second matrix W, as well as matrix information of the first matrix A and the second matrix W. The matrix information indicates the size of the first matrix A and the second matrix W. For example, the first matrix A is an M×K matrix, and the second matrix W is a K×N matrix.

[0081] In this application, the first computing node performs matrix multiplication of a first matrix A and a second matrix W through multiple matrix computing units included in the first computing node. Therefore, each matrix computing unit is used to calculate a portion of the matrix multiplication of the first matrix A and the second matrix W. For example, the matrix multiplication of the first matrix A and the second matrix W is decomposed into matrix multiplication of multiple sets of submatrices. Each set of submatrices includes a submatrix obtained from the first matrix A and a submatrix obtained from the second matrix W. Each matrix computing unit performs the multiplication of a portion of the submatrices, that is, each matrix computing unit performs a portion of the matrix multiplication of the first matrix A and the second matrix W respectively.

[0082] For example, when the first computing node performs matrix multiplication of the first matrix A and the second matrix W, each matrix computing unit obtains a row of data from the first matrix A as the first submatrix and a column of data from the second matrix W as the second submatrix. Each matrix computing unit performs matrix multiplication of the first submatrix and the second submatrix in one round of calculation. This is equivalent to dividing the first matrix A and the second matrix W into multiple submatrixes respectively, taking one submatrix of the first matrix A and one submatrix of the second matrix W as a group of submatrixes, and calculating the matrix multiplication of each group of submatrixes respectively. Each matrix computing unit performs matrix multiplication of multiple groups of submatrixes.

[0083] For example, if the first matrix A is divided into 8 submatrices by rows, namely A1, A2…A8, and each submatrix has a size of M / 8×K; and the second matrix W is divided into 10 submatrices by columns, namely W1, W2…W10, and each submatrix has a size of K×(N / 10), then the matrix multiplication between the first matrix A and the second matrix W can be expressed as Equation 1 below:

[0084]

[0085] Here, matrix O1 is the result of the matrix multiplication of the first matrix A and the second matrix W. As shown in the above formula, the matrix multiplication between the first matrix A and the second matrix W is transformed into the matrix multiplication of 80 sets of submatrices. A set of submatrices refers to the matrix multiplication between a submatrix in the first matrix A and a submatrix in the second matrix W; for example, A1 × W1 in Formula 1 is a set of matrix multiplications. If the first computation node includes 5 matrix computation units, and each matrix computation unit completes 16 sets of submatrix matrix multiplications, then the matrix multiplication of the first matrix A and the second matrix W can be realized.

[0086] In this application, the first fusion operator further includes submatrix information. This submatrix information indicates the size of the first submatrix and the size of the second submatrix participating in a round of computation when a matrix computation unit performs one round of computation. For example, if the submatrix information includes three parameters M1, K1, and N1, then the submatrix information indicates that a matrix computation unit obtains an M1×K1 first submatrix from the first matrix A and a K1×N1 second submatrix from the second matrix W each time, where M is an integer multiple of M1 and N is an integer multiple of N1. A round of computation by a matrix computation unit means that the matrix computation unit obtains the first submatrix from the first matrix A, obtains the second submatrix from the second matrix W, and performs matrix multiplication of the first and second submatrixes to obtain a result matrix.

[0087] For example, if multiple matrix calculation units of the first computing node calculate the matrix multiplication of the first matrix A and the second matrix W in the manner described in Formula 1 above, then the submatrix information indicates that a matrix calculation unit obtains an M / 8×K submatrix from the first matrix A and a K×(N / 10) submatrix from the second matrix W each time. A matrix calculation unit obtains an M / 8×K first submatrix from the first matrix A and a K×(N / 10) second submatrix from the second matrix W each time, performs the matrix multiplication operation of the first submatrix and the second submatrix, and obtains the result matrix of M / 8×(N / 10).

[0088] When multiple matrix computation units of the first computation node perform matrix multiplication of the first matrix A and the second matrix W according to the first fusion operator, one matrix computation unit obtains a first submatrix from the first matrix A and a second submatrix from the second matrix W according to the submatrix information, and then calculates the matrix multiplication of the first submatrix and the second submatrix to obtain a result matrix.

[0089] If the matrix multiplication of the first matrix A and the second matrix W is performed according to the method corresponding to Formula 1 and its related explanations, the first calculation node needs to perform a total of 80 sets of matrix multiplication operations of submatrices. If the first calculation node includes 5 matrix calculation units, then each matrix calculation unit needs to perform 16 sets of matrix multiplication operations of submatrices. That is, each matrix calculation unit needs to perform 16 rounds of calculation to realize the matrix multiplication operation of the first matrix A and the second matrix W and obtain the calculation results of the first matrix A and the second matrix W.

[0090] It should be noted that this application does not limit the parameters in the submatrix information, that is, this application does not limit the way the first matrix A and the second matrix W are partitioned. For example, the first computing node may partition only one of the matrices into submatrices, such as partitioning the first matrix A by rows and leaving the second matrix W unpartitioned; or leaving the first matrix A unpartitioned and partitioning the second matrix W by columns. Alternatively, the first matrix A may be partitioned into M1×K1 submatrices and the second matrix W into K1×N1 submatrices.

[0091] In this application, the first fusion operator further includes a communication triggering parameter n. Each matrix computation unit of the first computing node obtains a submatrix from the first matrix A and the second matrix W respectively based on the submatrix information, and completes one round of computation. After each matrix computation unit in the first computing node performs n rounds of computation, it completes part of the matrix multiplication operation of the first matrix and the second matrix to obtain the first computation result, and then triggers a communication task. The communication task is used to synchronize the first computation result between the first computing node and other computing nodes in the distributed computing system. For example, if the first computing node includes 5 matrix computation units, and each matrix computation unit needs to perform 16 rounds of computation, if the value of n is 4, then each matrix computation unit in the first computing node will complete 4 rounds of computation before triggering the communication task.

[0092] If each matrix computation unit in the first computing node needs to complete m rounds of computation before it can perform the matrix multiplication operation between the first matrix A and the second matrix W, then the aforementioned communication trigger parameter n is less than or equal to m. For example, if the first computing node needs to perform a total of 80 sets of submatrix matrix multiplication operations, and if the first computing node includes 5 matrix computation units, then each matrix computation unit needs to perform 16 sets of submatrix matrix multiplication operations, where the value of m is 16, and n is a positive integer less than or equal to 16.

[0093] When the communication trigger parameter n is less than m, it means that the multiple matrix calculation units of the first computing node only need to complete part of the matrix multiplication operation of the first matrix A and the second matrix W to trigger the communication task and execute the communication task with other computing nodes to synchronize the first calculation result. In this application, when each matrix calculation unit in the first computing node performs n rounds of calculation, and the first computing node performs the communication task with other computing nodes, each matrix calculation unit of the first computing node will start the (n+1)th round of calculation. During the execution of the communication task by the first computing node, each computing unit in the first computing node performs calculation synchronously, so that calculation and communication are parallel, improving the utilization rate of the computing and communication resources of the first computing node, and also improving the calculation efficiency.

[0094] In this application, the aforementioned first fusion operator further includes an output address, which indicates the storage location of the first computation result of the first computing node in memory. This output address is the address of the shared memory space of the first computing node. Each matrix computation unit directly writes the result matrix into the shared memory space of the first computing node. The shared memory space refers to an area that can be accessed and read / written by other computing nodes, allowing other computing nodes to obtain the first computation result of this first computing node.

[0095] See Figure 5 , Figure 5 This is a schematic diagram illustrating the data flow of calculation results in the storage area, as provided in this application. Figure 5 The system comprises two computing nodes: a first computing node and a second computing node. Each computing node's memory includes a standard address space and a shared memory space. Currently, in the method of matrix multiplication of two matrices by issuing subtasks to computing nodes through a management node, each matrix computation unit of the first computing node, after executing a subtask and obtaining its computation result, writes the result into the standard address space via a cache. When executing a communication task, the first computing node needs to copy the data to be transmitted, calculated by multiple matrix computation units, from the standard address space to the shared memory space so that it can be read by other nodes to complete the communication task. The data to be transmitted includes the computation results obtained by multiple matrix computation units. The second computing node obtains the data to be transmitted from the shared memory space of the first computing node and writes it into its own shared memory space. Then, the vector computation unit of the second computing node reads the aforementioned data to be transmitted and the data obtained from executing a subtask from the shared memory space of the second computing node, aggregates the data, and writes the aggregated result into the standard address space of the second computing node via a cache.

[0096] See Figure 6 , Figure 6 This is another schematic diagram illustrating the data flow of computation results in the storage area provided by this application. In this application, the fusion operator includes instructions to implement two matrix multiplication operations. That is, this application instructs the computation node to complete computation and communication from the instruction level. Therefore, the matrix computation unit and vector computation unit in this application can read and write memory using Memory Tagging Extension (MTE) instructions, such as store / load memory access instructions. In the fusion operator, the address where the matrix computation unit will write data after computation can be specified; therefore, it is possible to directly specify that the computation results of each computation performed by the matrix computation unit be written to the shared memory space.

[0097] The first fusion operator issued by the management node includes an output address. After each matrix computation unit of the first computing node performs a round of matrix multiplication of submatrices to obtain the result matrix, it writes the result matrix into the shared memory space through a cache using the store instruction. When the second computing node needs to retrieve the first computation result from the first computing node, the vector computation unit of the second computing node retrieves the first computation result from the shared memory space of the first computing node through the load instruction. Then, the vector computation unit of the second computing node aggregates the first computation result with the data obtained by the second computing node in n rounds to obtain the aggregated result, and writes the aggregated result into the standard address space of the second computing node through a cache.

[0098] Compared to the current method where computing nodes first write the computation results to the standard address space and then copy the computation results from the standard address space to the shared memory space so that other computing nodes can obtain the computation results from the shared memory space, the method of this application can reduce data copying operations and improve communication efficiency.

[0099] It is understandable that when the matrix computation unit writes the result matrix to the shared memory space, it first writes the result matrix to the cache, and then writes the result matrix from the cache to the shared memory space. If the result matrix is ​​still in the cache when the computing node completes n rounds of computation and executes the communication task, other nodes can obtain the result matrix from the cache, that is, other computing nodes can obtain the first computation result from the cache of the first computing node.

[0100] In this application, after each matrix computation unit performs a matrix multiplication operation on a set of submatrices to obtain a result matrix, when writing the result matrix into the shared memory space, it writes the result matrix of the matrix multiplication operation on the same set of submatrices into a contiguous storage space within the shared memory space. For example, if the first computation node completes the matrix multiplication operation between the first matrix A and the second matrix W in the manner corresponding to (Formula 1), the result matrix of the matrix multiplication of each set of submatrices is an M / 8 × (N / 10) matrix. When a matrix computation unit performs a matrix multiplication operation and writes the calculated result matrix into the shared memory space, it first writes the first row of data of the result matrix into the shared memory space, and then stores the second row of data contiguously with the first row of data, that is, it stores the (i+1)th row of data of the result matrix with the i-th row of data in a contiguous memory space. Here, i+1 is less than or equal to M / 8 and i is a positive integer. In this way, during the execution of communication tasks, when other computation nodes obtain the result matrix of the matrix multiplication of a set of submatrices, they can obtain it from a contiguous memory space instead of from different memory spaces, which can improve communication efficiency.

[0101] For example, such as Figure 7 As shown, Figure 7This is a schematic diagram illustrating the storage of computation results provided in this application. If a matrix computation unit of the first computation node completes the matrix multiplication operation of the submatrices A1 and W1 in one round of computation, matrix O is obtained. 11 In another round of calculations, the matrix multiplication of A1 and W2 was completed, resulting in matrix O. 12 Although matrix O 11 The first row of data and matrix O 12 The first row of data belongs to the same row of data in matrix O1, and matrix O 11 The last value of the first row of data and matrix O 12 The first value of the data in the first row is one of two adjacent elements in matrix O1, such as... Figure 7 Median a 14 Sum of values ​​a 15 In matrix O1, these are two adjacent values. However, when matrix O... 11 Sum of matrix O 12 When writing to shared memory space, first write matrix O 11 Write to a contiguous storage space in the shared memory space, i.e., matrix O. 11 The first value in the (i+1)th row is stored at an adjacent address to the last value in the i-th row. Then, matrix O... 12 Write to a contiguous block of memory within the shared memory space. For example... Figure 7 As shown, the value a 14 Sum of values ​​a 21 Adjacent storage, value a 24 Sum of values ​​a 31 Adjacent storage, value a 34 Sum of values ​​a 41 Adjacent storage, value a 44 Sum of values ​​a 15 Adjacent storage.

[0102] It should be noted that this application does not limit the matrix multiplication operations of multiple submatrices performed by each matrix computation unit in the computing node. That is, each matrix computation unit randomly performs matrix multiplication operations of multiple submatrices, and the order in which the matrix multiplication operations of multiple submatrices are performed by each matrix computation unit is not limited. For example, if the five matrix computation units in the first computing node mentioned above need to complete matrix multiplication operations of 80 submatrices, and each matrix computation unit needs to perform 16 rounds of computation, that is, each matrix computation unit needs to perform matrix multiplication operations of 16 submatrices, this application does not limit which 16 submatrices are performed by each matrix computation unit, nor does it limit the order in which a matrix computation unit performs matrix multiplication operations of 16 submatrices.

[0103] S403. The first computing node performs communication tasks with other computing nodes to synchronize the first computing result.

[0104] In this application, the fusion operator also includes a communication domain, which includes the identifiers of multiple computing nodes that need to exchange data with each other. The first computing node can determine other computing nodes that need to interact with each other when performing a communication task based on the communication domain.

[0105] After completing n rounds of computation, a computing node initiates a communication task. The node's vector computation unit first checks the communication domain within the fusion operator to see if other computing nodes in that domain have also completed n rounds. If all other computing nodes in the communication domain have completed n rounds, then the multiple computing nodes within that domain begin executing communication tasks to exchange data. For example, after completing one round of computation, the first computing node initiates a communication task to check if other computing nodes in the communication domain have completed this round. Only if the first computing node determines that all other computing nodes in the communication domain have completed this round will it begin executing the communication task.

[0106] In this application, each computing node has a flag bit indicating whether it has completed N rounds of computation. This flag bit has two values: a first value indicates that the computing node has completed one round of computation, and a second value indicates that the computing node has not yet completed one round of computation. Upon receiving the fusion operator, the first computing node first sets the flag bit to the second value through its vector computation unit, and then performs matrix multiplication of the first matrix A and the second matrix W according to the method described above. After each matrix computation unit of the first computing node completes n rounds of matrix multiplication of the submatrices, the first computing node sets the flag bit to the first value through its vector computation unit. In this application, this flag bit can be represented by a single bit; the first value is 1, and the second value is 0. Each matrix computation unit in the first computing node sets the flag bit to 1 through its vector computation unit after completing n rounds of matrix multiplication of the submatrices.

[0107] After the first computing node sets its flag to 1, the vector computing unit queries the flag values ​​of other computing nodes within the communication domain in the first cycle to determine whether the other computing nodes have completed this round of computation. Similarly, after any computing node in the communication domain completes n rounds of computation, it also sets its flag to 1 and queries the flag values ​​of other computing nodes in the first cycle to determine whether the other computing nodes have completed this round of computation. Once each computing node in the same communication domain has determined that the other computing nodes have completed this round of computation, the computing nodes in that communication domain begin to execute communication tasks through the vector computing unit. For example, data aggregation can be performed through set communication to synchronize their respective computation results, avoiding computational errors caused by communication before other computing nodes in the communication domain have completed n rounds of computation.

[0108] In this application, when computing nodes in the same communication domain execute communication tasks through vector computing units, each computing node also simultaneously begins the next round of computation through matrix computing units, that is, begins the (n+1)th round of computation. It should be understood that when each computing node performs the next round of computation, the shared memory space of each computing node still stores the computation results of the previous n rounds. Therefore, the computing nodes need to write the computation results obtained in the next round into other shared memory spaces.

[0109] In this application, the aforementioned first fusion operator also includes a transport granularity, which refers to the amount of data that a computing node can obtain from the shared memory space of another computing node by executing a single fetch instruction. When a computing node performs a communication task, it primarily executes fetch instructions to retrieve data from the shared memory space of other computing nodes. In this application, when the first computing node performs a communication task, it reads data from the shared memory space of other computing nodes through a vector computation unit. Since executing a single fetch instruction does not allow the first computing node to obtain the complete computation result obtained by another computing node through n rounds of computation, the first computing node executes multiple fetch instructions based on the transport granularity to obtain the complete computation result from the other computing node.

[0110] For ease of description, the process of the matrix computation unit in the aforementioned computing node performing n rounds of computation is referred to as executing one computation task. This application achieves parallelism between computation and communication tasks by masking shorter tasks with longer execution times.

[0111] When a compute node executes a communication task, it retrieves data from the shared memory space of other compute nodes by executing fetch instructions. A single fetch instruction from one compute node is insufficient to obtain the complete computation result from another node; therefore, multiple fetch instructions are required. Since compute nodes are also performing computation tasks while executing communication tasks, both tasks require memory access, consuming resources such as outstanding resources. The finer the granularity of the fetch instruction, the easier it is for the computation task to access memory. Therefore, by pre-determining the granularity of the fetch instruction, the memory resources used in a single fetch instruction can be controlled, enabling the rational allocation and scheduling of resources for both computation and communication tasks. This ensures efficient parallel execution of both tasks, improving the efficiency of the distributed computing system.

[0112] With the submatrix information and communication trigger parameters determined, the computational load of a single computation task and the communication load required for a single communication task can be determined. Based on the computing power and computational load of the computing node, the theoretical computation time for the computing node to complete a single computation task can be determined. Based on the communication load and communication bandwidth, the theoretical communication time of the computing node can be determined. If the theoretical communication time is longer than the theoretical computation time, the granularity of data transfer needs to be increased to improve communication efficiency; if the theoretical communication time is shorter than the theoretical computation time, the granularity of data transfer needs to be decreased. This effectively optimizes the resources used by the computation and communication tasks, making the actual computation time and actual communication time closer, and avoiding the waste of bandwidth resources after the communication task is completed due to the computation process.

[0113] In this application, the management node can determine the transport granularity of a computing node based on computational parameters and communication parameters. The computational parameters include the computational load and theoretical computation time of the computational task, while the communication parameters include the communication load and theoretical communication time of the communication task. The theoretical computation time is the predicted computational time based on the computational load, the computing power of the computing node, and the memory bandwidth. The theoretical communication time is the predicted communication time based on the communication load and communication bandwidth. The transport granularity S can be expressed as S = f(computational parameters, communication parameters). f is a function obtained by fitting a large amount of sample data including computational and communication parameters; it can be a linear or nonlinear function, and this application does not impose specific limitations.

[0114] The following example illustrates the communication method provided in this application, assuming a full-mesh network of multiple computing nodes within the same communication domain. These nodes interact via `allreduce`, for instance, using the `reduce_scatter` and `all_gather` methods to implement `allreduc` for data exchange. If the communication domain includes four computing nodes: R1, R2, R3, and R4, where nodes R2, R3, and R4 perform matrix multiplication of two matrices of the same dimension with node R1, the matrix multiplication method is the same as that used by node R1. Each computing node divides the results of the first n rounds of computation into four parts, such as... Figure 8 As shown, Figure 8This is a schematic diagram of a communication process provided in this application. The four calculation results of the first computing node R1 are a1 to a4, the four calculation results of the second computing node R2 are b1 to b4, the four calculation results of the third computing node R3 are c1 to c4, and the four calculation results of the fourth computing node R4 are d1 to d4. It should be noted that in the above example, the first computing node R1 needs to complete the matrix multiplication operation of the first matrix A and the second matrix W. To complete the matrix multiplication operation of the first matrix A and the second matrix W, a total of 80 sets of sub-matrix matrix multiplication operations need to be completed. The calculation result of the matrix multiplication operation of each set of sub-matrixes is a matrix block, such as matrix O in Formula 1. 11 If n is 4, then matrix O1 comprises 80 matrix blocks. If n is 4, the first computation node R1 completes 4 rounds of computation, performing matrix multiplication on 20 sets of submatrices, resulting in 20 matrix blocks. The first computation result obtained by the first computation node R1 after 4 rounds of computation is divided into four parts, each of which comprises 5 matrix blocks, meaning a1 to a4 each comprise 5 matrix blocks.

[0115] Similarly, the second computing node R2, the third computing node R3, and the fourth computing node R4, along with the first computing node R1, include the same number of matrix computing units. They perform matrix multiplication of two matrices of the same dimension, and the method of performing matrix multiplication is the same as that of the first computing node R1.

[0116] The process of ensemble communication among the four computing nodes consists of two steps: First, each computing node executes `reduce_scatter`, and each node obtains a copy of data from the other computing nodes through its vector computation unit. For example, the first computing node obtains b1, c1, and d1, resulting in a data set e1 = a1 + b1 + c1 + d1. The five matrix blocks comprising a1, b1, c1, and d1 occupy the same position in the results of matrix multiplication operations performed on each computing node. For example, the five matrix blocks comprising a1 are O1, b1, c1, and d1. 11 O 12 O 13 O 14 and O 15 Then, the five matrix blocks included in b1 are in the same positions as the five matrix blocks mentioned above in the final matrix obtained by the second computing node. The second computing node obtains a2, c2, and d2, and obtains the data e2 = a2 + b2 + c2 + d2; the third computing node obtains a3, b3, and d3, and obtains the data e3 = a3 + b3 + c3 + d3; the third computing node obtains a4, b4, and c4, and obtains the data e4 = a4 + b4 + c4 + d4. It should be understood that each of the four data sets e1, e2, e3, and e4 includes five matrix blocks.

[0117] After each compute node executes reduce_scatter, the compute nodes synchronize their respective copies of data obtained through reduce_scatter using all_gather, ensuring that the data in each compute node is identical. After all_gather is executed on the four compute nodes, each compute node contains data consisting of four copies: e1, e2, e3, and e4, meaning that each compute node contains 20 matrix blocks.

[0118] It should be noted that when the vector computation unit retrieves data from the shared memory space of each computing node, it directly reads the data from the shared memory space of other computing nodes into the cache, then retrieves this data from the cache and performs data aggregation. For example, the vector computation unit of the first computing node retrieves data b1 from the shared memory space of the second computing node, reads b1 into the cache of the first computing node, and then performs an addition operation on a1 and b1. After completing the data aggregation to obtain aggregated data e1, the vector computation unit of the first computing node writes the aggregated data e1 into the standard address space of the first computing node and sends data e1 to the other three computing nodes. Simultaneously, it receives aggregated data e2, e3, and e4 sent by other computing nodes and writes them into the standard address space.

[0119] Figure 9 This is a schematic diagram illustrating the storage of computation results provided in this application. The first computing node performs a matrix multiplication operation on the first matrix A and the second matrix W, resulting in matrix O1. If the matrix multiplication operations performed by the second, third, and fourth computing nodes result in O2, O3, and O4 respectively, the result of data aggregation by the four computing nodes is matrix O. Matrix O comprises 80 matrix blocks, and e1, e2, e3, and e4 each contain 20 data blocks from matrix O. When the vector computation unit writes the aggregated data into the standard address space, data belonging to the same row and adjacent to each other within the 20 data blocks contained in e1, e2, e3, and e4 are written to adjacent storage areas. For example, e1 contains a data block E. 11 and E 12 E 11 according to Figure 7 Matrix O in 11 E 12 according to Figure 7 O in 12 If we obtain E, then 11 and E 12 Their positional relationship in matrix O, and their relation to matrix O 11 Sum of matrix O 12 Their positions in matrix O1 are the same. Matrix E 11 The included numerical values ​​and matrix E 12The included values ​​are stored in the standard address space in the following way: Figure 9 As shown in the diagram. Matrix E 11 The first row of data and matrix E 12 The first row of data is stored contiguously; matrix E 11 The second row of data and matrix E 12 The second row of data is stored consecutively.

[0120] After completing the communication task, the first computing node resets the value of the aforementioned flag bit to the second value through the vector computing unit, so that the matrix computing unit can reuse the shared memory space occupied by the first calculation result.

[0121] The above Figures 1 to 9 The corresponding embodiment describes the process by which a computing node performs one computing task and one communication task. Each computing node performs multiple computing and communication tasks according to the above method, thus completing the task instructed by the fusion operator of each computing node.

[0122] This application uses only one operator to carry instructions for performing computation and communication tasks, enabling decoupling from AI model learning frameworks. It can be easily called by different AI learning frameworks and acceleration libraries, and can be applied to various AI model learning frameworks and scenarios. For example, it can be applied to large-scale model training scenarios based on model parallelism, which includes a process of computation before communication, such as performing matrix multiplication before allreduce communication; it can also be applied to large-scale model training scenarios based on model parallelism and sequence parallelism, which includes a process of computation before communication, such as performing matrix multiplication before reduce_scatter communication, and a process of communication before computation, such as performing allgather communication before matrix multiplication; it can also be applied to large-scale model training scenarios based on MoE parallelism, which includes a process of computation before communication, such as performing matrix multiplication before all2all communication, and a process of communication before computation, such as performing all2all communication before matrix multiplication. In all the scenarios described above, whether the process involves computation before communication or communication before computation, it can be achieved by the management node sending fusion operators to the computing node. This will not be described in detail here.

[0123] For the sake of simplicity, the above method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application. Therefore, other reasonable combinations of steps that those skilled in the art can conceive of based on the above description also fall within the scope of protection of this application.

[0124] This application also provides a method such as Figure 3 The computing node shown includes multiple matrix calculation units used to perform the above operations. Figures 2 to 9 The operations performed by the matrix calculation unit in the corresponding embodiment, wherein the calculation node includes multiple vector calculation units for performing the above-mentioned operations. Figures 2 to 9 The operations performed by the vector calculation unit in the corresponding embodiments will not be described in detail here.

[0125] See Figure 10 , Figure 10 This is a schematic diagram of a computing device provided in this application, such as... Figure 10 As shown, the computing device 100 includes a bus 101, a processor 102, multiple computing nodes 103, a communication interface 104, and a memory 105. The processor 102, the multiple computing nodes 103, the communication interface 104, and the memory 105 communicate with each other via the bus 101. It should be understood that this application does not limit the number of processors 102, computing nodes 103, and memory 105 in the computing device 100. The computing device can be a server, such as a server in a cloud data center, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer or a laptop computer.

[0126] Bus 101 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 10The bus 101 is represented by a single line, but this does not mean that there is only one bus or one type of bus. Bus 101 may include pathways for transmitting information between various components of computing device 100. A unified bus, for example, is a Lingqu bus.

[0127] The processor 102 may be a central processing unit (CPU), or may include a CPU and other hardware chips. The aforementioned hardware chips may be of various types, such as any one of the following: a programmable logic device (PLD), a complex programmable logic device (CPLD), a field programmable gate array (FPGA), or a digital signal processor (DSP). The computing device 100 may include one or more of the aforementioned types of hardware chips, or may include multiple types of the aforementioned hardware chips. This application embodiment does not make specific limitations.

[0128] The computing node 103 can be a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or other dedicated AI processing chips, etc., and this application does not impose specific restrictions.

[0129] Memory 105 can be non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Memory 105 can also be volatile memory, which can be random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM). In addition, the memory 105 can also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0130] It should be noted that the same type of storage medium can be configured in the same computing device to realize the function of memory 105, or two or more types of storage media can be configured to realize the function of memory 105. This application does not limit this.

[0131] The memory 105 stores executable program code, which the processor 102 executes to issue a fusion operator to the computing node 103, enabling the computing node to implement the fusion operator. Figures 2 to 9 The distributed computing method described in the corresponding method embodiments.

[0132] The communication interface 104 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.

[0133] This application also provides a computer program product containing instructions, which may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When the computer program product runs on the computing device, it causes the computing device to perform... Figures 2 to 9 The distributed computing method described in the corresponding method embodiments.

[0134] This application also provides a computer-readable storage medium, which can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to implement... Figures 2 to 9 The distributed computing method described in the corresponding method embodiments.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A method of distributed computing, the method comprising: The method is applied to a distributed computing system including a management node and a plurality of computing nodes, and the method comprises: The management node issues a first fusion operator to a first computing node in the plurality of computing nodes; wherein the first fusion operator comprises a first matrix and a second matrix, and the first fusion operator further comprises instructions for instructing the first computing node to implement matrix multiplication of the first matrix and the second matrix; The first computing node performs part of the matrix multiplication of the first matrix and the second matrix by using a plurality of matrix calculation units in the first computing node according to the first fusion operator, to obtain a first calculation result; The first computing node performs a communication task with other computing nodes in the plurality of computing nodes, and the communication task is used to synchronize the first calculation result with the other computing nodes.

2. The method of claim 1, wherein, Each matrix calculation unit in the first computing node needs to perform m rounds of part of the matrix multiplication of the first matrix and the second matrix, and the first fusion operator further comprises sub-matrix information and a communication trigger parameter n; the sub-matrix information is used to indicate the size of a first sub-matrix and a second sub-matrix when each round of operation is performed by one matrix calculation unit, and the communication trigger parameter n is less than or equal to m, and m is a positive integer greater than 1; The first computing node performs part of the matrix multiplication of the first matrix and the second matrix by using a plurality of matrix calculation units in the first computing node according to the first fusion operator, to obtain a first calculation result, comprising: The first computing node performs part of the matrix multiplication of the first matrix and the second matrix by using a plurality of matrix calculation units in the first computing node according to the first fusion operator, to obtain a first calculation result, comprising:

3. The method of claim 2, wherein, The method further comprises: When the first computing node performs the communication task with the other computing nodes in the plurality of computing nodes, the plurality of matrix calculation units start to perform the n+1th round of calculation.

4. The method according to claim 2 or 3, characterized in that, The first fusion operator further comprises an output address, and the output address is used to indicate the address of a shared memory space of the first computing node; The method further comprises: The first computing node stores the first calculation result into the shared memory space according to the output address, wherein the shared memory space can be read and written by the other computing nodes.

5. The method of claim 4, wherein The first computing node stores the first calculation result into the shared memory space according to the output address, comprising: Each matrix calculation unit in the first computing node stores a plurality of rows of data included in a result matrix obtained by performing one round of calculation into the shared memory space in sequence, wherein the i+1th row of the result matrix is stored continuously with the ith row in the shared memory space.

6. The method according to any one of claims 2-5, characterized in that, The first fusion operator further comprises a carrying granularity, and the carrying granularity is used to indicate the amount of data acquired from the shared memory space of another computing node by the first computing node once. Before the management node issues the first fusion operator to the first computing node in the plurality of computing nodes, the method further comprises: The management node determines the carrying granularity according to the calculation amount and the calculation time length of n rounds of calculation and the communication amount and the communication time length of the communication task.

7. The method of claim 6, wherein, The first computing node performs a communication task with other computing nodes in the plurality of computing nodes, including: The first computing node executes a plurality of acquisition instructions to acquire the calculation result of each computing node from the shared memory space of the other computing nodes according to the carrying granularity.

8. The method according to any one of claims 1 to 7, characterized in that, The first computing node performs a communication task with other computing nodes in the plurality of computing nodes, including:

9. The method according to any one of claims 1 to 8, characterized in that, The first computing node queries a flag bit of each computing node in the other computing nodes, the flag bit being used to indicate whether the computing node has completed n rounds of calculation; The first computing node determines that the flag bit of each computing node is a first value, and performs a communication task with other computing nodes in the plurality of computing nodes; the first value of the flag bit is used to indicate that the computing node has completed n rounds of calculation. The matrix multiplication operation is a calculation in an artificial intelligence AI model training process; the computing node includes a graphics processing unit GPU, a neural network processing unit NPU, a tensor processing unit TPU, or other special AI processing chips.

10. The method according to any one of claims 1 to 9, characterized in that, The distributed computing system includes a management node and a plurality of computing nodes, the management node is used to perform the operations performed by the management node in any one of claims 1-10, and any one of the plurality of computing nodes is used to perform the operations performed by the first computing node in any one of claims 1-10.

11. A distributed computing system, characterized in that, The method applied to a computing node in a distributed computing system, the distributed computing system including a management node and a plurality of computing nodes, the method including:

12. A method of distributed computing, the method comprising: The first computing node acquires a first fusion operator issued by the management node; wherein the first fusion operator includes a first matrix and a second matrix, and the first fusion operator further includes an instruction for instructing the first computing node to implement a matrix multiplication operation of the first matrix and the second matrix; the first computing node is one of the plurality of computing nodes; The first computing node uses a plurality of matrix calculation units in the first computing node to respectively perform part of the matrix multiplication operation of the first matrix and the second matrix according to the first fusion operator, to obtain a first calculation result; The first computing node performs a communication task with other computing nodes in the plurality of computing nodes, the communication task being used to synchronize the first calculation result with the other computing nodes. Each matrix calculation unit in the first computing node needs to perform m rounds of part of the matrix multiplication operation of the first matrix and the second matrix, the first fusion operator further includes sub-matrix information and a communication trigger parameter n; the sub-matrix information is used to indicate the size of a first sub-matrix and a second sub-matrix when each round of operation is performed by a matrix calculation unit, and the communication trigger parameter n is less than or equal to m, m being a positive integer greater than 1; 13. The method of claim 12, wherein, ​ The first computing node performs, according to the first fusion operator, partial operations in matrix multiplication of the first matrix and the second matrix by using a plurality of matrix calculation units in the first computing node respectively, to obtain a first calculation result, including: The first computing node performs, according to the first fusion operator, partial operations in matrix multiplication of the first matrix and the second matrix by using a plurality of matrix calculation units in the first computing node respectively, to obtain a first calculation result, including:

14. The method of claim 13, wherein, The method further includes: When the first computing node performs a communication task with other computing nodes in the plurality of computing nodes, the plurality of matrix calculation units start to perform the (n+1)th round of calculation.

15. The method according to claim 13 or 14, characterized in that, The first fusion operator further includes an output address, which is used to indicate an address of a shared memory space of the first computing node. The method further includes: The first computing node stores the first calculation result into the shared memory space according to the output address, wherein the shared memory space can be read and written by the other computing nodes.

16. The method of claim 15, wherein The first computing node stores the first calculation result into the shared memory space according to the output address, including: Each matrix calculation unit in the first computing node stores, in sequence, a plurality of rows of data included in a result matrix obtained by performing one round of calculation, into the shared memory space, wherein an (i+1)th row of the result matrix is stored continuously with an ith row in the shared memory space.

17. The method according to any one of claims 12-16, characterized by, The first fusion operator further includes a carrying granularity, which is used to indicate an amount of data acquired from a shared memory space of another computing node by the first computing node in one acquisition instruction; The first computing node performs a communication task with other computing nodes in the plurality of computing nodes, including: The first computing node performs a plurality of acquisition instructions to acquire the calculation result of each computing node from the shared memory space of the other computing nodes according to the carrying granularity.

18. The method according to any one of claims 12-17, characterized by, The first computing node performs a communication task with the other computing nodes based on a memory access instruction.

19. The method according to any one of claims 12-18, characterized in that, The first computing node performs a communication task with other computing nodes in the plurality of computing nodes, including: The first computing node queries a flag bit of each computing node in the other computing nodes, the flag bit being used to indicate whether the computing node has completed n rounds of calculation; The first computing node determines that the flag bit of each computing node is a first value, and performs a communication task with other computing nodes in the plurality of computing nodes; the first value of the flag bit is used to indicate that the computing node has completed n rounds of calculation.

20. The method according to any one of claims 12-19, characterized by, The matrix multiplication operation is a calculation in an artificial intelligence AI model training process; the computing node includes a graphics processing unit GPU, a neural network processing unit NPU, a tensor processing unit TPU or other special AI processing chips.

21. A computing node, characterized in that, The application is applied to a distributed computing system, the distributed computing system including a management node and a plurality of computing nodes, and a first computing node including a plurality of matrix calculation units and a plurality of vector calculation units, wherein The plurality of matrix calculation units are configured to receive a first fusion operator issued by a management node, the first fusion operator comprising a first matrix and a second matrix, and the first fusion operator further comprising an instruction for instructing the first calculation node to implement a matrix multiplication operation of the first matrix and the second matrix; the first calculation node is one of the plurality of calculation nodes; The plurality of matrix calculation units are further configured to respectively perform partial operations in the matrix multiplication operation of the first matrix and the second matrix according to the first fusion operator, to obtain a first calculation result. The plurality of vector calculation units are configured to perform a communication task between the first calculation node and other calculation nodes in the plurality of calculation nodes, and the communication task is configured to synchronize the first calculation result with the other calculation nodes.

22. The node of claim 21, wherein, Each matrix calculation unit in the first calculation node needs to perform m rounds of partial operations of the first matrix and the second matrix, the first fusion operator further comprises sub-matrix information and a communication trigger parameter n; the sub-matrix information is configured to indicate sizes of a first sub-matrix and a second sub-matrix when each matrix calculation unit performs each round of operation, and the communication trigger parameter n is less than or equal to m, and m is a positive integer greater than 1. The plurality of matrix calculation units are configured to perform the partial operations of the matrix multiplication operation in parallel according to the sub-matrix information, until n rounds of calculation are performed, to obtain the first calculation result.

23. The node of claim 22, wherein, The plurality of matrix calculation units are further configured to: start to perform an (n+1)th round of calculation when the plurality of vector calculation units perform the communication task between the first calculation node and the other calculation nodes in the plurality of calculation nodes.

24. The node of claim 22 or 23, characterized by The first calculation node further comprises a memory, the memory comprises a shared memory space, the first fusion operator further comprises an output address, and the output address is configured to indicate an address of the shared memory space of the memory. The plurality of matrix calculation units are further configured to: store the first calculation result into the shared memory space according to the output address, and the shared memory space can be read and written by the other calculation nodes.

25. The node of claim 24, wherein: the plurality of matrix calculation units are specifically configured to sequentially store a plurality of rows of data included in a result matrix obtained by performing one round of calculation into the shared memory space, and an (i+1)th row of the result matrix is stored continuously with an ith row in the shared memory space.

26. The node of any of claims 21-25, wherein, The first fusion operator further comprises a carrying granularity, and the carrying granularity is configured to indicate an amount of data acquired from a shared memory space of another calculation node by the first calculation node once according to an acquisition instruction; the plurality of vector calculation units are specifically configured to perform the acquisition instruction multiple times to acquire the calculation result of each calculation node from the shared memory space of the other calculation node according to the carrying granularity.

27. The node of any of claims 21-26, wherein, The plurality of vector calculation units perform the communication task between the first calculation node and the other calculation nodes based on a memory access instruction.

28. The node of any of claims 21-27, wherein, The plurality of vector calculation units are specifically configured to: query a flag bit of each of the other computing nodes, the flag bit being used to indicate whether the computing node has completed a round of computation; the second vector computation unit is one of the plurality of vector computation units; after determining that the flag bit of each of the computing nodes is a first value, perform a communication task with other computing nodes of the plurality of computing nodes; the first value of the flag bit is used to indicate that the computing node has completed n rounds of computation.

29. The node of any of claims 21-28, wherein, The matrix multiplication operation is a calculation in an artificial intelligence (AI) model training process. The computing node includes a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), or other special AI processing chips.

30. A computer-readable storage medium, characterized in that, The computer program instructions, when executed by a computing device, cause the computing device to perform the method of any one of claims 12-20.

Images