Computing system, model training method and computing device

Abstract

A computing system, a model training method and a computing device. The computing system comprises a first processing unit and a plurality of second processing units, wherein the first processing unit can instruct a plurality of local second processing units to divide a matrix of computation results into a plurality of blocks; and the plurality of second processing units in the computing system perform matrix multiplication computation on submatrices obtained after division by the first processing unit, and sequentially send, on the basis of the generation order of blocks in a submatrix of computation results, the blocks in the submatrix of computation results. Each second processing unit partitions a submatrix of computation results into a plurality of blocks, such that the time for the second processing unit to send a computation result each time can be reduced, thereby preventing the communication bandwidth of a processing unit or a memory, which receives results from a plurality of second processing units, from remaining in an idle state for a long time. The data volume of each block is relatively small, so that the processing unit or the memory receiving the results from the plurality of second processing units is less prone to a communication bottleneck, thereby achieving the effect of reducing the cost.

Description

Computing system, model training method and computing device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on February 7, 2024, with application number 202410175686.1 and application name “A computing system, model training method and computing device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of artificial intelligence technology, and in particular to a computing system, a model training method, and a computing device. Background Art

[0003] With the development of models such as GPT and chat AI, the number of model parameters is increasing, reaching hundreds of billions. If the number of model parameters or the intermediate values ​​generated by the calculations exceed the video memory capacity of a single graphics processing unit (GPU), a single GPU cannot complete the model training task alone. Therefore, the model can use a distributed training method with model parallelism to allow multiple GPUs to complete the model training task in parallel. When the number of model parameters or intermediate values ​​are distributed to multiple GPUs for training, multiple GPUs need to communicate with each other so that the parameter quantity of one GPU can be transferred to another GPU. Under normal circumstances, the communication bandwidth between GPUs is very small, and it is difficult to meet the communication needs of multiple GPUs for model training.

[0004] Summary of the Invention

[0005] To address the aforementioned issues, embodiments of the present application provide a computing system, a model training method, and a computing device. Multiple GPUs in the computing system can divide the matrix of computational results into multiple blocks, reducing the time it takes for the GPUs to send computational results each time, thereby reducing the communication bandwidth required for each GPU transmission and avoiding communication bottlenecks in the processing units or memories receiving multiple GPUs. Furthermore, the present application also provides a model training method and computing device corresponding to the computing system.

[0006] To this end, the following technical solutions are adopted in the embodiments of the present application:

[0007] In a first aspect, an embodiment of the present application provides a computing system, comprising: multiple second processing units, for receiving a first sub-matrix and a second sub-matrix, and performing matrix multiplication calculations on the first sub-matrix and the second sub-matrix, and sending the blocks of the third sub-matrix in sequence according to the generation order of the blocks in the third sub-matrix; wherein the first sub-matrix is ​​obtained by the first processing unit splitting the first matrix to be performed on the matrix multiplication, the second sub-matrix is ​​obtained by the first processing unit splitting the second matrix to be performed on the matrix multiplication, and the third sub-matrix is ​​a sub-matrix obtained by matrix multiplication of the first sub-matrix and the second sub-matrix.

[0008] In this embodiment, multiple second processing units in the computing system perform matrix multiplication on the submatrices split by the first processing unit and sequentially transmit the blocks of the submatrix of the calculation results in the order in which the blocks were generated. By splitting the submatrix of the calculation results into multiple blocks, the second processing units can reduce the time it takes to transmit the calculation results each time, thereby preventing the communication bandwidth of the processing units or memories receiving the results of the multiple second processing units from being idle for long periods of time. Because the amount of data in each block is relatively small, the processing units or memories receiving the results of the multiple second processing units are less likely to reach a communication bottleneck, thereby reducing costs.

[0009] In one embodiment, the second processing unit is specifically used to generate a first message according to the block of the third sub-matrix and send the first message; the first message includes a target address, a data length, a data type and a data payload, the target address refers to the location of the target storage space where the third processing unit stores the block of the third sub-matrix, the data length refers to the size of the block of the third sub-matrix, the data type refers to the data type of the block of the third sub-matrix, and the data payload is used to record the block of the third sub-matrix, and the third processing unit is a second processing unit or a memory among the multiple second processing units.

[0010] In this embodiment, the second processing unit can convert the third sub-matrix into a first message, so that the second processing unit can transmit it to the third processing unit via a communication bus, thereby increasing the communication mode between the second processing unit and the third processing unit. The second processing unit can add the storage address and data length stored in the third processing unit to the first message, so that after the third sub-matrix is ​​sent to the third processing unit, it can be stored in a specified location, and the third sub-matrix in the same storage location can be subjected to matrix addition calculation, thereby enabling the third processing unit to perform a matrix addition calculation function. The second processing unit can add a data type to the first message, allowing data of the same type to perform matrix addition calculation, thereby avoiding matrix addition calculation of data of different types, which may cause errors in the results processed by the third processing unit.

[0011] In one embodiment, the third processing unit is used to parse out the blocks of the third submatrix in the first message after receiving the first message from the multiple second processing units, and perform matrix addition calculation on the multiple blocks of the third submatrix to obtain blocks of the third matrix; the third matrix is ​​a matrix obtained by performing matrix addition calculation on the third submatrix generated by the multiple second processing units.

[0012] In this embodiment, after obtaining the result matrices of the multiple second processing units, the third processing unit performs matrix addition calculation on the result matrices at the same storage position to obtain the final calculation result, so that the third processing unit can perform the matrix addition calculation function.

[0013] In one embodiment, the third processing unit is specifically used to determine the target storage space based on the target address and the data length in the first message; read the data stored in the target storage space, perform matrix addition calculation on the data recorded in the data payload in the first message and the data stored in the target storage space to obtain blocks of the third matrix; clear the data stored in the target storage space, and write the blocks of the third matrix into the target storage space.

[0014] In this embodiment, after receiving a first message, the third processing unit can perform matrix addition calculation on the third sub-matrix carried by itself and the data at the same storage position stored locally according to the storage position in the first message, so that the third sub-matrix at the same storage position can be subjected to matrix addition calculation, thereby enabling the third processing unit to perform the matrix addition calculation function.

[0015] In one embodiment, the third processing unit is specifically used to obtain multiple first messages cached locally, perform matrix addition calculation on the data recorded in the data payload of the first message with the same target address and data length to obtain superimposed blocks; determine the storage space corresponding to the superimposed blocks based on the target address and the data length corresponding to the superimposed blocks; read the data stored in the storage space corresponding to the superimposed blocks, perform matrix addition calculation on the superimposed blocks and the data stored in the storage space corresponding to the superimposed blocks to obtain blocks of the third matrix; clear the data stored in the storage space corresponding to the superimposed blocks, and write the blocks of the third matrix into the storage space corresponding to the superimposed blocks.

[0016] In this embodiment, after receiving multiple first messages with the same destination address and data length, the third processing unit performs matrix addition calculations on the blocks of the third submatrix of the multiple first messages. The third processing unit performs matrix addition calculations on the superimposed blocks with the locally stored data at the same storage location based on the storage location of the superimposed blocks, thereby reducing the amount of computation and processing time.

[0017] In one embodiment, the multiple second processing units are specifically used to obtain the destination address from the preset destination address and the data length from the preset data length according to the generation order of the blocks in the third sub-matrix, and configure the destination address and data length for the first message corresponding to the blocks in the third sub-matrix.

[0018] In this embodiment, multiple second processing units can assign the same destination address and data length to the blocks at the same position in their respective third sub-matrices, so that the third processing unit can perform matrix addition calculations on the blocks at the same position in multiple third sub-matrices, thereby enabling the third processing unit to perform the matrix addition calculation function.

[0019] In one embodiment, the third processing unit is a second processing unit among the plurality of second processing units, and the third processing unit is further configured to send the blocks of the third matrix to a second processing unit connected to a local output port.

[0020] In this embodiment, each second processing unit can simultaneously perform matrix multiplication calculations on a sub-matrix, then send the completed sub-matrix to the next second processing unit, and so on, until all second processing units have stored the matrix for matrix multiplication calculations. Each second processing unit simultaneously sends the calculation results to the next second processing unit, ensuring that each second processing unit is in a communication state at all times, avoiding the problem of communication bottlenecks caused by multiple second processing units transmitting at a single time.

[0021] In one embodiment, the second processing unit is further used to generate a second message based on the blocks of the third matrix and transmit the second message to a local memory; the second message is used to instruct the local memory to store the blocks of the third matrix in a designated storage space.

[0022] In this embodiment, when the second processing unit transmits the generated blocks of the third matrix to the next second processing unit, the generated blocks of the third matrix can also be stored in a local memory, so that after multiple second processing units in the computing system cycle multiple times, each second processing unit stores a complete third matrix, so that each second processing unit has training results to perform different computing tasks.

[0023] In one embodiment, the first processing unit is further used to send a configuration instruction to the third processing unit; the configuration instruction is used to instruct the third processing unit to configure a storage space for storing the third matrix; the third processing unit is further used to, after receiving the configuration instruction, configure a storage space of a set size in the local storage space, and clear the storage space of the set size to 0.

[0024] In this embodiment, the first processing unit can instruct the third processing unit to configure storage space for the data of this training, and clear the storage space to 0, so that the third processing unit can store multiple calculation results of the second processing unit, and store the results of matrix addition calculation of multiple calculation results of the second processing unit.

[0025] In one embodiment, the first processing unit is further configured to send segmentation granularity information to the multiple second processing units; and the multiple second processing units are further configured to, after receiving the segmentation granularity information, divide the third submatrix into multiple blocks according to the segmentation granularity information.

[0026] In this embodiment, the first processing unit can assign segmentation granularity information to each second processing unit, dividing the submatrix output by the second processing unit into multiple blocks. This allows the second processing unit to send calculation results based on the size of the blocks. This can reduce the time it takes for each second processing unit to send calculation results, preventing the communication bandwidth of the processing unit or memory receiving the results of multiple second processing units from being idle for long periods of time. Because the amount of data in each block is relatively small, the processing unit or memory receiving the results of multiple second processing units is less likely to reach a communication bottleneck, thereby reducing costs.

[0027] In one embodiment, the multiple second processing units are also used to send a third message to the third processing unit; the third message includes a matrix identifier, and the matrix identifier is used to allow the reading device to determine the second processing unit to which the read data belongs; the third processing unit is also used to, after receiving the third message, perform an OR operation on the matrix identifier in the third message and the data already stored in the identifier storage space, and store the result of the OR operation in the identifier storage space; the identifier storage space is a storage space for storing matrix identifiers in the third processing unit.

[0028] In this embodiment, each second processing unit also sends a third message each time it sends a calculation result. The matrix identifier carried in the third message can indicate the sender of the calculation result, allowing the reading device to determine the data it needs to read based on the matrix identifier. After the third processing unit receives the calculation result and the matrix identifier, since the calculation results in the same storage location are calculated using matrix addition, it may be difficult to determine which second processing unit's calculation result the superimposed calculation result belongs to. Therefore, the third processing unit can use an OR operation on the matrix identifiers corresponding to the calculation results stored in the same storage space to avoid the reading device being unable to recognize the superimposed matrix identifiers.

[0029] In the second aspect, an embodiment of the present application provides a model training method, wherein the computing system includes a first processing unit and multiple second processing units, and the method is executed by a second processing unit among the multiple second processing units, and the method includes: receiving a first sub-matrix and a second sub-matrix; the first sub-matrix is ​​obtained by the first processing unit splitting the first matrix to be performed matrix multiplication, and the second sub-matrix is ​​obtained by the first processing unit splitting the second matrix to be performed matrix multiplication; performing matrix multiplication calculation on the first sub-matrix and the second sub-matrix; sending the blocks of the third sub-matrix in sequence according to the generation order of the blocks in the third sub-matrix; the third sub-matrix is ​​a sub-matrix obtained by matrix multiplication of the first sub-matrix and the second sub-matrix.

[0030] In one embodiment, the method further includes: generating a first message based on the block of the third sub-matrix, and sending the first message; the first message includes a target address, a data length, a data type and a data payload, the target address refers to the location of the target storage space where the third processing unit stores the block of the third sub-matrix, the data length refers to the size of the block of the third sub-matrix, the data type refers to the data type of the block of the third sub-matrix, and the data payload is used to record the block of the third sub-matrix, and the third processing unit is a second processing unit or a memory among the multiple second processing units.

[0031] In one embodiment, the method further includes: after receiving the first message sent by other second processing units, parsing out the blocks of the third submatrix in the first message; performing matrix addition calculation on multiple blocks of the third submatrix to obtain blocks of the third matrix; the third matrix is ​​a matrix obtained by performing matrix addition calculation on the third submatrices generated by the multiple second processing units.

[0032] In one embodiment, performing matrix addition calculation on multiple blocks of the third sub-matrix to obtain blocks of the third matrix specifically includes: determining the target storage space based on the target address and the data length in the first message; reading the data stored in the target storage space, performing matrix addition calculation on the data recorded in the data payload in the first message and the data stored in the target storage space to obtain blocks of the third matrix; clearing the data stored in the target storage space, and writing the blocks of the third matrix into the target storage space.

[0033] In one embodiment, performing matrix addition calculation on multiple blocks of the third sub-matrix to obtain blocks of the third matrix specifically includes: obtaining multiple first messages cached locally, performing matrix addition calculation on the data recorded in the data payload in the first message having the same target address and data length to obtain superimposed blocks; determining the storage space corresponding to the superimposed blocks based on the target address and the data length corresponding to the superimposed blocks; reading the data stored in the storage space corresponding to the superimposed blocks, performing matrix addition calculation on the superimposed blocks and the data stored in the storage space corresponding to the superimposed blocks to obtain blocks of the third matrix; clearing the data stored in the storage space corresponding to the superimposed blocks, and writing the blocks of the third matrix into the storage space corresponding to the superimposed blocks.

[0034] In one embodiment, the method further includes: obtaining a destination address from a preset destination address and obtaining a data length from a preset data length according to the generation order of the blocks in the third sub-matrix, and configuring the destination address and data length for the first message corresponding to the blocks in the third sub-matrix.

[0035] In one embodiment, the method further includes: generating a second message based on the blocks of the third matrix, and transmitting the second message to a local memory; the second message is used to instruct the local memory to store the blocks of the third matrix in a designated storage space.

[0036] In one embodiment, before receiving the first sub-matrix and the second sub-matrix, the method further includes: after receiving a configuration instruction sent by the first processing unit, configuring a storage space of a set size in a local storage space, and clearing the storage space of the set size to 0.

[0037] In one embodiment, before receiving the first sub-matrix and the second sub-matrix, the method further includes: after receiving the segmentation granularity information sent by the first processing unit, dividing the third sub-matrix into a plurality of blocks according to the segmentation granularity information.

[0038] In one embodiment, the method further includes: after receiving the third message, performing an OR operation on the matrix identifier in the third message and the data already stored in the identifier storage space, and storing the result of the OR operation in the identifier storage space; the identifier storage space is a storage space for locally storing matrix identifiers.

[0039] In a third aspect, an embodiment of the present application provides a model training device, comprising: a transceiver unit for receiving a first sub-matrix and a second sub-matrix; the first sub-matrix is ​​obtained by the first processing unit splitting the first matrix to be subjected to matrix multiplication, and the second sub-matrix is ​​obtained by the first processing unit splitting the second matrix to be subjected to matrix multiplication; a processing unit for performing matrix multiplication calculation on the first sub-matrix and the second sub-matrix; the blocks of the third sub-matrix are sent sequentially according to the generation order of the blocks in the third sub-matrix; the third sub-matrix is ​​a sub-matrix obtained by matrix multiplication of the first sub-matrix and the second sub-matrix.

[0040] In one embodiment, the processing unit is further used to generate a first message based on the blocks of the third sub-matrix and send the first message; the first message includes a target address, a data length, a data type and a data payload, the target address refers to the location of the target storage space where the third processing unit stores the blocks of the third sub-matrix, the data length refers to the size of the blocks of the third sub-matrix, the data type refers to the data type of the blocks of the third sub-matrix, and the data payload is used to record the blocks of the third sub-matrix, and the third processing unit is a second processing unit or a memory among the multiple second processing units.

[0041] In one embodiment, the transceiver unit is further used to parse out the blocks of the third submatrix in the first message after receiving the first message sent by other second processing units; the processing unit is further used to perform matrix addition calculation on multiple blocks of the third submatrix to obtain blocks of the third matrix; the third matrix is ​​a matrix obtained by performing matrix addition calculation on the third submatrices generated by the multiple second processing units.

[0042] In one embodiment, the processing unit is specifically used to determine the target storage space based on the target address and the data length in the first message; read the data stored in the target storage space, perform matrix addition calculation on the data recorded in the data payload in the first message and the data stored in the target storage space to obtain blocks of the third matrix; clear the data stored in the target storage space, and write the blocks of the third matrix into the target storage space.

[0043] In one embodiment, the transceiver unit is specifically used to obtain multiple first messages cached locally; the processing unit is specifically used to perform matrix addition calculation on the data recorded in the data payload in the first message with the same target address and data length to obtain superimposed blocks; determine the storage space corresponding to the superimposed blocks based on the target address and the data length corresponding to the superimposed blocks; read the data stored in the storage space corresponding to the superimposed blocks, perform matrix addition calculation on the superimposed blocks and the data stored in the storage space corresponding to the superimposed blocks, to obtain the blocks of the third matrix; clear the data stored in the storage space corresponding to the superimposed blocks, and write the blocks of the third matrix into the storage space corresponding to the superimposed blocks.

[0044] In one embodiment, the processing unit is further used to obtain the destination address from the preset destination address and the data length from the preset data length according to the generation order of the blocks in the third sub-matrix, and configure the destination address and data length for the first message corresponding to the blocks in the third sub-matrix.

[0045] In one embodiment, the processing unit is further used to generate a second message based on the blocks of the third matrix and transmit the second message to a local memory; the second message is used to instruct the local memory to store the blocks of the third matrix in a designated storage space.

[0046] In one embodiment, the transceiver unit is further configured to configure a storage space of a set size in the local storage space after receiving the configuration instruction sent by the first processing unit, and clear the storage space of the set size to 0.

[0047] In one embodiment, the transceiver unit is further configured to, after receiving the segmentation granularity information sent by the first processing unit, divide the third sub-matrix into a plurality of blocks according to the segmentation granularity information.

[0048] In one embodiment, the transceiver unit is further used to, after receiving the third message, perform an OR operation on the matrix identifier in the third message and the data already stored in the identifier storage space, and store the result of the OR operation in the identifier storage space; the identifier storage space is a storage space for locally storing matrix identifiers.

[0049] In a fourth aspect, an embodiment of the present application provides a computing device, comprising: at least one memory; and at least one processor, the processor being configured to execute instructions stored in the memory so that the computing device executes various possible implementations of the second aspect.

[0050] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the various possible implementations of the second aspect.

[0051] In a sixth aspect, an embodiment of the present application provides a computer program product comprising instructions, characterized in that the computer program product stores instructions that, when executed by a computing device, enable the computing device to implement each possible implementation embodiment of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The following is a brief introduction to the drawings required for describing the embodiments or prior art.

[0053] Figure 1 is a schematic diagram of the architecture of a transformer model;

[0054] FIG2 is a flow chart of a model provided in the related art that is trained using a tensor parallel approach;

[0055] FIG3 is a schematic diagram of the structure of a computing system provided in an embodiment of the present application;

[0056] FIG4 is a flow chart of a computing system provided in an embodiment of the present application performing training using a tensor parallel approach;

[0057] FIG5 is a schematic diagram of the structure of another computing system provided in an embodiment of the present application;

[0058] FIG6 is a schematic diagram of the structure of a computing system provided in an embodiment of the present application that performs training using a ring full specification method;

[0059] FIG7( a ) is a schematic diagram showing the distribution of sub-matrices allocated to each GPU after the CPU divides five output matrices according to an embodiment of the present application;

[0060] FIG7( b ) is a schematic diagram of submatrices stored on each GPU after the first iteration of the scatter-reduce phase according to an embodiment of the present application;

[0061] FIG7( c ) is a schematic diagram of submatrices stored on each GPU after the second iteration of the scatter-reduce phase according to an embodiment of the present application;

[0062] FIG7( d ) is a schematic diagram of submatrices stored on each GPU after the final iteration of the scatter-reduce phase according to an embodiment of the present application;

[0063] FIG7( e ) is a schematic diagram of sub-matrices stored by each GPU after the first iteration of the entire collection phase according to an embodiment of the present application;

[0064] FIG7( f ) is a schematic diagram of sub-matrices stored by each GPU after the final iteration of all collection phases provided in an embodiment of the present application;

[0065] FIG8 is a flow chart of a model training method provided in an embodiment of the present application;

[0066] FIG9 is a schematic structural diagram of a model training device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0068] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0069] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0070] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0071] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0072] Before introducing the technical solution protected by this application, several terms that appear in the technical solution protected by this application are introduced, namely:

[0073] In-network computing refers to the use of computing devices such as network cards and switches to perform online data calculations simultaneously during data transmission, so as to reduce communication delays and improve overall computing efficiency.

[0074] The model can be a transformer model, a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a self-attention model, etc. The following takes the transformer model as an example.

[0075] For example, Figure 1 is a schematic diagram of the architecture of a transformer model. As shown in Figure 1, the transformer model mainly consists of an encoder and a decoder. The encoder's input embedding can map each element in the input sequence into a low-dimensional vector representation through an embedding matrix. The encoder's positional encoding can add a fixed-length vector representation to each position to indicate the relative position information of the position in the sequence.

[0076] The encoder consists of multiple layers of identical encoding blocks. Each encoding block consists of two sublayers and a residual connection. The first sublayer is a multi-head self-attention layer. This layer uses multiple attention heads to model the relationships between positions in the input sequence, finding correlations between each position and other positions, thereby extracting global information from the input sequence.

[0077] The second sublayer is a position-based fully connected network layer, such as a feed-forward neural network (FFN). The fully connected network can serve as a supplement to the multi-head self-attention layer to perform nonlinear transformations on the features of each position, thereby enhancing the representation ability of the input sequence.

[0078] Residual connections can add the input sequence to the output of the multi-head self-attention layer and the fully connected network layer, and after layer normalization, serve as the input sequence for the next layer. The output of the last encoding block of the encoder can be a vector of global information of the input sequence, which serves as the input sequence of the decoder.

[0079] The decoder's target embedding maps each element in the target sequence into a low-dimensional vector representation through the embedding matrix. The decoder's positional encoding is the same as the encoder's, adding a fixed-length vector representation to each position to indicate its relative position in the sequence.

[0080] The decoder consists of multiple layers of identical decoding blocks. Each decoding block consists of three sublayers and a residual connection. The first sublayer is a multi-head self-attention mechanism. This multi-head self-attention mechanism is similar to the encoder's multi-head self-attention mechanism. Using multiple attention heads, it models the relationships between positions in the target sequence and finds the correlation between each position and every other position.

[0081] The second sublayer is the multi-head encoder-decoder attention layer. Unlike the multi-head self-attention sublayer, the multi-head encoder-decoder attention layer calculates the correlation between the decoder's current position and each position in the encoder's output sequence, thereby incorporating the encoder's global information into the decoder.

[0082] The third sublayer is a position-based fully connected network layer. The fully connected network layer can serve as a supplement to the multi-head encoder-decoder attention layer to perform nonlinear transformations on the features of each position, thereby enhancing the representation ability of the input sequence.

[0083] Residual connections add the input sequence to the outputs of the multi-head self-attention layer, the multi-head encoder-decoder attention layer, and the fully connected network layer. After layer normalization, this is used as the input sequence for the next layer. The output of the last decoding block of the decoder is processed by the softmax function to obtain the predicted sequence.

[0084] When a layer in a transformer model performs forward computation, it involves matrix multiplication XA=Y. Here, X and A represent the parameters of the current layer, and Y represents the output. For example, let's take X as the weight parameter and A as the activation parameter. In the transformer model, weight parameters exist in the form of matrices and are the weights connecting different neurons in the neural network, used to transmit information and calculate outputs. Activation parameters also exist in the form of matrices and refer to the output matrix of each layer in the neural network.

[0085] The transformer model uses a variety of model-parallel distributed training methods, such as tensor parallelism and ring allreduce. When using tensor parallelism, the transformer model can partition the weight and activation matrices, dividing a larger weight matrix into multiple smaller weight matrices and a larger activation matrix into multiple smaller activation matrices. These multiple smaller weight matrices and multiple smaller activation matrices are then distributed to multiple GPUs. After each GPU receives a smaller weight matrix and a smaller activation matrix, it performs matrix multiplication to obtain a smaller activation matrix for the next layer. Each GPU can input its own smaller activation matrix for the next layer into another GPU, which then performs matrix addition on multiple smaller activation matrices for the next layer to obtain the complete activation matrix for the next layer.

[0086] For example, FIG2 is a flow chart of a model provided in the related art using a tensor parallel method for training. As shown in FIG2, the first matrix X is The second matrix A is The number of GPUs used for transformer model training is 3, namely GPU 0, GPU 1 and GPU 2.

[0087] When training the transformer model, the first matrix X can be split into two sub-matrices according to the number of GPUs, namely sub-matrices and submatrix Split the second matrix A into two sub-matrices, and submatrix The transformer model can then assign submatrix X1 and submatrix A1 to GPU 0, and submatrix X2 and submatrix A2 to GPU 1.

[0088] After receiving submatrix X1 and submatrix A1, GPU 0 performs matrix multiplication on submatrix X1 and submatrix A1 to obtain submatrix Y1. The element in the first row and first column of submatrix Y1 is 0×10 + 1×11 = 11. The element in the first row and second column of submatrix Y1 is 0×14 + 1×15 = 15. The element in the second row and first column of submatrix Y1 is 4×10 + 4×11 = 95. The element in the second row and second column of submatrix Y1 is 4×14 + 5×15 = 131.

[0089] After receiving submatrix X2 and submatrix A2, GPU 1 performs matrix multiplication on submatrix X2 and submatrix A2 to obtain submatrix Y2. The element in the first row and first column of submatrix Y2 is 2×12 + 3×13 = 63. The element in the first row and second column of submatrix Y2 is 2×16 + 3×17 = 83. The element in the second row and first column of submatrix Y2 is 6×12 + 7×13 = 163. The element in the second row and second column of submatrix Y2 is 6×16 + 7×17 = 215.

[0090] GPU 0 can input submatrix Y1 to GPU 2. GPU 1 can input submatrix Y2 to GPU 2. After receiving submatrix Y1 and submatrix Y2, GPU 1 performs matrix addition on submatrix Y1 and submatrix Y2 to obtain the final matrix Y. Here, the element in the first row and first column of matrix Y is 11 + 63 = 74. The element in the first row and second column of matrix Y is 15 + 83 = 98. The element in the second row and first column of matrix Y is 95 + 163 = 258. The element in the second row and second column of matrix Y is 131 + 215 = 346.

[0091] When parallelizing the training parameters of a transformer model across multiple GPUs, it's necessary to choose how to distribute different operations to the available GPUs. Taking the simplest communication mechanism as an example, after receiving submatrices Xn and An, GPUs 0 and 1 perform matrix multiplication on them, generating submatrix Yn. GPUs 0 and 1 wait for the matrix multiplication to complete locally before sending the resulting submatrix Yn to GPU 2. As can be seen, if GPUs 0 and 1 receive very large matrices Xn and An, such as those with a scale of 1024×1024 or 2048×2048, the matrix multiplication takes a long time to complete, leaving GPU 2's communication bandwidth idle for extended periods. Furthermore, after receiving the computation results from GPUs 0 and 1, GPU 2 is likely to reach a communication bottleneck due to the large computation results, resulting in a high communication cost.

[0092] To address deficiencies in the related art, an embodiment of the present application provides a computing system. The computing system includes a first processing unit and multiple second processing units. After receiving a first sub-matrix and a second sub-matrix, the multiple second processing units perform matrix multiplication on the first sub-matrix and the second sub-matrix. The multiple second processing units can sequentially send blocks of the third sub-matrix according to the order in which the blocks in the third sub-matrix were generated. The first sub-matrix is ​​obtained by the first processing unit by splitting the first matrix to be subjected to matrix multiplication. The second sub-matrix is ​​obtained by the first processing unit by splitting the second matrix to be subjected to matrix multiplication. The third sub-matrix is ​​the sub-matrix obtained by performing matrix multiplication on the first sub-matrix and the second sub-matrix.

[0093] In an embodiment of the present application, the first processing unit can instruct multiple local second processing units to divide the matrix of the calculation results into multiple blocks. When each second processing unit performs a matrix multiplication calculation, it sends out the blocks of the matrix of the calculation results in sequence according to the generation order of the blocks in the matrix of the calculation results. This can reduce the time it takes for the second processing unit to send the calculation results each time, and avoid the communication bandwidth of the processing unit or memory that receives the results of multiple second processing units being idle for a long time. Since the amount of data in each block is relatively small, the processing unit or memory that receives the results of multiple second processing units is not likely to reach a communication bottleneck, thereby achieving the effect of reducing costs.

[0094] The following will introduce the implementation process of the technical solution protected by this application from the perspectives of a computing system that is trained in a tensor parallel manner and a computing system that is trained in a ring full specification manner.

[0095] Figure 3 is a schematic diagram of the structure of a computing system provided in an embodiment of the present application. As shown in Figure 3, computing system 300 includes a first processing unit 310, multiple second processing units 320, a communication bus 330, and a third processing unit 340. The first processing unit 310, multiple second processing units 320, and third processing unit 340 establish communication connections via the communication bus 330 for data transmission.

[0096] The first processing unit 310 can be a processor that manages multiple second processing units 320, such as a central processing unit (CPU), a neural processing unit (NPU), etc. The first processing unit 310 is used to receive parameter quantities and intermediate values ​​for model training. The first processing unit 310 is used to split the parameter quantities or intermediate values ​​in matrix form to obtain multiple small matrices. The first processing unit 310 is used to send multiple small matrices to each second processing unit 320 respectively. The first processing unit 310 is used to send a program to the second processing unit 320, so that each second processing unit 320 performs matrix multiplication calculations. And other functions.

[0097] In the embodiment of the present application, after receiving the first matrix and the second matrix to be subjected to matrix multiplication, the first processing unit 310 may divide the first matrix into a corresponding number of first sub-matrices, and divide the second matrix into a corresponding number of second sub-matrices, according to the number of second processing units 320. Then, the first processing unit 310 may respectively send the multiple first sub-matrices and the multiple second sub-matrices to the multiple second processing units 320, so that each second processing unit can receive one first sub-matrix and one second sub-matrix.

[0098] The second processing unit 320 can be the above-mentioned GPU, or it can be an NPU, a digital signal processor (DSP), a tensor processing unit (TPU), a field programming gate array (FPGA), etc. The second processing unit 320 is used to perform matrix multiplication on the two sub-matrices after receiving the two sub-matrices to obtain a new sub-matrix. Each time the second processing unit 320 calculates a block of the new sub-matrix, it can generate a message based on the block of the new sub-matrix, and then send it to the third processing unit 340 via the communication bus 330.

[0099] In an embodiment of the present application, the first processing unit 310 can send locally pre-stored segmentation granularity information or user-input segmentation granularity information to each second processing unit 320. After receiving the segmentation granularity information, the second processing unit 320 can divide the matrix of the calculation result into multiple blocks. When the second processing unit 320 performs the matrix multiplication calculation, it sends out the blocks of the matrix of the calculation result in sequence according to the generation order of the blocks in the matrix of the calculation result, which can reduce the time it takes for the second processing unit 320 to send the calculation result each time and avoid the communication bandwidth of the third processing unit 340 being idle for a long time. Since the amount of data in each block is relatively small, the third processing unit 340 is not likely to reach a communication bottleneck, thereby achieving the effect of reducing costs.

[0100] In an embodiment of the present application, the second processing unit 320 can generate a memory reduce write (MRWr) message based on the block of each transmission. The MRWr message can carry a target address, data length, data type and data payload. Among them, the target address is used to indicate the memory address of the data to be written in the third processing unit 340. The data length is used to indicate the length or size of the data to be written. The data length is usually in bytes. The data type is used to indicate the type of data to be written, which can be INT8 (8-bit integer), FP16 (16-bit floating point number), FP32 (32-bit floating point number) or other data types. The data payload is used to indicate the actual data to be written, such as all elements in the block.

[0101] In an embodiment of the present application, the multiple second processing units 320 can obtain a destination address from a preset destination address and a data length from a preset data length according to the order in which the blocks in the new sub-matrix are generated, and configure the destination address and data length for the MRWr message corresponding to the blocks in the new sub-matrix. The multiple second processing units 320 can assign the same destination address and data length to the blocks at the same position in their respective new sub-matrices, so that the third processing unit 340 can perform a matrix addition calculation on the blocks at the same position in the multiple new sub-matrices, thereby enabling the third processing unit 340 to perform a matrix addition calculation function.

[0102] The communication bus 330 may be a compute express link (CXL) bus, a peripheral component interconnect express (PCIe) bus, a universal serial bus (USB), etc. The communication bus 330 is provided with multiple interfaces, which are respectively coupled to interfaces of the first processing unit 310, the multiple second processing units 320, and the third processing unit 340, so as to enable data transmission between the first processing unit 310 and the second processing unit 320, between the second processing units 320, between the second processing unit 320 and the third processing unit 340, and between the first processing unit 310 and the third processing unit 340.

[0103] The third processing unit 340 may be a memory with certain computing capabilities, such as dynamic random access memory (DRAM) or high bandwidth memory (HBM). The third processing unit 340 may be a processor, such as the same processor as the first processing unit 310 or the second processing unit 320. After receiving MRWr messages sent by multiple second processing units 320, the third processing unit 340 parses the target address, data length, data type, and data payload in the MRWr messages.

[0104] After receiving an MRWr message, the third processing unit 340 can determine the target storage space of the local storage space based on the target address and data length of the MRWr message. The third processing unit 340 can read the data stored in the target storage space and perform a matrix addition calculation on the data recorded in the data payload of the MRWr message and the data stored in the target storage space to obtain superimposed matrix blocks. The third processing unit 340 can also clear the data stored in the target storage space and write the superimposed matrix blocks to the target storage space.

[0105] If the third processing unit 340 obtains multiple MRWr messages in a short time, it can read multiple MRWr messages from the local cache, and perform matrix addition calculation on the data recorded in the data payload of the MRWr message with the same target address and data length to obtain the superimposed blocks. The third processing unit 340 can determine the storage space corresponding to the superimposed blocks based on the target address and the data length corresponding to the superimposed blocks. The third processing unit 340 can read the data stored in the storage space corresponding to the superimposed blocks, perform matrix addition calculation on the superimposed blocks and the data stored in the storage space corresponding to the superimposed blocks, and obtain the blocks of the matrix superimposed for the second time. The third processing unit 340 can clear the data stored in the storage space corresponding to the superimposed blocks, and write the blocks of the matrix superimposed for the second time into the storage space corresponding to the superimposed blocks, which can reduce the amount of processing calculations and reduce the processing time.

[0106] In an embodiment of the present application, each time the second processing unit 320 generates an MRWr message, it may also generate a memory OR write (MORWr) message. The MORWr message includes a matrix identifier, which may indicate the sender of the MRWr message. After receiving the MORWr message, the third processing unit 340 may parse the matrix identifier in the MORWr message. After obtaining the matrix identifier, the third processing unit 340 may determine the local storage space configured for the matrix identifier, namely, the identifier storage space. The third processing unit 340 may read the identifier and detect whether the identifier storage space stores the matrix identifier. In one case, if the third processing unit 340 determines that the identifier storage space does not store the matrix identifier, the matrix identifier may be directly stored in the identifier storage space. In another case, if the third processing unit 340 determines that the identifier storage space stores the matrix identifier, the matrix identifier may be read from the identifier storage space, the stored matrix identifier may be ORed with the received matrix identifier, and the ORed matrix identifier may be stored again in the identifier storage space.

[0107] For example, assuming that the matrix identifier can be 2-bit data, the matrix identifier of the MORWr message sent by the first second processing unit 320 when sending the MRWr message is "01", and the matrix identifier of the MORWr message sent by the second second processing unit 320 when sending the MRWr message is "10". When the third processing unit 340 receives the matrix identifier "01" and determines that the identifier storage space does not store the matrix identifier, the third processing unit can perform an OR operation on "01" and "00" to obtain the matrix identifier "01", and store the matrix identifier "01" in the identifier storage space. Then, after receiving the matrix identifier "10", the third processing unit 340 can perform an OR operation on "01" and "10" to obtain the matrix identifier "11", and store the matrix identifier "11" in the identifier storage space.

[0108] When a reading device reads data, it typically reads the sum of the calculation results output by all second processing units 320. However, when the reading device issues a read operation, the third processing unit 340 may only receive some of the calculation results output by the second processing units 320, so the data read by the reading device at this time may be incomplete. Based on this, when the reading device issues a read operation, it can add a matrix identifier to the read operation. This matrix identifier can be the matrix identifier obtained by performing an OR operation on the matrix identifiers corresponding to the calculation results output by all second processing units 320.

[0109] Upon receiving a read operation from the reading device, the third processing unit 340 may detect and compare the matrix identifier in the read operation with the matrix identifier in the identifier storage space. If the third processing unit 340 determines that the matrix identifier in the read operation is the same as the matrix identifier in the identifier storage space, the target storage space may be read and the data in the target storage space may be migrated to the reading device. If the third processing unit 340 determines that the matrix identifier in the read operation is not the same as the matrix identifier in the identifier storage space, the read operation may be abandoned, the comparison may be repeated after a set time, or other operations may be performed.

[0110] In an embodiment of the present application, the first processing unit 310 of the computing system 300 can instruct the plurality of second processing units 310 in advance to divide the matrix of the calculation result into a plurality of blocks. After receiving two larger matrices to be performed on the matrix multiplication, the first processing unit 310 divides the two larger matrices into a plurality of smaller matrices, and distributes them to each second processing unit 320 through the communication bus 330. When each second processing unit 320 performs the matrix multiplication calculation, it sends the blocks of the matrix of the calculation result to the third processing unit 340 in sequence according to the generation order of the blocks in the matrix of the calculation result. Since the calculation time of the second processing unit 320 for processing the blocks is relatively short, the communication bandwidth of the third processing unit 340 can be avoided from being idle for a long time. Since the amount of data in each block is relatively small, the third processing unit 340 is not likely to reach a communication bottleneck, thereby achieving the effect of reducing costs.

[0111] For example, FIG4 is a flow chart of a computing system provided in an embodiment of the present application using a tensor parallel method for training. As shown in FIG4 , the first processing unit 310 of the computing system 300 is a CPU. The second processing unit 320 of the computing system 300 is a GPU, of which there are two, divided into two GPUs marked as GPU 0 and GPU 1. The communication bus 330 of the computing system 300 is a CXL bus. The third processing unit 340 of the computing system 300 is an HBM. Among them, the parameters of each device can be:

[0112] GPU 0 and GPU 1 have 4GB of video memory. HBM memory also has 4GB. The physical address spaces of GPU 0's video memory, GPU 1's video memory, and HBM memory are unified. GPU 0's video memory corresponds to an address space of 0x0-0xFFFFFFFF, GPU 1's video memory corresponds to an address space of 0x100000000-0x1FFFFFFFF, and HBM memory corresponds to an address space of 0x200000000-0x2FFFFFFFF. When computing system 300 performs matrix multiplication X×A=Y, the size of the first matrix X is 1024×2048, the size of the second matrix A is 2048×1024, and the size of the output matrix Y is 1024×1024.

[0113] The specific process of the computing system 300 performing training is as follows:

[0114] After receiving the first matrix X and the second matrix A, the CPU can split the first matrix X into two sub-matrices, sub-matrix X1 and sub-matrix X2, based on the number of GPUs. The size of sub-matrix X1 is 1024×1024, and the size of sub-matrix X2 is 1024×1024. The CPU also splits the second matrix A into two sub-matrices, sub-matrix A1 and sub-matrix A2. The size of sub-matrix A1 is 1024×1024, and the size of sub-matrix A2 is 1024×1024. After splitting the first matrix X and the second matrix A, the CPU can send sub-matrix X1 and sub-matrix A1 to GPU 0, and send sub-matrix X2 and sub-matrix A2 to GPU 1.

[0115] Since the sizes of submatrix X1, submatrix X2, submatrix A1 and submatrix A2 are still relatively large, the CPU can send segmentation granularity information to each GPU, instructing each GPU to split the matrix of the calculation result into multiple blocks. For example, the GPU can split the matrix Y of size 1024×1024 into 128×128 blocks of size 8×8. Taking the block size of matrix Y as 8×8 as an example, GPU 0 splits submatrix X1 into 128×128 blocks, which are blocks X, A, and A. 1,1,1 , block X 1,1,2 ,…, block X 1,1,128 、X 1,2,1 ,…, block X 1,2,128 ,…, block X 1,128,128 The same applies to other GPUs.

[0116] The CPU can send storage space configuration instructions to GPU 0, GPU 1, and HBM. After receiving the configuration instructions, GPU 0 can reserve a specified storage space in the local GPU memory. The physical address of the storage space is 0x0-0xFFFFFFFF, which is used to store sub-matrix X1 and sub-matrix A1. After receiving the configuration instructions, GPU 1 can reserve a specified storage space in the local GPU memory. The physical address of the storage space is 0x100000000-0x1FFFFFFFF, which is used to store sub-matrix X2 and sub-matrix A2. After receiving the configuration instructions, HBM can reserve a specified storage space in the local memory. The physical address of the storage space is 0x200000000-0x2FFFFFFFF, which is used to store the calculation results of GPU 0, or the calculation results of GPU 1, or the result of matrix addition calculation between the calculation results of GPU 0 and GPU 1.

[0117] GPU 0 blocks X of submatrix X1 1,1,1 With the block A of submatrix A1 1,1,1Take the matrix multiplication calculation process as an example. GPU 0 generally includes an interface, a GPU core, a memory controller, and a GPU memory. The interface can receive sub-matrix X1 and sub-matrix Y1 via the CXL bus, and send MRWr messages. The GPU memory is used to store sub-matrix X1 and sub-matrix Y1. After receiving the GPU core read instruction, the memory controller can sequentially read the blocks of sub-matrix X1 and the blocks of sub-matrix Y1 stored in the GPU memory. The GPU core is used to perform matrix multiplication calculations on the blocks of sub-matrix X1 and the blocks of sub-matrix Y1, and generate MRWr messages.

[0118] GPU core blocks X of submatrix X1 1,1,1 With the block A of submatrix A1 1,1,1 Perform matrix multiplication to obtain matrix Y 1,1,1 The GPU core can use an instruction or function to calculate the matrix Y 1,1,1 Generate an MRWr message and write HBM in the MRWr message. The target address is 0x200000000, the data length is 128B, the data type is FP16, and the data payload is the matrix Y 1,1,1 Each element of the MRWr message. Each time the GPU core generates a MRWr message, it can be sent to the HBM immediately. After the GPU core sends the MRWr message, it sends a read instruction to the memory controller to obtain the next block of the sub-matrix X1 and the next block of the sub-matrix Y1, and performs matrix multiplication calculation on the next block of the sub-matrix X1 and the next block of the sub-matrix Y1. And so on, it processes until the matrix Y 1,128,128 Corresponding MRWr message.

[0119] At the same time, GPU 1 also executes the operations of GPU 0.

[0120] HBM generally includes an interface, a memory controller, and a memory. The interface is used to receive MRWr messages. After receiving the MRWr message, the memory controller can parse the MRWr message and obtain the target address as 0x200000000, the data length as 128B, the data type as FP16, and the data payload as the matrix Y. 1,1,1 elements of the .

[0121] The memory controller determines the target storage space of the memory based on the target address of 0x200000000 and the data length of 128B. The memory controller determines that the target storage space does not store data and can store the matrix Y 1,1,1 Each element of Y is directly written into the storage space 0x200000000-0x2001FFFFF of the memory, or the matrix Y 1,1,1Each element of Y is added to 0 and the result is written to the memory space 0x200000000-0x2001FFFFF. The memory controller determines that the target memory space has data and can read the matrix Y from the memory. 2,1,1 Then the matrix Y 1,1,1 Each element of the matrix Y 2,1,1 Perform matrix addition calculation on each element of to obtain the matrix Y 1,1 The memory controller can then erase the elements of the matrix stored in the target memory space and store the matrix Y 1,1 Each element is stored in the memory space of 0x200000000-0x2001FFFFF.

[0122] After the memory controller completes the matrix addition calculation for all blocks of matrix Y1 and matrix Y2, the data stored in the memory space 0x200000000-0x2FFFFFFFF is the final matrix Y. After completing this training, the memory controller can distribute the final matrix Y to GPU 0 and GPU 1 for the next training.

[0123] FIG5 is a schematic diagram of the structure of another computing system provided in an embodiment of the present application. As shown in FIG5 , computing system 500 includes a first processing unit 510 and multiple second processing units 520. Multiple second processing units 520 are respectively second processing unit 520-1, second processing unit 520-2, second processing unit 520-3, ..., second processing unit 520-N-1, and second processing unit 520-N. N is a positive integer greater than 1.

[0124] The type of first processing unit 510 is the same as first processing unit 310 in Figure 3 , and reference can be made to the description of Figure 3 for details. In the embodiment of the present application, first processing unit 510 can establish communication connections with multiple second processing units 520 and can distribute multiple small matrices to each second processing unit 520. First processing unit 510 is used to distribute programs to each second processing unit 520, enabling each second processing unit 520 to perform matrix multiplication calculations and other functions.

[0125] The type of the second processing unit 520 is the same as the second processing unit 320 in Figure 3. The function of the second processing unit 520 is the same as that of the second processing unit 320 and the third processing unit 340 in Figure 3. For details, please refer to the description of Figure 3. In the embodiment of the present application, the sending end (send) of each second processing unit 520 establishes a communication connection with the receiving end (receive) of another second processing unit 520, and the local calculation results can be sent to another second processing unit 520. Specifically: the sending end of the second processing unit 520-1 establishes a communication connection with the receiving end of the second processing unit 520-2, the sending end of the second processing unit 520-2 establishes a communication connection with the receiving end of the second processing unit 520-3, ..., the sending end of the second processing unit 520-N-1 establishes a communication connection with the receiving end of the second processing unit 520-N, and the sending end of the second processing unit 520-N establishes a communication connection with the receiving end of the second processing unit 520-1, so that a ring is formed between the sending ends and receiving ends of multiple second processing units 520.

[0126] The process of the computing system 500 performing data training using the ring-all-reduce method can be divided into two stages: a scatter-reduce stage and an allgather stage.

[0127] In the scatter-reduce phase, the first processing unit 510 divides the matrix into multiple sub-matrices and then distributes them to different second processing units 520. After receiving the sub-matrix for which it is responsible, each second processing unit 520 performs local calculations and reduction operations to obtain a partial result.

[0128] During the full collection phase, each second processing unit 520 sends its partial results to other second processing units 520 and receives partial results from other second processing units 520. This allows all second processing units 520 to obtain the partial results of other second processing units 520. Through multiple iterations, each second processing unit 520 will eventually aggregate the partial results of all other second processing units 520 to obtain a complete final result. This process is equivalent to gathering the partial results from each second processing unit 520, ensuring that all second processing units 520 have the complete final result.

[0129] Figure 6 is a structural diagram of the computing system provided in an embodiment of the present application using a ring full reduction method for training. As shown in Figure 6, the first processing unit 510 of the computing system 500 is a CPU. The second processing unit 520 of the computing system 500 is a GPU, which is 5 in number and is divided into GPUs marked as GPU 0, GPU 1, GPU 2, GPU 3 and GPU 4. Among them, the transmitting end of GPU 0 establishes a communication connection with the receiving end of GPU 1. The transmitting end of GPU 1 establishes a communication connection with the receiving end of GPU 2. The transmitting end of GPU 2 establishes a communication connection with the receiving end of GPU 3. The transmitting end of GPU 3 establishes a communication connection with the receiving end of GPU 4. The transmitting end of GPU 4 establishes a communication connection with the receiving end of GPU 0. Each GPU only needs to send data to one GPU and receive data sent by one GPU, so that the communication of multiple GPUs participating in the training is constant and the cost is the same. The parameters of each device can be:

[0130] The video memory of GPU 0, GPU 1, GPU 2, GPU 3, and GPU 4 is 4GB. The physical address space of the video memory of GPU 0, GPU 1, GPU 2, GPU 3, and GPU 4 is unified. The address space corresponding to the video memory of GPU 0 is 0x0-0xFFFFFFFF, the address space corresponding to the video memory of GPU 1 is 0x100000000-0x1FFFFFFFF, the address space corresponding to the video memory of GPU 2 is 0x200000000-0x2FFFFFFFF, the address space corresponding to the video memory of GPU 3 is 0x300000000-0x3FFFFFFFF, and the address space corresponding to the video memory of GPU 4 is 0x4000000000-0x4FFFFFFFF. Computing system 500 processes five matrices: matrix a, matrix b, matrix c, matrix d, and matrix e. The size of the five matrices is 1024×1024, and the elements of each matrix are 2-byte half-precision floating point numbers (FP16).

[0131] The specific process of the computing system 500 performing training is as follows:

[0132] After receiving matrices a, b, c, d, and e, the CPU can split matrix a into five sub-matrices based on the number of GPUs: sub-matrix a0, sub-matrix a1, sub-matrix a2, sub-matrix a3, and sub-matrix a4. The size of each of the five sub-matrices is 1024 × 1024. The CPU performs the same division as matrix a on matrices b, c, d, and e.

[0133] It should be noted that since this embodiment involves five matrices for matrix multiplication, to reduce the complexity of the scheme description, the two matrices for matrix multiplication are collectively referred to as one matrix. For example, matrix a refers to the two matrices for which matrix multiplication is to be performed to obtain matrix a.

[0134] As shown in Figure 7(a), the CPU can send sub-matrix a0, sub-matrix b0, sub-matrix c0, sub-matrix d0 and sub-matrix e0 to GPU 0, send sub-matrix a1, sub-matrix b1, sub-matrix c1, sub-matrix d1 and sub-matrix e1 to GPU 1, send sub-matrix a2, sub-matrix b2, sub-matrix c2, sub-matrix d2 and sub-matrix e2 to GPU 2, send sub-matrix a3, sub-matrix b3, sub-matrix c3, sub-matrix d3 and sub-matrix e3 to GPU 3, and send sub-matrix a4, sub-matrix b4, sub-matrix c4, sub-matrix d4 and sub-matrix e4 to GPU 4.

[0135] Since the sizes of submatrix a0, submatrix b0, submatrix c0, submatrix d0, submatrix e0, ..., submatrix e4 are still relatively large,

[0136] The CPU can send segmentation granularity information to each GPU, instructing each GPU to split the matrix of the calculation result into multiple blocks. For example, GPU 0 can split the 1024×1024 matrix a0 into 128×128 8×8 blocks.

[0137] Taking the block size of matrix a0 as 8×8 as an example, GPU 0 divides the sub-matrix a0 into 128×128 blocks according to the segmentation granularity information, which are block a 0,1,1 , block a 0,1,2 ,…, block a 0,1,128 , block a 0,2,1 ,…, block a 0,2,128 ,…, block a 0,128,128 Similarly, GPU 0 splits submatrix b0 into 128×128 blocks, GPU 0 splits submatrix c0 into 128×128 blocks, GPU 0 splits submatrix d0 into 128×128 blocks, and GPU 0 splits submatrix e0 into 128×128 blocks.

[0138] Similarly, GPU 1, GPU 2, GPU 3, GPU 4, and GPU 0 divide the five local sub-matrices into 128×128 blocks according to the segmentation granularity information.

[0139] The CPU can send storage space configuration instructions to GPU 0, GPU 1, GPU 2, GPU 3, and GPU 4. After receiving the configuration instructions, GPU 0, GPU 1, GPU 2, GPU 3, and GPU 4 can reserve the specified storage space in their local GPU memory and flush the specified storage space to zero. For example, the physical address of the storage space reserved by GPU 0 for submatrix a0 is 0x0-0x1FFFFF. The physical address of the storage space reserved by GPU 1 for submatrix a1 is 0x100000000-0x1001FFFFF. The physical address of the storage space reserved by GPU 2 for submatrix a2 is 0x200000000-0x2001FFFFF. The physical address of the storage space reserved by GPU 3 for submatrix a3 is 0x300000000-0x3001FFFFF. The physical address of the storage space reserved by GPU 4 for submatrix a4 is 0x400000000-0x4001FFFFF. The physical addresses of the storage spaces reserved by GPU 0, GPU 1, GPU 2, GPU 3, and GPU 4 for other submatrices are similar, and so on.

[0140] As shown in Figure 7(b), GPU 0 is used to map the submatrix a0 to block a. 0,1,1 Take the first iteration as an example.

[0141] GPU 0 generally includes an interface, a GPU core, a memory controller, and GPU memory. The interface can receive submatrix a0, submatrix b0, submatrix c0, submatrix d0, and submatrix e0 via the CXL bus, and send MRWr messages. The GPU memory is used to store submatrix a0, submatrix b0, submatrix c0, submatrix d0, and submatrix e0. After receiving a read instruction from the GPU core, the memory controller can sequentially read the blocks of submatrix a0, submatrix b0, submatrix c0, submatrix d0, and submatrix e0 stored in the GPU memory. The GPU core is used to perform ring full reduction calculations on each block of each submatrix.

[0142] GPU core splits block a 0,1,1 After completing the ring full reduction calculation, you can use an instruction or function to calculate the ring full reduction according to the block a. 0,1,1 Generate a memory write (MWr) message and send the MWr message to the memory controller, instructing the memory controller to write block a 0,1,1 Write to GPU memory. The GPU core can then use an instruction or function to write 0,1,1Generate an MRWr message and write it to GPU 1. The target address is 0x100000000, the data length is 128B, the data type is FP16, and the data payload is block a. 0,1,1 Each time the GPU core generates a MRWr message, it can be sent to GPU 1 immediately. After the GPU core sends the MRWr message, it sends a read instruction to the memory controller to obtain the next block a 0,1,2 , and the next block a 0,1,2 Process it. And so on, until block a is processed. 0,128,128 Corresponding MRWr message.

[0143] At the same time, GPU 1 performs the first iteration on the blocks of submatrix b0. GPU 2 performs the first iteration on the blocks of submatrix c0. GPU 3 performs the first iteration on the blocks of submatrix d0. GPU 4 performs the first iteration on the blocks of submatrix e0. For details, refer to the process of GPU 0 performing the first iteration on the blocks of submatrix a0.

[0144] Subsequently, the GPU core of GPU 0 sequentially performs ring full reduction calculations on each block of submatrix b0, each block of submatrix c0, each block of submatrix d0, and each block of submatrix e0.

[0145] At the same time, the GPU core of GPU 1 sequentially performs a circular full reduction calculation on each block of submatrix b1, each block of submatrix c1, each block of submatrix d1, and each block of submatrix e1. The GPU core of GPU 2 sequentially performs a circular full reduction calculation on each block of submatrix b2, each block of submatrix c2, each block of submatrix d2, and each block of submatrix e2. The GPU core of GPU 3 sequentially performs a circular full reduction calculation on each block of submatrix b3, each block of submatrix c3, each block of submatrix d3, and each block of submatrix e3. The GPU core of GPU 4 sequentially performs a circular full reduction calculation on each block of submatrix b4, each block of submatrix c4, each block of submatrix d4, and each block of submatrix e4.

[0146] After the GPU core of GPU 1 receives the MRWr message, it can parse the MRWr message and obtain the target address as 0x100000000, the data length as 128B, the data type as FP16, and the data payload as block a. 0,1,1 The memory controller of GPU 1 determines the target storage space of GPU 1 based on the target address 0x100000000 and the data length 128B. The memory controller of GPU 1 reads block a from the target storage space.1,1,1 Each element of the block a 0,1,1 Each element of the block a 1,1,1 Perform matrix addition calculation on each element of to obtain block a 0,1,1 +Block a 1,1,1 Finally, the memory controller can erase the data stored in the target storage space and block a 0,1,1 +Block a 1,1,1 Each element of the memory is written to the memory space 0x100000000-0x1001FFFFF of GPU 1.

[0147] Similarly, the GPU core of GPU 1 continuously receives MRWr messages to complete the subsequent blocks a 0,1,2 ,…,a 0,128,128 Respectively with block a 1,1,2 ,…, block a 1,128,128 Perform matrix addition calculation to obtain block a 0,1,2 +Block a 1,1,2 ,…, block a 0,128,128 +Block a 1, 128,128 At this point, the GPU core of GPU 1 completes the first superposition of matrix a, obtaining submatrix a0 + submatrix a1. The data stored in the storage space 0x100000000-0x1FFFFFFFF of the GPU memory of GPU 1 is the final submatrix a0 + submatrix a1.

[0148] Meanwhile, GPU 2 performs its first iteration on matrix b0, obtaining submatrix b2 + submatrix b1. GPU 3 performs its first iteration on matrix c0, obtaining submatrix c3 + submatrix c2. GPU 4 performs its first iteration on matrix d0, obtaining submatrix d4 + submatrix d3. GPU 0 performs its first iteration on matrix e0, obtaining submatrix e0 + submatrix e4. For details, refer to the process of GPU 1 performing its first iteration on matrix a.

[0149] As shown in FIG7( c ), GPU 0 performs the second iteration on submatrix e0 + submatrix e4 as an example.

[0150] The GPU core of GPU 0 sends a read command to the memory controller to obtain block e 0,1,1 +Block 4,1,1 The GPU core of GPU 0 can use an instruction or function to 0,1,1 +Block 4,1,1Generate an MRWr message and write it to GPU 1. The target address is 0x100800000, the data length is 128B, the data type is FP16, and the data payload is block e 0,1,1 +Block 4,1,1 The superimposed elements.

[0151] Each time the GPU core of GPU 0 generates a MRWr message, it can be sent to GPU 1 immediately. After the GPU core of GPU 0 sends the MRWr message, it sends a read instruction to the memory controller to obtain the next block e 0,1,2 +Block 4,1,2 , and the next block e 0,1,2 +Block 4,1,2 Process it. And so on, until the block e 0,128,128 +Block 4,128,128 Corresponding MRWr message.

[0152] At the same time, GPU 1 performs a second iteration on the blocks of submatrix a1 and submatrix a0. GPU 2 performs a second iteration on the blocks of submatrix b2 and submatrix b1. GPU 3 performs a second iteration on the blocks of submatrix c3 and submatrix c2. GPU 4 performs a second iteration on the blocks of submatrix d4 and submatrix d3.

[0153] After the GPU core of GPU 1 receives the MRWr message, it can parse the MRWr message and obtain the target address as 0x100800000, the data length as 128B, the data type as FP16, and the data payload as block e 0,1,1 +Block 4,1,1 The memory controller of GPU 1 determines the target storage space of GPU 1 based on the target address 0x100800000 and the data length 128B. The memory controller of GPU 1 reads the block e from the target storage space. 1,1,1 Each element of the 1,1,1 Each element of the block e 0,1,1 +Block 4,1,1 The superimposed elements are subjected to matrix addition calculation to obtain the block e 0,1,1 +Block 4,1,1 +Block 1,1,1 The memory controller can erase the data stored in the target storage space and block e 0,1,1 +Block 4,1,1 +Block 1,1,1 The superimposed elements are stored and written into the storage space 0x100800000-0x1008FFFFF of the memory of GPU 1.

[0154] Similarly, the GPU core of GPU 1 continuously receives MRWr messages to complete the subsequent blocks e 0,1,2 +Block 4,1,2 ,…, block e 0,128,128 +Block 4,128,128 Separately with block e 1,1,2 ,…, block e 1,128,128 Perform matrix addition calculation to obtain block e 0,1,2 +Block 4,1,2 +Block 1,1,2 ,…, block e 0,128,128 +Block 4,128,128 +Block 1,128,128 At this point, the GPU core of GPU 1 completes the second superposition of matrix e, resulting in submatrix e0 + submatrix e4 + submatrix e1. The data stored in the memory space 0x100000000-0x1FFFFFFFF of GPU 1's GPU memory is the final submatrix e0 + submatrix e4 + submatrix e1.

[0155] At the same time, GPU 2 stores submatrix a1 + submatrix a0, resulting in submatrix a1 + submatrix a0 + submatrix a2. GPU 3 stores submatrix b2 + submatrix b1, resulting in submatrix b2 + submatrix b1 + submatrix b3. GPU 4 stores submatrix c3 + submatrix c2, resulting in submatrix c3 + submatrix c2 + submatrix c4. GPU 0 stores submatrix d4 + submatrix d3, resulting in submatrix d4 + submatrix d3 + submatrix d0. For details, refer to the process of GPU 1 storing matrix e.

[0156] Similarly, GPU 0, GPU 1, GPU 2, GPU 3, and GPU 4 perform the third and fourth iterations. The iterative process can refer to the process of the second iteration. Finally, in the scatter-reduce phase, as shown in Figure 7(d), GPU 0 stores the matrix a (also known as a0), the matrix b (also known as b0+b1+b2+b3+b4), the matrix c (also known as c0+c2+c3+c4), the matrix d (also known as d0+d3+d4), and the matrix e (also known as e0+e4). GPU 1 stores the matrix a (also known as a0+a1), the matrix b (also known as b1), the matrix c (also known as c0+c1+c2+c3+c4), the matrix d (also known as d0+d1+d3+d4), and the matrix e (also known as e0+e1+e4). GPU 2 stores the matrix a (i.e., a0+a1+a2), matrix b (i.e., b1+b2), matrix c (i.e., c2), matrix d (i.e., d0+d1+d2+d3+d4), and matrix e (i.e., e0+e1+e2+e4). GPU 3 stores the matrix a (i.e., a0+a1+a2+a3), matrix b (i.e., b1+b2+b3), matrix c (i.e., c2+c3), matrix d (i.e., d3+d4), and matrix e (i.e., e0+e1+e2+e3+e4). GPU 4 stores the matrix a (i.e., a0+a1+a2+a3+a4), matrix b (i.e., b1+b2+b3+b4), matrix c (i.e., c2+c3+c4), matrix d (i.e., d3+d4), and matrix e (i.e., e4).

[0157] During the first stacking process of all GPUs in the collection phase, as shown in Figure 7(e), the GPU core of GPU 0 sends a read instruction to the memory controller to obtain block b 0,1,1 +Block b 1,1,1 +Block b 2,1,1 +Block b 3,1,1 +Block b 4,1,1 The GPU core of GPU 0 can use an instruction or function to 0,1,1 +Block b 1,1,1 +Block b 2,1,1 +Block b 3,1,1 +Block b 4,1,1 Generate an MRWr message and write it to GPU 1. The target address is 0x100800000, the data length is 128B, the data type is FP16, and the data payload is block b. 0,1,1 +Block b 1,1,1 +Block b 2,1,1 +Block b 3,1,1 +Block b 4,1,1 The superimposed elements.

[0158] Each time the GPU core of GPU 0 generates a MRWr message, it can be sent to GPU 1 immediately. After the GPU core of GPU 0 sends the MRWr message, it sends a read instruction to the memory controller to obtain the next block b 0,1,2 +Block b 1,1,2 +Block b 2,1,2 +Block b 3,1,2 +Block b 4,1,2 , and the next block b 0,1,2 +Block b 1,1,2 +Block b 2,1,2 +Block b 3,1,2 +Block b 4,1,2 Process it. And so on, until block b 0,128,128 +Block b 1,128,128 +Block b 2,128,128 +Block b 3,128,128 +Block b 4,128,128 Corresponding MRWr message.

[0159] At the same time, GPU 1 performs the first iteration on the block consisting of submatrix c0, submatrix c1, submatrix c2, submatrix c3, and submatrix c4. GPU 2 performs the first iteration on the block consisting of submatrix d0, submatrix d1, submatrix d2, submatrix d3, and submatrix d4. GPU 3 performs the first iteration on the block consisting of submatrix e0, submatrix e1, submatrix e2, submatrix e3, and submatrix e4. GPU 4 performs the first iteration on the block consisting of submatrix a0, submatrix a1, submatrix a2, submatrix a3, and submatrix a4.

[0160] After receiving the MRWr message, the GPU core of GPU 1 can parse the MRWr message and obtain the target address as 0x100800000, the data length as 128B, the data type as FP16, and the data payload as block b. 0,1,1 +Block b 1,1,1 +Block b 2,1,1 +Block b 3,1,1 +Block b 4,1,1 The memory controller of GPU 1 determines the target storage space of GPU 1 based on the target address 0x100800000 and the data length 128B. The memory controller can erase the block b stored in the target storage space. 1,1,1 Each element of will be divided into blocks b 0,1,1 +Block b 1,1,1 +Block b 2,1,1 +Block b 3,1,1 +Block b 4,1,1The storage of each superimposed element is written into the storage space 0x100800000-0x1008FFFFF of the memory of GPU 1.

[0161] Similarly, the GPU core of GPU 1 continuously receives MRWr messages to complete the subsequent blocks b 0,1,2 +Block b 1,1,2 +Block b 2,1,2 +Block b 3,1,2 +Block b 4,1,2 ,…, block b 0,128,128 +Block b 1,128,128 +Block b 2,128,128 +Block b 3,128,128 +Block b 4,128,128 At this point, the GPU core of GPU 1 completes the storage of matrix b. The data stored in the storage space 0x100000000-0x1FFFFFFFF of the GPU memory of GPU 1 is the final matrix b (i.e., b0+b1+b2+b3+b4).

[0162] At the same time, GPU 2 stores the blocks of submatrix c0, submatrix c1, submatrix c2, submatrix c3, and submatrix c4 to obtain matrix c. GPU 3 stores the blocks of submatrix d0, submatrix d1, submatrix d2, submatrix d3, and submatrix d4 to obtain matrix d. GPU 4 stores the blocks of submatrix e0, submatrix e1, submatrix e2, submatrix e3, and submatrix e4 to obtain matrix e. GPU 0 stores the blocks of submatrix a0, submatrix a1, submatrix a2, submatrix a3, and submatrix a4 to obtain matrix a.

[0163] Similarly, GPU 0, GPU 1, GPU 2, GPU 3, and GPU 4 perform the second, third, and fourth iterations. The iterative process can refer to the second iteration process. Finally, in the full collection phase, as shown in Figure 7(f), GPU 0, GPU 1, GPU 2, GPU 3, and GPU 4 all store matrix a (i.e., a0+a1+a2+a3+a4), matrix b (i.e., b0+b1+b2+b3+b4), matrix c (i.e., c0+c1+c2+c3+c4), matrix d (i.e., d0+d1+d2+d3+d4), and matrix e (i.e., e0+e1+e2+e3+e4).

[0164] In an embodiment of the present application, the first processing unit 510 of the computing system 500 can instruct multiple second processing units 520 in advance to divide the matrix of the calculation result into multiple blocks. The first processing unit 510 divides the matrix that needs to be subjected to matrix multiplication calculation into multiple sub-matrices, and then assigns them to different second processing units 520. Each second processing unit 520 can perform matrix multiplication calculation on the sub-matrix at the same time, and send the blocks of the matrix of the calculation result to the next second and third processing units 520 in sequence according to the generation order of the blocks in the matrix of the calculation result, and so on, until all the second processing units 520 have stored the matrix for matrix multiplication calculation. Since the calculation time of the second processing unit 520 to process the blocks is relatively small, the communication bandwidth of each second processing unit 520 can be avoided from being idle for a long time. Since the amount of data in each block is relatively small, each second processing unit 520 is not likely to reach a communication bottleneck, thereby achieving the effect of reducing costs.

[0165] The following is a flowchart to introduce the implementation process of the technical solution protected by this application.

[0166] FIG8 is a flow chart of a model training method provided in an embodiment of the present application. As shown in FIG8 , the method can be executed by the computing system 300 shown in FIG3 and the computing system 500 shown in FIG5 , and the specific implementation process is as follows:

[0167] Step S801: The second processing unit receives a first sub-matrix and a second sub-matrix.

[0168] The first sub-matrix and the second sub-matrix are allocated by the upper-level control processor of the second processing unit, as well as the first processing unit 310 in the computing system 300 and the first processing unit 510 in the computing system 500. After obtaining the first matrix and the second matrix to be performed on the matrix multiplication calculation, the first processing unit divides the first matrix into a corresponding number of first sub-matrices and the second matrix into a corresponding number of second sub-matrices according to the number of second processing units. Then, the first processing unit can send the multiple first sub-matrices and the multiple second sub-matrices to the multiple second processing units respectively, so that each second processing unit can receive one first sub-matrix and one second sub-matrix.

[0169] In an embodiment of the present application, the first processing unit may send segmentation granularity information to each second processing unit. After receiving the segmentation granularity information, each second processing unit may divide the matrix of the calculation result into a plurality of blocks. The size of each block is smaller than the size of the matrix of the calculation result.

[0170] In step S802 , the second processing unit performs matrix multiplication on the first sub-matrix and the second sub-matrix.

[0171] Step S803: The second processing unit sends the blocks of the third sub-matrix in sequence according to the generation order of the blocks in the third sub-matrix.

[0172] When the second processing unit performs a matrix multiplication calculation on the first submatrix and the second submatrix, the blocks of the matrix of the calculation result can be sent out sequentially according to the order in which the blocks in the matrix of the calculation result were generated. This can reduce the time it takes for the second processing unit to send the calculation result each time, and prevent the communication bandwidth of the processing unit or memory receiving the results of multiple second processing units from being idle for a long time. Because the amount of data in each block is relatively small, the processing unit or memory receiving the results of multiple second processing units is less likely to reach a communication bottleneck, thereby achieving the effect of reducing costs.

[0173] In an embodiment of the present application, the second processing unit can generate a first message, that is, the above-mentioned MRWr message, based on the blocks transmitted each time. The MRWr message can carry a target address, a data length, a data type, and a data payload. Among them, the target address is used to indicate the memory address of the block to be written in the processing unit or memory that receives the results of multiple second processing units. The data length is used to indicate the length or size of the block to be written. The data length is usually in bytes. The data type is used to indicate the type of data of the block to be written. The data payload is used to indicate all elements in the actual block to be written.

[0174] In one embodiment, the multiple second processing units can obtain the destination address from the preset destination address and the data length from the preset data length according to the generation order of the blocks in the new sub-matrix, and configure the destination address and data length for the MRWr message corresponding to the blocks in the new sub-matrix. The multiple second processing units can assign the same destination address and data length to the blocks at the same position in their respective new sub-matrices, so that the processing unit or memory receiving the results of the multiple second processing units can perform a matrix addition calculation on the blocks at the same position in the multiple new sub-matrices, thereby allowing the processing unit or memory receiving the results of the multiple second processing units to perform the matrix addition calculation function.

[0175] In computing system 300, the third processing unit 340 receives the results of multiple second processing units. After receiving an MRWr message, the third processing unit 340 can determine the target storage space of the local storage space based on the target address and data length of the MRWr message. The third processing unit 340 can read the data stored in the target storage space and perform a matrix addition calculation on the data recorded in the data payload of the MRWr message and the data stored in the target storage space to obtain the superimposed matrix blocks. The third processing unit 340 can also clear the data stored in the target storage space and write the superimposed matrix blocks to the target storage space.

[0176] After the third processing unit 340 obtains multiple MRWr messages in a short time, it can read multiple MRWr messages from the local cache, and perform matrix addition calculation on the data recorded in the data payload of the MRWr message with the same target address and data length to obtain the superimposed blocks. The third processing unit 340 can determine the storage space corresponding to the superimposed blocks based on the target address and data length corresponding to the superimposed blocks. The third processing unit 340 can read the data stored in the storage space corresponding to the superimposed blocks, perform matrix addition calculation on the superimposed blocks and the data stored in the storage space corresponding to the superimposed blocks, and obtain the blocks of the matrix superimposed for the second time. The third processing unit 340 can clear the data stored in the storage space corresponding to the superimposed blocks, and write the blocks of the matrix superimposed for the second time into the storage space corresponding to the superimposed blocks, which can reduce the amount of processing calculations and reduce the processing time.

[0177] In computing system 500, receiving multiple second processing units results in a connected second processing unit 520. After receiving the MRWr message, third processing unit 520 can determine the target storage space of the local storage space based on the target address and data length of the MRWr message. Second processing unit 520 can read the data stored in the target storage space and perform a matrix addition calculation on the data recorded in the data payload of the MRWr message and the data stored in the target storage space to obtain a superimposed matrix block. Second processing unit 520 can generate an MRWr message based on the block of each transmission and send the MRWr message to the next second processing unit 520.

[0178] At the same time, the second processing unit 520 generates a second message, namely the aforementioned MWr message, based on the superimposed matrix blocks. The MWr message also carries a target address, data length, data type, and data payload. The second processing unit 520 can send the MWr message to a local memory controller. After receiving the MWr message, the local memory controller parses the superimposed matrix blocks. The local memory controller can determine the storage space corresponding to the superimposed matrix blocks based on the target address and data length corresponding to the superimposed matrix blocks. The local memory controller can read the data stored in the storage space corresponding to the superimposed matrix blocks and perform a matrix addition calculation on the superimposed matrix blocks and the data stored in the storage space corresponding to the superimposed matrix blocks to obtain the second superimposed matrix blocks. The local memory controller can clear the data stored in the storage space corresponding to the superimposed matrix blocks and write the second superimposed matrix blocks into the storage space corresponding to the superimposed matrix blocks, thereby reducing the amount of processing computation and processing time.

[0179] In an embodiment of the present application, each time the second processing unit generates an MRWr message, it may also generate a third message, namely, the aforementioned MORWr message. The MORWr message includes a matrix identifier, which may indicate the sender of the MRWr message. After receiving the MORWr message, the third processing unit 340 or the second processing unit 520 may parse the matrix identifier in the MORWr message. After obtaining the matrix identifier, the third processing unit 340 or the second processing unit 520 may determine the storage space locally configured for the matrix identifier, namely, the identifier storage space. The third processing unit 340 or the second processing unit 520 may read the identifier to detect whether the identifier storage space stores the matrix identifier. In one scenario, if the third processing unit 340 or the second processing unit 520 determines that the identifier storage space does not store the matrix identifier, the matrix identifier may be directly stored in the identifier storage space. In another case, the third processing unit 340 or the second processing unit 520 determines that the identification storage space stores a matrix identification, and can read the matrix identification in the identification storage space, perform an OR operation on the stored matrix identification and the received matrix identification, and then store the OR-operated matrix identification in the identification storage space again.

[0180] Upon receiving a read operation from the reading device, the third processing unit 340 or the second processing unit 520 may detect and compare the matrix identifier in the read operation with the matrix identifier in the identifier storage space. If the third processing unit 340 or the second processing unit 520 determines that the matrix identifier in the read operation is the same as the matrix identifier in the identifier storage space, the target storage space may be read and the data in the target storage space may be migrated to the reading device. If the third processing unit 340 or the second processing unit 520 determines that the matrix identifier in the read operation is not the same as the matrix identifier in the identifier storage space, the read operation may be abandoned, the comparison may be repeated after a set time, or other operations may be performed.

[0181] In an embodiment of the present application, the first processing unit can instruct multiple local second processing units to divide the matrix of the calculation results into multiple blocks. When each second processing unit performs a matrix multiplication calculation, it sends out the blocks of the matrix of the calculation results in sequence according to the generation order of the blocks in the matrix of the calculation results. This can reduce the time it takes for the second processing unit to send the calculation results each time, and avoid the communication bandwidth of the processing unit or memory that receives the results of multiple second processing units being idle for a long time. Since the amount of data in each block is relatively small, the processing unit or memory that receives the results of multiple second processing units is not likely to reach a communication bottleneck, thereby achieving the effect of reducing costs.

[0182] Figure 9 is a schematic diagram of the structure of a model training device provided in an embodiment of the present application. As shown in Figure 9, the model training device 900 can execute the process performed by the third processing unit 340 of the computing system 300 in the embodiment shown in Figure 3, or the process performed by the second processing unit 520 of the computing system 500 in the embodiment shown in Figure 5. For details, please refer to the relevant description of the above method embodiments. The model training device 900 can be divided into a transceiver unit 910 and a processing unit 920 based on the execution function.

[0183] The transceiver unit 910 can implement corresponding communication functions and can also be called a communication interface or a communication module.

[0184] The processing unit 920 is used for data processing.

[0185] Optionally, the model training device 900 may further include a third processing unit, which may be used to store instructions and / or data, and the processing unit 920 may read the instructions and / or data in the third processing unit so that the model training device 900 implements the aforementioned method embodiment.

[0186] The model training device 900 can be used to execute the actions performed by the third processing unit 340 or the second processing unit 520 in the above method embodiments. The model training device 900 can be the third processing unit 340 of the computing system 300 or the second processing unit 520 of the computing system 500, or a component that can be configured in the third processing unit 340 or the second processing unit 520. The processing unit 920 is used to execute the processing-related operations on the side of the third processing unit 340 or the second processing unit 520 in the above method embodiments. The transceiver unit 910 is used to execute the reception-related operations on the side of the third processing unit 340 or the second processing unit 520 in the above method embodiments.

[0187] Optionally, the transceiver unit 910 may include a sending unit and a receiving unit. The sending unit is configured to perform the sending operation in the above method embodiment. The receiving unit is configured to perform the receiving operation in the above method embodiment.

[0188] It should be noted that the model training device 900 may include a sending unit but not a receiving unit. Alternatively, the model training device 900 may include a receiving unit but not a sending unit. Specifically, it can be determined whether the above solution executed by the model training device 900 includes both sending and receiving actions.

[0189] In one embodiment, the model training apparatus 900 may perform the functions of the third processing unit 340 of the computing system 300 described in FIG. 3 , specifically:

[0190] The transceiver unit 910 is configured to receive a first submatrix and a second submatrix. The first submatrix is ​​obtained by the first processing unit by segmenting the first matrix to be subjected to matrix multiplication. The second submatrix is ​​obtained by the first processing unit by segmenting the second matrix to be subjected to matrix multiplication. The processing unit 920 is configured to perform matrix multiplication on the first submatrix and the second submatrix. The blocks of the third submatrix are sequentially transmitted in the order in which the blocks in the third submatrix were generated. The third submatrix is ​​the submatrix obtained by matrix multiplication of the first submatrix and the second submatrix.

[0191] In one embodiment, the processing unit 920 is further configured to generate a first message based on the blocks of the third submatrix and send the first message. The first message includes a target address, a data length, a data type, and a data payload. The target address refers to the location of the target storage space where the third processing unit stores the blocks of the third submatrix. The data length refers to the size of the blocks of the third submatrix. The data type refers to the data type of the blocks of the third submatrix. The data payload is used to record the blocks of the third submatrix. The third processing unit is a second processing unit among the plurality of second processing units or a memory.

[0192] In one embodiment, the transceiver unit 910 is further configured to, upon receiving a first message sent by another second processing unit, parse the first message to obtain blocks of the third submatrix. The transceiver unit 910 is further configured to perform matrix addition calculations on the blocks of the multiple third submatrices to obtain blocks of a third matrix. The third matrix is ​​a matrix obtained by performing matrix addition calculations on the third submatrices generated by the multiple second processing units.

[0193] In one embodiment, the processing unit 920 is specifically configured to determine a target storage space based on the target address and data length in the first message. The processing unit 920 is specifically configured to read data stored in the target storage space, perform matrix addition calculations on the data recorded in the data payload of the first message and the data stored in the target storage space, and obtain blocks of the third matrix. The processing unit 920 is specifically configured to clear the data stored in the target storage space and write the blocks of the third matrix into the target storage space.

[0194] In one embodiment, the transceiver unit 910 is specifically used to obtain multiple first messages cached locally. The processing unit 920 is specifically used to perform matrix addition calculations on the data recorded in the data payload of the first message having the same target address and data length to obtain superimposed blocks. The processing unit 920 is specifically used to determine the storage space corresponding to the superimposed blocks based on the target address and data length corresponding to the superimposed blocks. The processing unit 920 is specifically used to read the data stored in the storage space corresponding to the superimposed blocks, perform matrix addition calculations on the superimposed blocks and the data stored in the storage space corresponding to the superimposed blocks, and obtain blocks of a third matrix. The processing unit 920 is specifically used to clear the data stored in the storage space corresponding to the superimposed blocks, and write the blocks of the third matrix into the storage space corresponding to the superimposed blocks.

[0195] In one embodiment, the processing unit 920 is further configured to obtain a destination address from a preset destination address according to the generation order of the blocks in the third sub-matrix. The processing unit 920 is specifically configured to obtain a data length from a preset data length and configure a destination address and data length for the first message corresponding to the block in the third sub-matrix.

[0196] In one embodiment, the processing unit 920 is further configured to generate a second message based on the blocks of the third matrix and transmit the second message to a local memory. The second message is used to instruct the local memory to store the blocks of the third matrix in a designated storage space.

[0197] In one embodiment, the transceiver unit 910 is further configured to configure a storage space of a set size in the local storage space after receiving the configuration instruction sent by the first processing unit, and clear the storage space of the set size to 0.

[0198] In one embodiment, the transceiver unit 910 is further configured to, after receiving the segmentation granularity information sent by the first processing unit, divide the third sub-matrix into a plurality of blocks according to the segmentation granularity information.

[0199] In one embodiment, the transceiver unit 910 is further configured to, upon receiving the third message, perform an OR operation on the matrix identifier in the third message and the data already stored in the identifier storage space, and store the result of the OR operation in the identifier storage space. The identifier storage space is a storage space for locally storing matrix identifiers.

[0200] A computing device is also provided in an embodiment of the present application. The computing device includes a processor, which can execute the technical solutions shown in Figures 3-8 and the corresponding protections above, so that the computing device has the technical effects of the technical solutions protected above.

[0201] A computer-readable storage medium is also provided in an embodiment of the present application, including computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes any one of the methods described in Figures 3 to 8 and the corresponding descriptions.

[0202] An embodiment of the present application also provides a computer program product comprising instructions, characterized in that the computer program product stores instructions that, when executed by a computing device, enable the computing device to implement any one of the methods described in Figures 3-8 and the corresponding descriptions.

[0203] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0204] In addition, various aspects or features of the embodiments of the present application can be implemented as methods, devices or products using standard programming and / or engineering techniques. The term "product" used in this application covers computer programs that can be accessed from any computer-readable device, carrier or medium. For example, computer-readable media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks or tapes, etc.), optical disks (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), smart cards and flash memory devices (e.g., erasable programmable read-only memories (EPROMs), cards, sticks or key drives, etc.). In addition, the various storage media described herein may represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0205] In the above embodiment, the training device 900 of the model in Figure 9 can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0206] It should be understood that in various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0207] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0208] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0209] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0210] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or an access network device, etc.) to execute all or part of the steps of the method described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0211] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited to this. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in the embodiment of the present application, and they should all be covered by the protection scope of the embodiment of the present application.

Claims

1. A computing system, characterized in that: include: Multiple second processing units are used to receive a first sub-matrix and a second sub-matrix, perform matrix multiplication on the first sub-matrix and the second sub-matrix, and send the blocks of the third sub-matrix in sequence according to the generation order of the blocks in the third sub-matrix; wherein the first sub-matrix is obtained by the first processing unit by splitting the first matrix to be subjected to matrix multiplication, the second sub-matrix is obtained by the first processing unit by splitting the second matrix to be subjected to matrix multiplication, and the third sub-matrix is a sub-matrix obtained by matrix multiplication of the first sub-matrix and the second sub-matrix.

2. The computing system according to claim 1, wherein: The second processing unit is specifically configured to generate a first message according to the blocks of the third submatrix and send the first message; the first message includes a target address, a data length, a data type, and a data payload, wherein the target address refers to the location of a target storage space in which the third processing unit stores the blocks of the third submatrix, the data length refers to the size of the blocks of the third submatrix, the data type refers to the data type of the blocks of the third submatrix, and the data payload is used to record the blocks of the third submatrix. The third processing unit is a second processing unit or a memory among the multiple second processing units.

3. The computing system according to claim 2, wherein: The third processing unit is configured to, after receiving the first messages from the plurality of second processing units, parse the first messages to obtain blocks of the third submatrix, and perform matrix addition calculation on the plurality of blocks of the third submatrix to obtain blocks of a third matrix; The third matrix is a matrix obtained by performing matrix addition calculation on the third sub-matrices generated by the multiple second processing units.

4. The computing system according to claim 3, wherein: The third processing unit is specifically configured to determine the target storage space according to the target address and the data length in the first message; Reading data stored in the target storage space, performing matrix addition calculation on the data recorded in the data payload in the first message and the data stored in the target storage space to obtain blocks of the third matrix; The data stored in the target storage space is cleared, and the blocks of the third matrix are written into the target storage space.

5. The computing system according to claim 3, wherein: The third processing unit is specifically configured to obtain multiple first messages cached locally, and perform matrix addition calculation on the data recorded in the data payload in the first messages having the same target address and data length to obtain superimposed blocks; Determining a storage space corresponding to the superimposed blocks according to the target address and the data length corresponding to the superimposed blocks; Reading data stored in a storage space corresponding to the superimposed blocks, performing matrix addition calculation on the superimposed blocks and the data stored in the storage space corresponding to the superimposed blocks, to obtain the blocks of the third matrix; The data stored in the storage space corresponding to the superimposed blocks is cleared, and the blocks of the third matrix are written into the storage space corresponding to the superimposed blocks.

6. The computing system according to any one of claims 2 to 5, wherein: The multiple second processing units are specifically used to obtain the destination address from the preset destination address and the data length from the preset data length according to the generation order of the blocks in the third sub-matrix, and configure the destination address and data length for the first message corresponding to the blocks in the third sub-matrix.

7. The computing system according to any one of claims 3 to 6, wherein: The third processing unit is one of the plurality of second processing units. The third processing unit is further configured to send the blocks of the third matrix to a second processing unit connected to a local output port.

8. The computing system according to any one of claims 3 to 7, wherein: The second processing unit is further used to generate a second message based on the blocks of the third matrix and transmit the second message to a local memory; the second message is used to instruct the local memory to store the blocks of the third matrix in a designated storage space.

9. The computing system according to any one of claims 3 to 8, wherein: The first processing unit is further configured to send a configuration instruction to the third processing unit; the configuration instruction is used to instruct the third processing unit to configure a storage space for storing the third matrix; The third processing unit is further configured to configure a storage space of a set size in the local storage space after receiving the configuration instruction, and clear the storage space of the set size to 0.

10. The computing system according to any one of claims 1 to 9, wherein: The first processing unit is further configured to send segmentation granularity information to the plurality of second processing units; The multiple second processing units are further configured to, after receiving the segmentation granularity information, divide the third sub-matrix into multiple blocks according to the segmentation granularity information.

11. The computing system according to any one of claims 3 to 9, wherein: The plurality of second processing units are further configured to send a third message to the third processing unit; the third message includes a matrix identifier, and the matrix identifier is used to allow the reading device to determine the second processing unit to which the read data belongs; The third processing unit is further configured to, after receiving the third message, perform an OR operation on the matrix identifier in the third message and the data already stored in the identifier storage space, and store the result of the OR operation in the identifier storage space; the identifier storage space is a storage space for storing matrix identifiers in the third processing unit.

12. A model training method, characterized in that: The computing system includes a first processing unit and a plurality of second processing units, the method being performed by one of the plurality of second processing units, the method including: receiving a first submatrix and a second submatrix, wherein the first submatrix is obtained by the first processing unit splitting the first matrix to be subjected to matrix multiplication, and the second submatrix is obtained by the first processing unit splitting the second matrix to be subjected to matrix multiplication; performing matrix multiplication calculation on the first sub-matrix and the second sub-matrix; The blocks of the third submatrix are sent sequentially according to the generation order of the blocks in the third submatrix; the third submatrix is a submatrix obtained by matrix multiplication of the first submatrix and the second submatrix.

13. The method according to claim 12, characterized in that The method further comprises: A first message is generated according to the block of the third submatrix, and the first message is sent; the first message includes a target address, a data length, a data type, and a data payload, the target address refers to the location of the target storage space where the third processing unit stores the block of the third submatrix, the data length refers to the size of the block of the third submatrix, the data type refers to the data type of the block of the third submatrix, and the data payload is used to record the block of the third submatrix, and the third processing unit is one of the multiple second processing units or a memory.

14. The method according to claim 13, characterized in that The method further comprises: After receiving the first message sent by the other second processing unit, parsing the blocks of the third sub-matrix in the first message; Matrix addition calculation is performed on multiple blocks of the third sub-matrix to obtain blocks of a third matrix; the third matrix is a matrix obtained by performing matrix addition calculation on the third sub-matrices generated by the multiple second processing units.

15. The method according to claim 14, characterized in that The performing matrix addition calculation on the multiple blocks of the third submatrix to obtain blocks of the third matrix specifically includes: determining the target storage space according to the target address and the data length in the first message; Reading data stored in the target storage space, performing matrix addition calculation on the data recorded in the data payload in the first message and the data stored in the target storage space to obtain blocks of the third matrix; The data stored in the target storage space is cleared, and the blocks of the third matrix are written into the target storage space.

16. The method according to claim 14, characterized in that The performing matrix addition calculation on the multiple blocks of the third submatrix to obtain blocks of the third matrix specifically includes: Acquire multiple first messages cached locally, and perform matrix addition calculation on the data recorded in the data payload in the first messages having the same target address and the same data length to obtain superimposed blocks; Determining a storage space corresponding to the superimposed blocks according to the target address and the data length corresponding to the superimposed blocks; Reading data stored in a storage space corresponding to the superimposed blocks, performing matrix addition calculation on the superimposed blocks and the data stored in the storage space corresponding to the superimposed blocks, to obtain the blocks of the third matrix; The data stored in the storage space corresponding to the superimposed blocks is cleared, and the blocks of the third matrix are written into the storage space corresponding to the superimposed blocks.

17. The method according to any one of claims 13 to 16, characterized in that: The method further comprises: According to the generation order of the blocks in the third sub-matrix, the destination address is obtained from the preset destination address, and the data length is obtained from the preset data length, and the destination address and data length are configured for the first message corresponding to the blocks in the third sub-matrix.

18. The method according to any one of claims 12 to 17, characterized in that: The method further comprises: A second message is generated according to the blocks of the third matrix, and the second message is transmitted to a local memory; the second message is used to instruct the local memory to store the blocks of the third matrix in a designated storage space.

19. The method according to any one of claims 12 to 18, wherein: Before receiving the first sub-matrix and the second sub-matrix, the method further includes: After receiving the configuration instruction sent by the first processing unit, a storage space of a set size is configured in the local storage space, and the storage space of the set size is cleared to 0.

20. The method according to any one of claims 12 to 19, characterized in that: Before receiving the first sub-matrix and the second sub-matrix, the method further includes: After receiving the segmentation granularity information sent by the first processing unit, the third sub-matrix is divided into a plurality of blocks according to the segmentation granularity information.

21. The method according to any one of claims 12 to 20, characterized in that: The method further comprises: After receiving the third message, the matrix identifier in the third message is ORed with the data already stored in the identifier storage space, and the result of the ORed operation is stored in the identifier storage space; the identifier storage space is a storage space for locally storing matrix identifiers.

22. A computing device, characterized in that include: at least one memory; At least one processor, wherein the processor is configured to execute instructions stored in the memory so that the computing device performs the method according to any one of claims 12 to 21.

23. A computer-readable storage medium, characterized in that The method comprises computer program instructions, and when the computer program instructions are executed by a computing device, the computing device performs the method according to any one of claims 12 to 21.

24. A computer program product comprising instructions, characterized in that The computer program product stores instructions, which, when executed by a computing device, cause the computing device to implement the method according to any one of claims 12 to 21.

Images