Data processing method and related device

Abstract

The present application provides a data processing method, which can be applied to a compute cluster deployed with a multi-head attention network. A first compute card performs an allgather operation on first data to obtain second data, the first data being a sub-matrix of a feature matrix of input data of the multi-head attention network; the first compute card performs QKV computation on the second data and a first sub-model, the first sub-model being a sub-matrix obtained by performing row partitioning and column partitioning on a weight matrix of the multi-head attention network; and the first compute card performs a reducescatter operation on the result of the QKV computation to obtain third data, the third data being used for acquiring a processing result of the first data. In the present solution, in terms of model parallelism, two communication domains are set, the weight matrix of the multi-head attention network is partitioned along two dimensions, each training node only stores a sub-matrix, and the sub-matrix occupies less GPU memory, thereby significantly reducing communication volume.

Description

A data processing method and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202411397872.6, filed with the China National Intellectual Property Administration on September 30, 2024, entitled "A Data Processing Method and Related Equipment", and to Chinese Patent Application No. 202410939012.4, filed with the China National Intellectual Property Administration on July 12, 2024, entitled "A Data Processing Method and Related Equipment", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and more particularly to a data processing method and related equipment. Background Technology

[0003] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0004] Training on very long sequences is a challenge in Natural Language Processing (NLP), involving the processing and generation of long text sequences. Processing long sequences requires significant memory and computational resources. In Transformer architecture models, the memory requirements of the self-attention layer are proportional to the square of the sequence length, meaning that increasing the sequence length rapidly increases the computational resource requirements.

[0005] Therefore, improving data processing efficiency is a pressing technical problem to be solved when using the Transformer architecture for model process data processing. Summary of the Invention

[0006] This application provides a data processing method and related equipment, which reduces the total communication overhead by introducing multiple communication operations corresponding to multiple parallel strategies.

[0007] The first aspect of this application provides a data processing method applied in a computing cluster deploying a multi-head attention network, enabling parallel data processing of the multi-head attention network. Specifically, the method can be executed by the computing cluster (including multiple computing cards), or by some components of the computing cluster (e.g., processors, chips, or chip systems), or by a logic module or software capable of implementing all or part of the computing card functions. In this first aspect and its possible implementations, the method is described using the execution of a first computing card as an example.

[0008] The computing cluster includes N computing cards and performs multiple communication operations, including: reducescatter operation, alltoall operation and allgather operation, and the first computing card is any one of the N computing cards;

[0009] In this method, the first computing card performs an allgather operation on the first data to obtain the second data, where the first data is a submatrix of the feature matrix of the input data of the multi-head attention network; the first computing card performs QKV calculation on the second data and the first sub-model, where the first sub-model is a submatrix obtained by row-slicing and column-slicing the weight matrix of the multi-head attention network; the first computing card performs a reducescatter operation on the result of the QKV calculation to obtain the third data, which is used to obtain the processing result of the first data.

[0010] In terms of model parallelism, this scheme sets up two communication domains and partitions the weight matrix of the multi-head attention network in two dimensions. Row and column partitioning of the weight matrix ensures that each training node only needs to store a sub-matrix, resulting in lower GPU memory usage and significantly reduced communication overhead. Furthermore, column partitioning of the input data further reduces static memory usage. Since the partitioned data or model may not meet the requirements of matrix multiplication, this scheme achieves computational equivalence by inserting allgather and reducescatter operators before and after QKV computation.

[0011] In one possible implementation, the method further includes: dividing the feature matrix of the input data of the multi-head attention network in the sequence dimension to obtain C sub-sequence data; and performing average column cutting on any one of the C sub-sequence data to obtain the first data.

[0012] By segmenting the input data into sequences, the GPU memory usage is further reduced. In large model and long sequence training and inference scenarios, this solution reduces overall overhead by combining model parallelism and hierarchical sequence parallelism.

[0013] The first computing card performs the allgather operation on the KV matrix in the third data to obtain the fourth data; it then performs self-attention calculation on the Q matrix in the third data and the KV matrix in the fourth data to obtain the fifth data; the first computing card performs the allgather operation on the fifth data to obtain the sixth data; the sixth data is multiplied by the second sub-model to obtain the seventh data, where the second sub-model is a sub-matrix obtained by row-slicing and column-slicing the weight matrix of the multi-head attention network; and the seventh data is subjected to the reducescatter operation to obtain the processing result of the first data.

[0014] In one possible implementation, the first data is the average column cut matrix of the feature matrix of the input data of the multi-head attention network.

[0015] In one possible implementation, the first sub-model is a sub-matrix obtained based on a first partitioning scheme, and the second sub-model is a sub-matrix obtained based on a second partitioning scheme. The method further includes:

[0016] Based on the first constraint, the second constraint, and the third constraint, the first segmentation scheme and the second segmentation scheme are calculated.

[0017] The parameter matrix of the multi-head attention network is segmented based on the first segmentation scheme and the second segmentation scheme respectively to obtain a first sub-model and a second sub-model;

[0018] The first constraint is satisfied:

[0019] The second constraint satisfies: A = p1 * p2

[0020] The third constraint satisfies: A=p1′*p2′

[0021] Where A represents the parallelism of the first parallel strategy.

[0022] p1 is the number of rows in the first segmentation scheme.

[0023] p2 is the number of columns in the first segmentation scheme.

[0024] b, s, and h represent the batch size of the input data, the sequence length of the input data, and the hidden layer dimension of the multi-head attention network, respectively, and p1' represents the number of rows in the second segmentation scheme.

[0025] p2' is the number of columns in the second segmentation scheme.

[0026] The optimal segmentation scheme is obtained by minimizing the communication cost. The first segmentation scheme can be the same as or different from the second segmentation scheme. The optimal segmentation scheme of the model is calculated through the first, second and third constraints. It does not rely on human experience and has higher accuracy.

[0027] The second aspect of this application provides a data processing method applied to a multi-head attention network in a computing cluster. The computing cluster includes N computing cards and performs multiple communication operations, including all-to-all operations and parallel operations. The parallel operations are all-gather operations or ring operations. The first computing card is any one of the N computing cards, where N is an integer greater than 1.

[0028] Secondly, taking the first computing card as an example, the data processing method provided in the first aspect is introduced. In this method, the first computing card first performs QKV calculation. Then, the first computing card performs an all-to-all operation based on the result of the QKV calculation. Based on the result of the all-to-all operation, the first computing card performs parallel operations, which are used to obtain the data processing results of the multi-head attention network.

[0029] QKV computation can be understood as QKV computation within a multi-head attention network. Specifically, it includes the computation of the Q matrix, K matrix, and V matrix within the multi-head attention network. The alltoall operation corresponds to the second parallel strategy, and the parallel operation corresponds to the third parallel strategy. The second parallel strategy can also be called a communication-efficient sequence parallel strategy, and the third parallel strategy can also be called a memory / memory-efficient sequence parallel strategy. This approach can also be understood as multiple communication operations including: an alltoall operation (the second communication operation) + an allgather / ring operation (the third communication operation or parallel operation).

[0030] Based on the above scheme, compared with the existing schemes that use a single parallel strategy for data processing, the scheme provided in this application can improve the data processing efficiency of multi-head attention networks for input data. Specifically, by introducing multiple parallel strategies and limiting the execution order between parallel strategies (starting with the second parallel strategy and ending with the third parallel strategy), the communication overhead introduced by each parallel strategy can be reduced, thereby reducing the total communication overhead and improving data processing efficiency. In addition, the scheme provided in this application can also achieve communication computation masking. For example, the second communication operation of the Q matrix can mask the calculation of the K and V matrices, the second and third communication operations of the K matrix can mask the calculation of the V matrix, and the second and third communication operations of the V matrix can mask the calculation of the attention score (i.e., the process of obtaining the second output matrix), thereby increasing throughput. Furthermore, compared with the existing two-dimensional model parallel scheme (i.e., the input sequence cannot be segmented and the length of the input sequence is limited), this application can not only achieve sequence segmentation but also model parallelism through multi-dimensional parallelism, thereby improving the data processing efficiency of multi-head attention networks.

[0031] In one possible implementation, the aforementioned communication operations also include an allreduce operation. The first computing card can also perform an allreduce operation based on the results of parallel operations, which is used to obtain the data processing results.

[0032] The `allreduce` operation (the first communication operation) corresponds to the first parallel strategy, which can also be called the model parallel strategy. This approach can also be understood as multiple communication operations including: `alltoall` operation + `allgather / ring` operation + `allreduce` operation.

[0033] Based on the above scheme, by further adding allreduce operations to multiple communication operations and limiting the execution order between parallel strategies (first parallel strategy starts, second parallel strategy starts, third parallel strategy starts and ends, second parallel strategy ends, first parallel strategy ends), the communication overhead introduced by each parallel strategy can be reduced, thereby reducing the total communication overhead and improving data processing efficiency. Furthermore, the scheme provided in this application can also achieve communication computation masking. For example, the second communication operation of the Q matrix can mask the computation of the K and V matrices; the second and third communication operations of the K matrix can mask the computation of the V matrix; and the second and third communication operations of the V matrix can mask the computation of the attention score (i.e., the process of obtaining the second output matrix), thereby increasing throughput. In addition, compared to existing two-dimensional model parallel schemes (i.e., the input sequence cannot be segmented, and the length of the input sequence is limited), this application, through multi-dimensional parallelism, can not only achieve sequence segmentation but also model parallelism, thereby improving the data processing efficiency of multi-head attention networks.

[0034] In one possible implementation, N = A * B * C; A represents the parallelism of the first parallel strategy corresponding to the allreduce operation, B represents the parallelism of the second parallel strategy corresponding to the alltoall operation, and C represents the parallelism of the third parallel strategy corresponding to the parallel operation. The second parallel strategy has multiple second-group computing cards running in parallel, and the third parallel strategy has multiple third-group computing cards running in parallel. A, B, and C are positive integers greater than 1. The first computing card performs an alltoall operation based on the result of the QKV computation, including: within the second-group computing card to which the first computing card belongs, the first computing card performs an alltoall operation based on the result of the QKV computation. The first computing card performs parallel operations based on the result of the alltoall operation, including: within the third-group computing card to which the first computing card belongs, the first computing card performs parallel operations based on the result of the alltoall operation. The first computing card performs a fully reduced allreduce operation based on the result of the parallel operation, including: the first computing card performs both alltoall and allreduce operations based on the result of the parallel operation.

[0035] Based on the above scheme, by limiting the groups that each communication operation is adapted to, the first computing card can obtain data from other computing cards in the computing cluster through different communication operations under each group, thereby achieving more efficient parallel operation.

[0036] In one possible implementation, the first parallel strategy uses B*C first-group computation cards, the second parallel strategy uses A*C second-group computation cards, and the third parallel strategy uses A*B third-group computation cards. The first computation card is ranked Xth in its first-group, Yth in its second-group, and Zth in its third-group, where X is greater than 0 and less than or equal to A; Y is greater than 0 and less than or equal to B; and Z is greater than 0 and less than or equal to C. Specifically, the first computation card performs QKV calculation based on the first submatrix (i.e., a submatrix of the subsequent feature matrix) and the second submatrix (i.e., a submatrix of the subsequent complete weight matrix) to obtain the QKV calculation result. The first computation card also performs an all-to-all operation based on the third submatrix to obtain the all-to-all operation result. Finally, the first computation card performs parallel operations based on the fourth submatrix to obtain the parallel operation result.

[0037] The QKV calculation results include a third submatrix. The first submatrix is ​​the B*(Z-1)+Yth submatrix from top to bottom after the feature matrix is ​​divided into B*C equal rows; the feature matrix is ​​the feature matrix corresponding to the input data of the multi-head attention network. The second submatrix is ​​the Xth submatrix from left to right after the complete weight matrix is ​​divided into A equal columns. The third submatrix is ​​the B*(Z-1)+Yth submatrix from top to bottom after the rows of the complete QKV matrix are divided into B*C equal rows. The third submatrix is ​​the Xth submatrix from left to right after the columns of the complete QKV matrix are divided into A equal columns. The complete QKV matrix is ​​related to the feature matrix and the complete weight matrix. The alltoall operation results in a fourth submatrix. The fourth submatrix is ​​the (X-1)*B+Yth submatrix from left to right after the columns of the complete QKV matrix are divided into A*B equal columns. The fourth submatrix is ​​the Zth submatrix from top to bottom after the rows of the complete QKV matrix are divided into C equal rows. The result of the parallel operation is the first output matrix of the self-attention module. The multi-head attention network includes a self-attention module and a linear module. The first output matrix has the same number of rows and columns as the fourth sub-matrix, but the contents are different.

[0038] Based on the above scheme, by obtaining the corresponding number of sub-matrices through each communication operation and limiting the execution order between the communication operations (starting with the first parallel strategy, starting with the second parallel strategy, ending with the third parallel strategy, ending with the second parallel strategy, and ending with the first parallel strategy), the communication overhead introduced by each parallel strategy can be reduced, thereby reducing the total communication overhead and improving data processing efficiency. Furthermore, the scheme provided in this application can also achieve communication computation masking. For example, the second communication operation of the Q matrix can mask the calculations of the K and V matrices; the second and third communication operations of the K matrix can mask the calculation of the V matrix; and the second and third communication operations of the V matrix can mask the calculation of the attention score (i.e., the process of obtaining the second output matrix), thereby increasing throughput. In addition, compared to existing two-dimensional model parallel schemes (i.e., the input sequence cannot be segmented, and the length of the input sequence is limited), this application, through multi-dimensional parallelism, can not only achieve sequence segmentation but also model parallelism, thereby improving the data processing efficiency of multi-head attention networks.

[0039] In one possible implementation, the first computation card performs an allgather operation on the KV matrix in the fourth submatrix to obtain the fifth submatrix. The first computation card then performs attention calculations on the Q matrix in the fourth submatrix and the fifth submatrix to obtain the first output matrix. The fifth submatrix is ​​the (X-1)*B+Yth part from left to right of the KV matrix in the complete QKV matrix, which has been divided into A*B parts by average column division.

[0040] Based on the above scheme, when the parallel operation is performed using the allgather method, on the one hand, the allgathering of the K matrix can mask the calculation of the V matrix. The allgathering of the V matrix can also mask the calculation of the attention score, thereby increasing throughput. On the other hand, the fifth submatrix can be directly obtained from multiple other computation cards through allgathering, allowing the first output matrix to be calculated directly from the fifth submatrix.

[0041] In one possible implementation, the first computing card performs attention calculations based on the fourth submatrix to obtain the third output matrix. The first computing card performs a ring operation with the second computing card based on the fourth submatrix to obtain a sixth submatrix, and the second computing card is located in the third group of computing cards to which the first computing card belongs. The first computing card performs attention calculations based on the sixth submatrix to obtain the fourth output matrix. The first computing card obtains the first output matrix based on the third and fourth output matrices.

[0042] Based on the above scheme, when parallel operations are performed using a ring method, on the one hand, the ring operation of the K matrix can mask the computation of the V matrix. The ring operation of the V matrix can also mask the computation of the attention score, thereby increasing throughput. On the other hand, the ring method allows the sixth submatrix to be obtained from the second computation card first, and the third output matrix obtained from the fourth submatrix and the fourth output matrix obtained from the sixth submatrix to obtain the first output matrix. Compared to the allgather method, the ring method requires multiple communication and accumulation operations.

[0043] In one possible implementation, the first computing card is specifically located within the second group of computing cards to which it belongs. The first computing card performs an all-to-all operation based on the result of the parallel operations to obtain the seventh submatrix. The first computing card then obtains the eighth submatrix based on the mapping matrix between the seventh submatrix and the average row-cut matrix. Within the first group of computing cards to which the first computing card belongs, the first computing card performs an all-reduce operation based on the eighth submatrix to obtain the second output matrix.

[0044] The seventh submatrix has the same number of rows as the B*(Z-1)+Yth row group from top to bottom after the rows of the complete QKV matrix are divided into B*C parts, but its content is different. The seventh submatrix also has the same number of columns as the Xth column group from left to right after the columns of the complete QKV matrix are divided into A parts, but its content is different. The eighth submatrix has the same number of rows and columns as the first submatrix, but its content is different. The eighth submatrix also has the same number of rows and columns as the second output matrix, but its content is different. The mapping matrix has the same number of rows as the complete weight matrix, and the mapping matrix has the same number of columns as the complete weight matrix. The second output matrix has the same number of rows and columns as the first submatrix, but its content is different. The second output matrix is ​​the output matrix of the linear module in the multi-head attention network.

[0045] Based on the above scheme, the seventh submatrix is ​​first obtained through the alltoall operation of the first output matrix, and then the second output matrix is ​​obtained through the allreduce operation of the seventh submatrix. That is, the output of the first computing card can be obtained through the second communication operation and the first communication operation, and then the data processing results of the input data can be obtained by combining the outputs of other computing cards, thereby improving the parallel processing efficiency of the input data.

[0046] In one possible implementation, N = B * C; B represents the parallelism of the second parallel strategy corresponding to the alltoall operation, and C represents the parallelism of the third parallel strategy corresponding to the parallel operation. The second parallel strategy has multiple second-group computing cards, and the third parallel strategy has multiple third-group computing cards. B and C are positive integers greater than 1. Within the second-group computing card to which the first computing card belongs, the first computing card performs the alltoall operation based on the result of the QKV calculation. Within the third-group computing card to which the first computing card belongs, the first computing card performs parallel operations based on the result of the alltoall operation. Within the second-group computing card to which the first computing card belongs, the first computing card can also perform the alltoall operation based on the result of the parallel operation. The alltoall operation is used to obtain the data processing result. This method can also be understood as: alltoall operation + allgather / ring operation + alltoall operation, that is, the second parallel strategy starts, the third parallel strategy starts and ends, and the second parallel strategy ends.

[0047] Based on the above scheme, compared with the existing schemes that use a single parallel strategy for data processing, the scheme provided in this application can improve the data processing efficiency of multi-head attention networks for input data. Specifically, by introducing multiple parallel strategies and limiting the execution order between parallel strategies (starting with the second parallel strategy, ending with the third parallel strategy, and ending with the second parallel strategy), the communication overhead introduced by each parallel strategy can be reduced, thereby reducing the total communication overhead and improving data processing efficiency. In addition, the scheme provided in this application can also achieve communication computation masking. For example, the second communication operation of the Q matrix can mask the calculation of the K and V matrices, the second and third communication operations of the K matrix can mask the calculation of the V matrix, and the second and third communication operations of the V matrix can mask the calculation of the attention score (i.e., the process of obtaining the second output matrix), thereby increasing throughput. Furthermore, compared with the existing two-dimensional model parallel scheme (i.e., the input sequence cannot be segmented and the length of the input sequence is limited), this application can not only achieve sequence segmentation but also model parallelism through multi-dimensional parallelism, thereby improving the data processing efficiency of multi-head attention networks.

[0048] In one possible implementation, the second parallel strategy uses C second-group computation cards in parallel, and the third parallel strategy uses B third-group computation cards in parallel. The first computation card is ranked Y-th in its second-group and Z-th in its third-group; Y is greater than 0 and less than or equal to B; Z is greater than 0 and less than or equal to C. The first computation card specifically performs QKV calculation based on the first submatrix and the complete weight matrix to obtain the QKV calculation result. The first computation card specifically performs an all-to-all operation based on the ninth submatrix to obtain the all-to-all operation result. The first computation card specifically performs parallel operations based on the tenth submatrix to obtain the parallel operation result. The first computation card specifically performs an all-to-all operation based on the fifth output matrix to obtain the eleventh submatrix. The first computation card specifically obtains the sixth output matrix based on the eleventh submatrix and the mapping matrix.

[0049] The QKV calculation results include: a ninth submatrix, where the first submatrix is ​​the B*(Z-1)+Yth submatrix from top to bottom after the feature matrix has been divided into B*C rows on average; the feature matrix is ​​the feature matrix corresponding to the input data of the multi-head attention network; the rows of the ninth submatrix are the B*(Z-1)+Yth submatrix from top to bottom after the rows of the complete QKV matrix have been divided into B*C rows on average, and the columns of the ninth submatrix are the columns of the complete QKV matrix. The complete QKV matrix is ​​related to the feature matrix and the complete weight matrix. The alltoall operation results in a tenth submatrix, where the columns of the tenth submatrix are the Yth submatrix from left to right after the columns of the complete QKV matrix have been divided into B rows on average, and the rows of the tenth submatrix are the Zth submatrix from top to bottom after the rows of the complete QKV matrix have been divided into C rows on average. The parallel operation results in the fifth output matrix of the self-attention module. The multi-head attention network includes a self-attention module and a linear module. The fifth output matrix has the same number of rows and columns as the tenth submatrix, but the content is different. The eleventh submatrix has the same number of rows as the B*(Z-1)+Yth division from top to bottom of the full QKV matrix after the rows are divided into B*C equal parts, but its content is different. The eleventh submatrix also has the same number of columns as the full QKV matrix, but its content is different. The mapping matrix has the same number of rows as the full weight matrix, and the mapping matrix has the same number of columns as the full weight matrix.

[0050] Based on the above scheme, by limiting the grouping corresponding to each parallel strategy, the first computing card performs multiple communication operations in an ordered manner according to the order of different groups (starting with the second parallel strategy, ending with the third parallel strategy, and ending with the second parallel strategy). This reduces the communication overhead introduced by each parallel strategy, thereby reducing the total communication overhead and improving data processing efficiency. Furthermore, the scheme provided in this application can also achieve communication computation masking. For example, the all-to-all operation of the Q matrix can mask the computation of the K and V matrices; the all-to-all operation and parallel operation of the K matrix can mask the computation of the V matrix; and the all-to-all operation and parallel operation of the V matrix can mask the computation of the attention score, thereby increasing throughput.

[0051] In one possible implementation, the first computation card performs an allgather operation on the KV matrix in the tenth submatrix to obtain the twelfth submatrix. The first computation card then performs attention calculations on the Q matrix in the tenth submatrix and the twelfth submatrix to obtain the fifth output matrix. The twelfth submatrix is ​​the (X-1)*B+Yth part from left to right of the KV matrix in the complete QKV matrix after it has been divided into A*B parts by average columns.

[0052] Based on the above scheme, when the parallel operation is performed using the allgather method, on the one hand, the allgathering of the K matrix can mask the computation of the V matrix. The allgathering of the V matrix can also mask the computation of the attention score, thereby increasing throughput. On the other hand, the twelfth submatrix can be directly obtained from multiple other computation cards through allgathering, thus allowing the fifth output matrix to be directly calculated from the twelfth submatrix.

[0053] In one possible implementation, the first computation card performs attention calculations based on the tenth submatrix to obtain the seventh output matrix. The first computation card performs a ring operation with the second computation card based on the tenth submatrix to obtain the thirteenth submatrix, and the second computation card is located in the third group of computation cards to which the first computation card belongs. The first computation card performs attention calculations based on the thirteenth submatrix to obtain the eighth output matrix. The first computation card uses the seventh and eighth output matrices to obtain the fifth output matrix.

[0054] Based on the above scheme, when parallel operations are performed using a ring method, on the one hand, the ring operation of the K matrix can mask the computation of the V matrix. The ring operation of the V matrix can also mask the computation of the attention score, thereby increasing throughput. On the other hand, the ring method allows the thirteenth submatrix to be obtained from the second computation card first, and the seventh output matrix obtained from the tenth submatrix and the eighth output matrix obtained from the thirteenth submatrix to obtain the fifth output matrix. Compared to the allgather method, the ring method requires multiple communication and accumulation operations.

[0055] The third aspect of this application provides a data processing method applied in a computing cluster deploying a multi-head attention network, enabling parallel data processing of the multi-head attention network. Specifically, the method can be executed by the computing cluster (including multiple computing cards), or by some components of the computing cluster (e.g., processors, chips, or chip systems), or by a logic module or software capable of implementing all or part of the computing card functions. In this third aspect and its possible implementations, the method is described using the example of execution by a first computing card.

[0056] Before describing the method, let's first give a preliminary explanation of the features involved in the third aspect: The method is applied in a computing cluster that deploys a multi-head attention network. The computing cluster includes N computing cards. The N computing cards perform multiple communication operations, including: all-to-all operation and all-reduce operation. The first computing card is any one of the N computing cards.

[0057] Secondly, taking the first computing card as an example, the data processing method provided in the second aspect is introduced. In this method, the first computing card performs QKV calculation. The first computing card performs an all-to-all operation based on the result of the QKV calculation. The first computing card performs attention calculation based on the result of the all-to-all operation. The first computing card performs an all-reduce operation based on the result of the attention calculation; the all-reduce operation is used to obtain the data processing result of the multi-head attention network.

[0058] QKV computation can be understood as QKV computation within a multi-head attention network. Specifically, this includes the computation of the Q matrix, K matrix, and V matrix within the multi-head attention network. The `allreduce` operation corresponds to the first parallel strategy, and the `alltoall` operation corresponds to the second parallel strategy. The second parallel strategy can also be called a communication-efficient sequence parallel strategy, and the first parallel strategy can also be called a model parallel strategy. This approach can also be understood as multiple communication operations including: an `alltoall` operation + an `allreduce` operation.

[0059] Based on the above scheme, compared with the existing scheme of processing data using a single parallel strategy, the scheme provided in this application can improve the data processing efficiency of multi-head attention networks for input data. Specifically, by introducing multiple parallel strategies and limiting the execution order between parallel strategies (first parallel strategy, second parallel strategy starts, second parallel strategy ends), the communication overhead introduced by each parallel strategy can be reduced, thereby reducing the total communication overhead and improving data processing efficiency. Furthermore, the scheme provided in this application can also achieve communication computation masking. For example, the all-to-all operation of the Q matrix can mask the computation of the K and V matrices, the all-to-all operation of the K matrix can mask the computation of the V matrix, and the all-to-all operation of the V matrix can mask the computation of the attention score, thereby increasing throughput.

[0060] In one possible implementation, N = A * B; A represents the parallelism of the first parallel strategy corresponding to the allreduce operation, and B represents the parallelism of the second parallel strategy corresponding to the alltoall operation. The first parallel strategy has B first-group computation cards running in parallel, and the second parallel strategy has A second-group computation cards running in parallel. A and B are positive integers greater than 1. The first computation card is ranked Xth in its first-group computation card and Yth in its second-group computation card, where X is greater than 0 and less than or equal to A; Y is greater than 0 and less than or equal to B. The first computation card specifically performs QKV computation based on the first and second submatrices to obtain the QKV computation result. Within the second-group computation card to which the first computation card belongs, the first computation card performs an alltoall operation based on the third submatrix to obtain the alltoall operation result. The first computation card specifically performs attention computation based on the fourth submatrix to obtain the attention computation result, which is the first output matrix of the self-attention module.

[0061] The QKV calculation results include: a third submatrix; the first submatrix is ​​the Y-th submatrix from top to bottom after the feature matrix is ​​divided into B equal rows; the feature matrix is ​​the feature matrix corresponding to the input data of the multi-head attention network; the second submatrix is ​​the X-th submatrix from left to right after the complete weight matrix is ​​divided into A equal columns; the third submatrix is ​​the Y-th submatrix from top to bottom after the rows of the complete QKV matrix are divided into B equal rows; the third submatrix is ​​the X-th submatrix from left to right after the columns of the complete QKV matrix are divided into A equal columns; the complete QKV matrix is ​​related to the feature matrix and the complete weight matrix. The alltoall operation results in a fourth submatrix, whose columns are the (X-1)*B+Y-th submatrix from left to right after the columns of the complete QKV matrix are divided into A*B equal columns; and the fourth submatrix is ​​the row of the complete QKV matrix. The multi-head attention network includes a self-attention module and a linear module. The first output matrix has the same number of rows and columns as the fourth submatrix, but their contents differ.

[0062] Based on the above scheme, by limiting the groups that each communication operation is adapted to, the first computing card can obtain data from other computing cards in the computing cluster through different communication operations under each group, thereby achieving more efficient parallel operation.

[0063] In one possible implementation, the first computing card specifically performs all-to-all and all-reduce operations based on the results of parallel operations.

[0064] Based on the above scheme, the execution order between parallel strategies is limited (first parallel strategy starts, second parallel strategy starts, second parallel strategy ends, first parallel strategy ends), that is, all-to-all and all-reduce operations are introduced after parallel operations. This can reduce the communication overhead introduced by each parallel strategy, thereby reducing the total communication overhead and improving data processing efficiency. Furthermore, the scheme provided in this application can also achieve communication computation masking. For example, the all-to-all operation of the Q matrix can mask the computation of the K and V matrices, the all-to-all operation of the K matrix can mask the computation of the V matrix, and the all-to-all operation of the V matrix can mask the computation of the attention score, thereby increasing throughput.

[0065] In one possible implementation, within the second group of computing cards to which the first computing card belongs, the first computing card specifically performs an all-to-all operation based on the first output matrix to obtain a seventh submatrix. The first computing card then obtains an eighth submatrix based on the mapping matrix between the seventh submatrix and the average row-cut matrix. Within the first group of computing cards to which the first computing card belongs, the first computing card performs an all-reduce operation based on the eighth submatrix to obtain a second output matrix.

[0066] In this network, the seventh submatrix has the same number of rows as the first output matrix, and the same number of columns as the first output matrix. The eighth submatrix has the same number of rows and columns as the first submatrix, but different content. The eighth submatrix also has the same number of rows and columns as the second output matrix, but different content. The mapping matrix has the same number of rows as the complete weight matrix, and the mapping matrix has the same number of columns as the complete weight matrix. The second output matrix has the same number of rows and columns as the first submatrix, but different content. The second output matrix is ​​the output matrix of the linear module in the multi-head attention network.

[0067] Based on the above scheme, by limiting the groups that each communication operation is adapted to, the first computing card can obtain data from other computing cards in the computing cluster through different communication operations under each group, thereby achieving more efficient parallel operation.

[0068] The fourth aspect of this application provides a data processing method applied in a computing cluster deploying a multi-head attention network, enabling parallel data processing of the multi-head attention network. Specifically, the method can be executed by the computing cluster (including multiple computing cards), or by some components of the computing cluster (e.g., processors, chips, or chip systems), or by a logic module or software capable of implementing all or part of the computing card functions. In this fourth aspect and its possible implementations, the method is described using the example of execution by a first computing card.

[0069] Before describing the method, let's first give a preliminary explanation of the features involved in the fourth aspect: The method is applied in a computing cluster that deploys a multi-head attention network. The computing cluster includes N computing cards. The N computing cards perform multiple communication operations, including parallel operations and allreduce operations. The parallel operations are allgather operations or ring operations. The first computing card is any one of the N computing cards.

[0070] Secondly, taking the first computing card as an example, the data processing method provided in the third aspect is introduced. In this method, the first computing card performs QKV computation. The first computing card performs parallel operations based on the results of the QKV computation. The first computing card performs an allreduce operation based on the results of the parallel operations, and the allreduce operation is used to obtain the data processing results of the multi-head attention network.

[0071] QKV computation can be understood as QKV computation within a multi-head attention network. Specifically, this includes the computation of the Q matrix, K matrix, and V matrix within the multi-head attention network. The `allreduce` operation corresponds to the first parallel strategy, and the parallel operation corresponds to the third parallel strategy. The first parallel strategy can also be called the model parallel strategy, and the third parallel strategy can also be called the memory / GPU efficient sequence parallel strategy. This approach can also be understood as multiple communication operations including: parallel operations (`allgather / ring`) + `allreduce` operations.

[0072] Based on the above solution, compared with the existing technology that uses a single parallel strategy for data processing, the solution provided in this application can improve the data processing efficiency of multi-head attention networks for input data. Specifically, by introducing multiple parallel strategies and limiting the execution order between them (first parallel strategy starts, third parallel strategy starts and ends, first parallel strategy ends), the communication overhead introduced by each parallel strategy can be reduced, thereby reducing the total communication overhead and improving data processing efficiency. Furthermore, the solution provided in this application can also achieve communication computation masking. For example, parallel operations on the V matrix can mask the calculation of attention scores, thereby increasing throughput.

[0073] In one possible implementation, N = A * C; A represents the parallelism of the allreduce operation corresponding to the first parallel strategy, and C represents the parallelism of the parallel operation corresponding to the third parallel strategy. The first parallel strategy has C first-group computing cards running in parallel, and the third parallel strategy has A third-group computing cards running in parallel. A and C are positive integers greater than 1. The first computing card is ranked Xth in its first-group computing card and Zth in its third-group computing card, where X is greater than 0 and less than or equal to A; Z is greater than 0 and less than or equal to C. Specifically, the first computing card performs QKV calculation based on the first and second submatrices to obtain the QKV calculation result. Within the third-group computing cards to which the first computing card belongs, the first computing card performs parallel operations based on the third submatric to obtain the parallel operation result. Specifically, the first computing card performs an allreduce operation based on the first output matrix and the mapping matrix that has been averaged and row-cut to obtain the eighth submatrix. Within the first-group computing cards to which the first computing card belongs, the first computing card performs an allreduce operation based on the eighth submatrix to obtain the second output matrix.

[0074] The QKV calculation results include: a third submatrix; the first submatrix is ​​the Zth submatrix from top to bottom after the feature matrix is ​​divided into C equal rows; the feature matrix is ​​the feature matrix corresponding to the input data of the multi-head attention network; the second submatrix is ​​the Xth submatrix from left to right after the complete weight matrix is ​​divided into A equal columns; the rows of the third submatrix are the Zth submatrix from top to bottom after the rows of the complete QKV matrix are divided into C equal rows; the columns of the third submatrix are the Xth submatrix from left to right after the columns of the complete QKV matrix are divided into A equal columns. The complete QKV matrix is ​​related to the feature matrix and the complete weight matrix. The result of the parallel operation is the first output matrix of the self-attention module. The multi-head attention network includes a self-attention module and a linear module. The first output matrix has the same number of rows and columns as the third submatrix, but the content is different. The eighth submatrix has the same number of rows and columns as the first submatrix, but its content is different. The eighth submatrix also has the same number of rows and columns as the second output matrix, but its content is different. The number of rows in the mapping matrix is ​​the same as the number of columns in the complete weight matrix, and the number of columns in the mapping matrix is ​​the same as the number of rows in the complete weight matrix. The second output matrix has the same number of rows and columns as the first submatrix, but its content is different. The second output matrix is ​​the output matrix of the linear module in the multi-head attention network.

[0075] Based on the above scheme, by limiting the groups that each communication operation is adapted to, the first computing card can obtain data from other computing cards in the computing cluster through different communication operations under each group, thereby achieving more efficient parallel operation.

[0076] In one possible implementation, the first computation card performs an allgather operation on the KV matrix in the third submatrix to obtain the fifth submatrix. The first computation card then performs attention calculations on the Q matrix in the third submatrix and the fifth submatrix to obtain the first output matrix. The fifth submatrix is ​​the Xth part from left to right of the KV matrix in the complete QKV matrix after it has been divided into A parts by average columns.

[0077] Based on the above scheme, when the parallel operation is performed using the allgather method, on the one hand, the allgathering of the K matrix can mask the calculation of the V matrix. The allgathering of the V matrix can also mask the calculation of the attention score, thereby increasing throughput. On the other hand, the fifth submatrix can be directly obtained from multiple other computation cards through allgathering, allowing the first output matrix to be calculated directly from the fifth submatrix.

[0078] In one possible implementation, the first computing card performs attention calculations based on the third submatrix to obtain a third output matrix. The first computing card then performs a third communication operation with the second computing card based on the third submatrix to obtain a sixth submatrix. The first computing card performs attention calculations based on the sixth submatrix to obtain a fourth output matrix. Finally, the first computing card obtains a first output matrix based on the third and fourth output matrices. The second computing card is located within the third group of computing cards to which the first computing card belongs.

[0079] Based on the above scheme, when parallel operations are performed using a ring method, on the one hand, the ring operation of the K matrix can mask the computation of the V matrix. The ring operation of the V matrix can also mask the computation of the attention score, thereby increasing throughput. On the other hand, the ring method allows the sixth sub-matrix to be obtained from the second computation card first, and the first output matrix to be obtained from the third output matrix (derived from the third sub-matrix) and the fourth output matrix (derived from the sixth sub-matrix). Compared to the allgather method, the ring method requires multiple communication and accumulation operations.

[0080] The fifth aspect of this application provides a data processing device, which can be a cloud device or a terminal device, or the device can be a component (e.g., a processor, chip, or chip system) in the cloud device or terminal device, or the device can be a logic module or software capable of implementing all or part of the functions of the data processing device. The data processing device can also be understood as a computing node or a computing card within a computing node in a computing cluster. The computing cluster deploys a multi-head attention network, and the computing cluster includes N computing cards, performing multiple communication operations, including: all-to-all operations and parallel operations. The parallel operations are all-gather operations or ring operations, and the first computing card is any one of the N computing cards.

[0081] The data processing equipment in the fifth aspect includes a processing module.

[0082] The processing module performs QKV computation. It also performs an all-to-all operation based on the QKV computation result. Furthermore, it performs parallel operations based on the all-to-all operation result to obtain the data processing results from the multi-head attention network.

[0083] In one possible implementation, the multiple communication operations also include a full reduction (allreduce) operation. The processing module is further configured to perform an allreduce operation on the first computing card based on the results of the parallel operations; the allreduce operation is used to obtain the data processing results.

[0084] In one possible implementation, N = A * B * C; A represents the parallelism of the first parallel strategy corresponding to the allreduce operation, B represents the parallelism of the second parallel strategy corresponding to the alltoall operation, and C represents the parallelism of the third parallel strategy corresponding to the parallel operation. The second parallel strategy has multiple second-group computing cards, and the third parallel strategy has multiple third-group computing cards. A, B, and C are positive integers greater than 1. Within the second-group computing card to which the first computing card belongs, a processing module is specifically used to perform alltoall operations based on the results of QKV computation. Within the third-group computing card to which the first computing card belongs, a processing module is specifically used to perform parallel operations based on the results of alltoall operations. A processing module is specifically used by the first computing card to perform alltoall and allreduce operations based on the results of parallel operations.

[0085] In one possible implementation, the first parallel strategy uses B*C first-group computation cards, the second parallel strategy uses A*C second-group computation cards, and the third parallel strategy uses A*B third-group computation cards. The first computation card is ranked Xth in its first-group, Yth in its second-group, and Zth in its third-group, where X is greater than 0 and less than or equal to A; Y is greater than 0 and less than or equal to B; and Z is greater than 0 and less than or equal to C. A processing module is specifically used to perform QKV calculations based on the first and second submatrices to obtain the QKV calculation result. A processing module is specifically used to perform an all-to-all operation based on the third submatrix to obtain the all-to-all operation result. A processing module is specifically used to perform parallel operations based on the fourth submatrix to obtain the parallel operation result.

[0086] The QKV calculation results include a third submatrix. The first submatrix is ​​the B*(Z-1)+Yth submatrix from top to bottom after the feature matrix is ​​divided into B*C equal rows; the feature matrix is ​​the feature matrix corresponding to the input data of the multi-head attention network. The second submatrix is ​​the Xth submatrix from left to right after the complete weight matrix is ​​divided into A equal columns. The third submatrix is ​​the B*(Z-1)+Yth submatrix from top to bottom after the rows of the complete QKV matrix are divided into B*C equal rows. The third submatrix is ​​the Xth submatrix from left to right after the columns of the complete QKV matrix are divided into A equal columns. The complete QKV matrix is ​​related to the feature matrix and the complete weight matrix. The alltoall operation results in a fourth submatrix. The fourth submatrix is ​​the (X-1)*B+Yth submatrix from left to right after the columns of the complete QKV matrix are divided into A*B equal columns. The fourth submatrix is ​​the Zth submatrix from top to bottom after the rows of the complete QKV matrix are divided into C equal rows. The result of the parallel operation is the first output matrix of the self-attention module. The multi-head attention network includes a self-attention module and a linear module. The first output matrix has the same number of rows and columns as the fourth sub-matrix, but the contents are different.

[0087] In one possible implementation, the processing module is specifically used to perform an allgather operation based on the KV matrix in the fourth submatrix to obtain the fifth submatrix. The processing module is specifically used to perform attention calculations based on the Q matrix in the fourth submatrix and the fifth submatrix to obtain the first output matrix. The fifth submatrix is ​​the (X-1)*B+Yth part from left to right of the KV matrix in the complete QKV matrix after it has been divided into A*B parts by average columns.

[0088] In one possible implementation, the processing module is specifically used to perform attention calculation based on the fourth submatrix to obtain the third output matrix. In another possible implementation, the processing module is specifically used to perform a ring operation based on the fourth submatrix and the second computing card, where the second computing card is located in the third group of computing cards to which the first computing card belongs. The processing module is specifically used to perform attention calculation based on the sixth submatrix to obtain the fourth output matrix. The processing module is specifically used to obtain the first output matrix based on the third and fourth output matrices.

[0089] In one possible implementation, within the second group of computing cards to which the first computing card belongs, a processing module is specifically used to perform an all-to-all operation based on the result of parallel operations to obtain the seventh submatrix. A further processing module is specifically used to obtain the eighth submatrix based on the mapping matrix between the seventh submatrix and the average row-cut matrix. Within the first group of computing cards to which the first computing card belongs, a processing module is specifically used to perform an all-reduce operation based on the eighth submatrix to obtain the second output matrix.

[0090] The seventh submatrix has the same number of rows as the B*(Z-1)+Yth row group from top to bottom after the rows of the complete QKV matrix are divided into B*C parts, but its content is different. The seventh submatrix also has the same number of columns as the Xth column group from left to right after the columns of the complete QKV matrix are divided into A parts, but its content is different. The eighth submatrix has the same number of rows and columns as the first submatrix, but its content is different. The eighth submatrix also has the same number of rows and columns as the second output matrix, but its content is different. The mapping matrix has the same number of rows as the complete weight matrix, and the mapping matrix has the same number of columns as the complete weight matrix. The second output matrix has the same number of rows and columns as the first submatrix, but its content is different. The second output matrix is ​​the output matrix of the linear module in the multi-head attention network.

[0091] In one possible implementation, N = B * C; B represents the parallelism of the second parallel strategy corresponding to the alltoall operation, and C represents the parallelism of the third parallel strategy corresponding to the parallel operation. The second parallel strategy has multiple second-group computing cards, and the third parallel strategy has multiple third-group computing cards. B and C are positive integers greater than 1. Within the second-group computing card to which the first computing card belongs, the processing module is specifically used to perform the alltoall operation based on the result of the QKV calculation. Within the third-group computing card to which the first computing card belongs, the processing module is specifically used to perform parallel operations based on the result of the alltoall operation. The processing module is also used to perform the alltoall operation based on the result of the parallel operation within the second-group computing card to which the first computing card belongs. The alltoall operation is used to obtain the data processing result.

[0092] In one possible implementation, the second parallel strategy uses C second-group computation cards in parallel, and the third parallel strategy uses B third-group computation cards in parallel; the first computation card is ranked Y-th in its second-group and Z-th in its third-group; Y is greater than 0 and less than or equal to B; Z is greater than 0 and less than or equal to C. The processing module specifically performs QKV calculation based on the first submatrix and the complete weight matrix to obtain the QKV calculation result. The processing module specifically performs an all-to-all operation based on the ninth submatrix to obtain the all-to-all operation result. The processing module specifically performs parallel operations based on the tenth submatrix to obtain the parallel operation result. The processing module specifically performs an all-to-all operation based on the fifth output matrix to obtain the eleventh submatrix. The processing module specifically obtains the sixth output matrix based on the eleventh submatrix and the mapping matrix, where the number of rows in the mapping matrix is ​​the same as the number of columns in the complete weight matrix, and the number of columns in the mapping matrix is ​​the same as the number of rows in the complete weight matrix.

[0093] The QKV calculation results include: a ninth submatrix, where the first submatrix is ​​the B*(Z-1)+Yth submatrix from top to bottom after the feature matrix has been divided into B*C rows on average; the feature matrix is ​​the feature matrix corresponding to the input data of the multi-head attention network; the rows of the ninth submatrix are the B*(Z-1)+Yth submatrix from top to bottom after the rows of the complete QKV matrix have been divided into B*C rows on average, and the columns of the ninth submatrix are the columns of the complete QKV matrix. The complete QKV matrix is ​​related to the feature matrix and the complete weight matrix. The alltoall operation results in a tenth submatrix, where the columns of the tenth submatrix are the Yth submatrix from left to right after the columns of the complete QKV matrix have been divided into B rows on average, and the rows of the tenth submatrix are the Zth submatrix from top to bottom after the rows of the complete QKV matrix have been divided into C rows on average. The parallel operation results in the fifth output matrix of the self-attention module. The multi-head attention network includes a self-attention module and a linear module. The fifth output matrix has the same number of rows and columns as the tenth submatrix, but the content is different. The eleventh submatrix has the same number of rows as the B*(Z-1)+Yth division from top to bottom of the complete QKV matrix after the rows are divided into B*C equal parts, but the content is different. The eleventh submatrix also has the same number of columns as the complete QKV matrix, but the content is different.

[0094] In one possible implementation, the processing module is specifically used to perform an allgather operation based on the KV matrix in the tenth submatrix to obtain the twelfth submatrix. The processing module is also specifically used to perform attention calculations based on the Q matrix in the tenth submatrix and the twelfth submatrix to obtain the fifth output matrix. Here, the twelfth submatrix is ​​the (X-1)*B+Yth part from left to right of the KV matrix in the complete QKV matrix after it has been divided into A*B parts by average columns.

[0095] In one possible implementation, the processing module is specifically used to perform attention calculation based on the tenth submatrix to obtain the seventh output matrix. The processing module is specifically used to perform a ring operation based on the tenth submatrix and a second computing card, where the second computing card is located in the third group of computing cards to which the first computing card belongs. The processing module is specifically used to perform attention calculation based on the thirteenth submatrix to obtain the eighth output matrix. The processing module is specifically used to obtain the fifth output matrix based on the seventh and eighth output matrices.

[0096] The sixth aspect of this application provides a data processing device, which can be a cloud device or a terminal device, or the device can be a component (e.g., a processor, chip, or chip system) in the cloud device or terminal device, or the device can be a logic module or software capable of implementing all or part of the functions of the data processing device. The data processing device can also be understood as a computing node or a computing card in a computing node within a computing cluster. The computing cluster deploys a multi-head attention network, and the computing cluster includes N computing cards. The N computing cards perform multiple communication operations, including: all-to-all operations and all-reduce operations. The first computing card is any one of the N computing cards.

[0097] The data processing device in the sixth aspect includes a processing module.

[0098] The processing module performs QKV calculations. It also performs an all-to-all operation based on the QKV calculation results. Furthermore, it performs attention calculations based on the all-to-all operation results. Finally, it performs an all-reduce operation based on the attention calculation results, which is used to obtain the data processing results of the multi-head attention network.

[0099] In one possible implementation, N = A * B; A represents the parallelism of the first parallel strategy corresponding to the allreduce operation, and B represents the parallelism of the second parallel strategy corresponding to the alltoall operation. The first parallel strategy has B first-group computation cards running in parallel, and the second parallel strategy has A second-group computation cards running in parallel. A and B are positive integers greater than 1. The first computation card is ranked Xth in its first-group computation card and Yth in its second-group computation card. X is greater than 0 and less than or equal to A; Y is greater than 0 and less than or equal to B. The processing module is specifically used to perform QKV calculation based on the first and second submatrices to obtain the QKV calculation result. The processing module is specifically used to perform an alltoall operation based on the third submatrix within the second-group computation card to which the first computation card belongs, to obtain the alltoall operation result. The processing module is specifically used to perform attention calculation based on the fourth submatrix to obtain the attention calculation result.

[0100] The QKV calculation results include: a third submatrix; the first submatrix is ​​the Y-th submatrix from top to bottom after the feature matrix is ​​divided into B equal rows; the feature matrix is ​​the feature matrix corresponding to the input data of the multi-head attention network; the second submatrix is ​​the X-th submatrix from left to right after the complete weight matrix is ​​divided into A equal columns; the third submatrix is ​​the Y-th submatrix from top to bottom after the rows of the complete QKV matrix are divided into B equal rows; the third submatrix is ​​the X-th submatrix from left to right after the columns of the complete QKV matrix are divided into A equal columns; the complete QKV matrix is ​​related to the feature matrix and the complete weight matrix. The alltoall operation results in a fourth submatrix, whose columns are the (X-1)*B+Y-th submatrix from left to right after the columns of the complete QKV matrix are divided into A*B equal columns; and the fourth submatrix is ​​the row of the complete QKV matrix. The attention calculation result is the first output matrix of the self-attention module. The multi-head attention network includes a self-attention module and a linear module. The first output matrix has the same number of rows and columns as the fourth submatrix, but the content is different.

[0101] In one possible implementation, the processing module is specifically used to perform all-to-all and all-reduce operations based on the results of parallel operations.

[0102] In one possible implementation, the processing module, specifically within the second group of computing cards to which the first computing card belongs, performs an all-to-all operation based on the first output matrix to obtain a seventh submatrix. The processing module, specifically within the first group of computing cards to which the first computing card belongs, obtains an eighth submatrix based on the mapping matrix between the seventh submatrix and the average row-cut matrix. The processing module, specifically within the first group of computing cards to which the first computing card belongs, performs an all-reduce operation based on the eighth submatrix to obtain a second output matrix.

[0103] In this network, the seventh submatrix has the same number of rows as the first output matrix, and the same number of columns as the first output matrix. The eighth submatrix has the same number of rows and columns as the first submatrix, but different content. The eighth submatrix also has the same number of rows and columns as the second output matrix, but different content. The mapping matrix has the same number of rows as the complete weight matrix, and the mapping matrix has the same number of columns as the complete weight matrix. The second output matrix has the same number of rows and columns as the first submatrix, but different content. The second output matrix is ​​the output matrix of the linear module in the multi-head attention network.

[0104] The seventh aspect of this application provides a data processing device, which can be a cloud device or a terminal device, or the device can be a component (e.g., a processor, chip, or chip system) in the cloud device or terminal device, or the device can be a logic module or software capable of implementing all or part of the functions of the data processing device. The data processing device can also be understood as a computing node or a computing card within a computing node in a computing cluster. The computing cluster deploys a multi-head attention network, and the computing cluster includes N computing cards. The N computing cards perform multiple communication operations, including parallel operations and all-reduce operations. The parallel operations are all-gather operations or ring operations, and the first computing card is any one of the N computing cards.

[0105] The data processing equipment in the seventh aspect includes a processing module.

[0106] The processing module performs QKV computation. It also performs parallel operations based on the results of the QKV computation. Furthermore, it performs an allreduce operation based on the results of the parallel operations to obtain the data processing results from the multi-head attention network.

[0107] In one possible implementation, N = A * C; A represents the degree of parallelism of the allreduce operation corresponding to the first parallel strategy, and C represents the degree of parallelism of the parallel operation corresponding to the third parallel strategy. The first parallel strategy has C first-group computing cards in parallel, and the third parallel strategy has A third-group computing cards in parallel. A and C are positive integers greater than 1. The first computing card is ranked as Xth in its first-group computing card and as Zth in its third-group computing card. X is greater than 0 and less than or equal to A; Z is greater than 0 and less than or equal to C.

[0108] The processing module is specifically used to perform QKV calculations based on the first and second submatrices to obtain the QKV calculation result. The processing module is specifically used within the third group of computing cards to which the first computing card belongs, to perform parallel operations based on the third submatrix to obtain the parallel operation result. The processing module is specifically used to perform an allreduce operation based on the first output matrix and the mapping matrix that has been averaged and row-cut to obtain the eighth submatrix. The processing module is specifically used within the first group of computing cards to which the first computing card belongs, to perform an allreduce operation based on the eighth submatrix to obtain the second output matrix.

[0109] The QKV calculation results include: a third submatrix; the first submatrix is ​​the Zth submatrix from top to bottom after the feature matrix is ​​divided into C equal rows; the feature matrix is ​​the feature matrix corresponding to the input data of the multi-head attention network; the second submatrix is ​​the Xth submatrix from left to right after the complete weight matrix is ​​divided into A equal columns; the rows of the third submatrix are the Zth submatrix from top to bottom after the rows of the complete QKV matrix are divided into C equal rows; the columns of the third submatrix are the Xth submatrix from left to right after the columns of the complete QKV matrix are divided into A equal columns. The complete QKV matrix is ​​related to the feature matrix and the complete weight matrix. The result of the parallel operation is the first output matrix of the self-attention module. The multi-head attention network includes a self-attention module and a linear module. The first output matrix has the same number of rows and columns as the third submatrix, but the content is different. The eighth submatrix has the same number of rows and columns as the first submatrix, but its content is different. The eighth submatrix also has the same number of rows and columns as the second output matrix, but its content is different. The number of rows in the mapping matrix is ​​the same as the number of columns in the complete weight matrix, and the number of columns in the mapping matrix is ​​the same as the number of rows in the complete weight matrix. The second output matrix has the same number of rows and columns as the first submatrix, but its content is different. The second output matrix is ​​the output matrix of the linear module in the multi-head attention network.

[0110] In one possible implementation, the processing module is specifically used to perform an allgather operation based on the KV matrix in the third submatrix to obtain the fifth submatrix. The processing module is also specifically used to perform attention calculations based on the Q matrix in the third submatrix and the fifth submatrix to obtain the first output matrix. Here, the fifth submatrix is ​​the Xth part from left to right of the KV matrix in the complete QKV matrix after it has been divided into A parts by average columns.

[0111] In one possible implementation, the processing module is specifically used to perform attention calculation based on the third submatrix to obtain a third output matrix. The processing module is specifically used to perform a third communication operation with the second computing card based on the third submatrix to obtain a sixth submatrix, wherein the second computing card is located in the third group of computing cards to which the first computing card belongs. The processing module is specifically used to perform attention calculation based on the sixth submatrix to obtain a fourth output matrix. The processing module is specifically used to obtain a first output matrix based on the third and fourth output matrices.

[0112] Eighthly, a data processing apparatus is provided, including at least one processor coupled to a memory; the memory is used to store a program or instructions; the at least one processor is used to execute the program or instructions to enable the data processing apparatus to implement any possible implementation of any of the first to fourth aspects described above.

[0113] A ninth aspect provides a data processing apparatus, including at least one logic circuit and an input / output interface; the logic circuit is configured to perform the method as described in any possible implementation of any of the first to fourth aspects.

[0114] In a tenth aspect, a computing device cluster is provided, comprising at least one device as described in any of the possible implementations of the fifth to ninth aspects.

[0115] Eleventhly, a computing device cluster is provided, comprising at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to enable the computing device cluster to implement any possible implementation of any of the first to fourth aspects described above.

[0116] In a twelfth aspect, a computer-readable storage medium is provided for storing one or more computer-executable instructions, which, when executed by a processor, perform the method as described in any possible implementation of any of the first to fourth aspects described above.

[0117] In a thirteenth aspect, a computer program product is provided, wherein when a computer program in the computer program product is executed by the processor, the processor executes any possible implementation of any one of the first to fourth aspects described above.

[0118] In a fourteenth aspect, a chip or chip system is provided, the chip or chip system including at least one processor for supporting a method for a communication device to implement any possible implementation of any of the first to fourth aspects described above.

[0119] In one possible design, the chip system may further include a memory for storing program instructions and data necessary for the communication device. The chip system may be composed of chips or may include chips and other discrete devices. Optionally, the chip system may also include interface circuitry that provides program instructions and / or data to at least one processor.

[0120] In a fifteenth aspect, a data processing apparatus is provided, comprising a chip system as described in the fourteenth aspect above, the chip system including a processor and a communication interface for communicating with a module outside the chip system, the processor for running computer programs or instructions such that the data processing apparatus can perform the methods of any of the above aspects.

[0121] In a sixteenth aspect, a data processing device cluster is provided, comprising at least one data processing device as described in the fifth or ninth aspect, wherein any one data processing device is configured to run a computer program or instructions, enabling the data processing device cluster to perform the methods of any of the above aspects. Alternatively, some or all of the data processing devices may be used together to run a computer program or instructions, enabling the data processing device cluster to perform the methods of any of the above aspects.

[0122] The technical effects of any of the design methods in aspects five through sixteen can be found in the technical effects of the different design methods in aspects one through four above, and will not be repeated here.

[0123] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0124] Figure 1A is an example diagram of the first communication operation provided in an embodiment of this application;

[0125] Figure 1B is an example diagram of the second communication operation provided in an embodiment of this application;

[0126] Figure 1C is an example diagram of the third communication operation provided in an embodiment of this application;

[0127] Figure 1D is an example diagram of a communication operation provided in an embodiment of this application;

[0128] Figure 2 is a schematic diagram of the system architecture provided in the embodiment of this application;

[0129] Figure 3A is a schematic diagram of a data processing system provided in an embodiment of this application;

[0130] Figure 3B is another structural schematic diagram of the data processing system provided in the embodiment of this application;

[0131] Figure 4 is another structural schematic diagram of the system architecture provided in the embodiment of this application;

[0132] Figures 5 and 6A are schematic diagrams of several processes of the first data processing method provided in the embodiments of this application;

[0133] Figure 6B is a flowchart illustrating an embodiment of this application for obtaining a third submatrix.

[0134] Figure 7 is a flowchart illustrating an example of obtaining the fourth submatrix according to an embodiment of this application;

[0135] Figure 8 is a flowchart illustrating an example of obtaining the first output matrix according to an embodiment of this application.

[0136] Figure 9 is another flowchart illustrating the process of obtaining the first output matrix provided in an embodiment of this application.

[0137] Figure 10 is a flowchart illustrating an embodiment of this application for obtaining a second output matrix.

[0138] Figure 11 is a schematic diagram of the system architecture of the 3D parallel sequence provided in the embodiment of this application;

[0139] Figure 12 is another flowchart illustrating the first data processing method provided in the embodiments of this application;

[0140] Figure 13 is another flowchart illustrating the acquisition of the third submatrix provided in an embodiment of this application.

[0141] Figure 14 is another flowchart illustrating the acquisition of the fourth submatrix provided in an embodiment of this application.

[0142] Figure 15 is another flowchart illustrating the acquisition of the first output matrix provided in an embodiment of this application.

[0143] Figure 16 is another flowchart illustrating the acquisition of the first output matrix provided in an embodiment of this application.

[0144] Figure 17 is a flowchart illustrating an example of obtaining the second output matrix according to an embodiment of this application.

[0145] Figure 18 is another schematic flowchart of the first data processing method provided in the embodiment of this application;

[0146] Figures 19A and 19B are schematic diagrams of several processes of the second data processing method provided in the embodiments of this application;

[0147] Figure 20 is another flowchart illustrating the acquisition of the third submatrix provided in an embodiment of this application.

[0148] Figure 21 is another flowchart illustrating the acquisition of the first output matrix provided in an embodiment of this application.

[0149] Figure 22 is another flowchart illustrating the process of obtaining the first output matrix provided in an embodiment of this application.

[0150] Figure 23 is a flowchart illustrating an example of obtaining the second output matrix according to an embodiment of this application.

[0151] Figures 24A and 24B are schematic diagrams of several processes of the third data processing method provided in the embodiments of this application;

[0152] Figure 25 is another flowchart illustrating the acquisition of the third submatrix provided in an embodiment of this application.

[0153] Figure 26 is another flowchart illustrating the acquisition of the fourth submatrix provided in an embodiment of this application.

[0154] Figure 27 is a flowchart illustrating an example of obtaining the second output matrix according to an embodiment of this application.

[0155] Figure 28 is a flowchart illustrating a fourth data processing method provided in an embodiment of this application.

[0156] Figure 29 is a flowchart illustrating a fourth data processing method provided in an embodiment of this application.

[0157] Figure 30 is another flowchart illustrating the fourth data processing method provided in the embodiments of this application;

[0158] Figures 31 to 33 are several schematic diagrams of the data processing equipment involved in the embodiments of this application;

[0159] Figure 34 is a schematic diagram of the structure of a data processing equipment cluster provided in an embodiment of this application;

[0160] Figure 35 is a schematic diagram of another data processing device cluster provided in an embodiment of this application. Detailed Implementation

[0161] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0162] To facilitate understanding, the relevant technologies mainly involved in the embodiments of this application will be introduced below.

[0163] 1. Neural Networks

[0164] The neural network in this embodiment can be composed of neural units, and a neural unit can refer to a neural unit represented by X. s The operation unit takes the intercept b as input, and the output of this operation unit can be:

[0165] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For X sThe weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input to the next convolutional layer. The activation function can be a ReLU function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0166] The work of each layer in a neural network can be described by the mathematical expression y = a(Wx + b). From a physical perspective, the work of each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are performed by Wx, operation 4 by +b, and operation 5 by a(). The term "space" is used here because the objects being classified are not individual things, but a class of things, and space refers to the set of all individuals of this class of things. Here, W is the weight vector, and each value in this vector represents the weight value of a neuron in that layer of the neural network. This vector W determines the spatial transformation from the input space to the output space, that is, the weight W of each layer controls how the space is transformed. The purpose of training a neural network is to ultimately obtain the weight matrix of all layers of the trained neural network (a weight matrix formed by the vectors W of many layers). Therefore, the training process of a neural network is essentially about learning how to control the transformation space, and more specifically, learning the weight matrix.

[0167] 2. Loss Function

[0168] During neural network training, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weight vector of each layer based on the difference. (Of course, there's usually an initialization process before the first update, where parameters are pre-configured for each layer). For example, if the network's prediction is too high, the weight vector is adjusted to predict a lower value, and this adjustment continues until the neural network can predict the target value accurately. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training the neural network becomes a process of minimizing this loss as much as possible.

[0169] 3. Multilayer perceptron (MLP)

[0170] A multilayer perceptron, also known as a multilayer perceptron, is a feedforward artificial neural network model that maps inputs to a single output.

[0171] 4. transformer

[0172] A transformer structure is a feature extraction network that includes both an encoder and a decoder (classified as a convolutional neural network). Of course, in some cases, a transformer structure may not include an encoder but may include a decoder.

[0173] Encoder: Learns features, such as pixel features, in the global receptive field through self-attention.

[0174] Decoder: Learns the features of the desired modules, such as the features of the output box, through self-attention and cross-attention.

[0175] The following is a description of attention (also known as the attention mechanism):

[0176] Attention mechanisms can quickly extract important features from sparse data. Attention occurs between the encoder and decoder, or more specifically, between the input and generated sentences. In contrast, the self-attention mechanism in a self-attention model occurs within the encoding matrix or the output sequence, extracting connections between distant words within the same sentence, such as syntactic features (phrase structure). Self-attention provides an effective modeling method for capturing global contextual information through QKV (key-value pairs). Assuming the input is Q (query), and the context is stored as key-value pairs (K, V), then the attention mechanism is essentially a mapping function from the query to a series of key-value pairs (key, value). The essence of the attention function can be described as a mapping from a query to a series of (key-value) pairs. Attention essentially assigns a weight coefficient to each element in the sequence, which can also be understood as soft addressing. If each element in the sequence is stored in (K, V) form, then attention performs addressing by calculating the similarity between Q and K. The similarity calculated between Q and K reflects the importance of the extracted V values, i.e., the weights, and then a weighted sum is obtained to obtain the final feature value.

[0177] Attention calculation mainly consists of three steps. The first step is to calculate the similarity between the query and each key to obtain weights. Common similarity functions include dot product, concatenation, and perceptron. The second step typically uses a softmax function (which can normalize the data, resulting in a probability distribution where the sum of all weight coefficients is 1, and also highlights the weights of important elements) to normalize these weights. Finally, the weights and their corresponding key values ​​are weighted and summed to obtain the final feature value. The specific calculation formula (hereinafter referred to as attention calculation) is as follows:

[0178] Where d is the dimension of matrix QK.

[0179] Furthermore, attention includes self-attention and cross-attention. Self-attention can be understood as a special type of attention where the inputs to the QKV features are consistent. Cross-attention, on the other hand, involves inconsistent inputs to the QKV features. Attention integrates the queried features as updated values ​​for the current features using the similarity between features (e.g., inner product) as weights. Self-attention is attention extracted based on the attention drawn from the feature map itself.

[0180] For convolutional networks, the kernel size limits the receptive field, often requiring multiple layers to focus on the entire feature map. Self-attention, on the other hand, offers the advantage of global focus; it can acquire global spatial information of the feature map through simple queries and assignments. A unique aspect of self-attention in query-key-value (QKV) models is that the inputs for each QKV value are consistent.

[0181] 5. Parallel Strategy

[0182] Parallelism is a concept introduced to improve data processing efficiency. Parallelism strategies correspond to parallelism degree and usually introduce communication operations. The following is an example description of different parallelism strategies and their corresponding communication operations.

[0183] (1) Model Parallelism (MP)

[0184] Multiprocessing (MP) encompasses both pipelined parallelism and tensor parallelism. Pipeline parallelism refers to partitioning model parameters across different layers onto different devices for computation. Tensor parallelism, on the other hand, involves partitioning parameters within a neural network model layer onto different devices. The MP discussed in this application is presumably tensor parallelism. In tensor parallelism, each device is responsible for handling the forward and backward propagation of its assigned model parameters. MP is a commonly used parallel strategy in training large models, partitioning model parameters inter-layer or intra-layer and distributing these partitions across different devices to reduce the memory consumption of a single computing device.

[0185] Optionally, common communication operations in MP include allreduce, reducescatter, etc.

[0186] AllReduce has multiple data senders and multiple data receivers. It performs the same reduce operation on all nodes in the cluster and can send the results of the data reduction operation on all nodes in the cluster to all nodes.

[0187] For example, one example of allreduce is shown in Figure 1A. Before allreduce, XPU1 stores data A, XPU2 stores data B, XPU3 stores data C, and XPU4 stores data D. After allreduce, XPU1 stores data A + data B + data C + data D, XPU2 stores data A + data B + data C + data D, XPU3 stores data A + data B + data C + data D, and XPU4 stores data A + data B + data C + data D.

[0188] For example, as shown in Figure 1D, before reducescatter, the first compute card XPU1 stores data A1 and data B1, and the second compute card XPU2 stores data A2 and data B2. After reducescatter, the first compute card XPU1 sends data B1 to the second compute card, so that data B1 and data B2 form complete data B; the second compute card XPU2 sends data A1 to the second compute card, so that A1 and A2 form complete data A.

[0189] (2) Data Parallelism (DP)

[0190] Dynamic programming (DP) is a parallel acceleration technique for model training. It involves dividing the training data samples into multiple smaller datasets and sending them to different devices for parallel computation. Each computing device has a complete copy of the model, and after each device completes the forward and backward passes, the calculated gradients are synchronized to update the model.

[0191] (3) Sequence Parallelism (SP)

[0192] SP (Sequence Parallelism) is a parallel acceleration technique for model training, primarily used to address the memory bottleneck in training large models with long sequences. It splits the training data along the sequence dimension and computes it on multiple devices in parallel. Each device has a complete copy of the model, calculates its gradients, and then updates the model synchronously.

[0193] Furthermore, SP can be divided into two types based on its parallelism effect: Communication-Efficient Sequence Parallelism (CE-SP) and Memory-Efficient Sequence Parallelism (ME-SP).

[0194] The specific process of CE-SP is as follows: First, the input data X is divided into multiple parts along the sequence on an average basis, and these parts are distributed to different processors for parallel processing. The input data X is then processed with the weight coefficient matrices Wq, Wk, and Wv, resulting in the QKV of the average sequence division on each processor. Next, three all-to-all operations are performed within the sequence parallel group, ensuring that the QKV on each processor contains a portion of the complete sequence from the head. The calculations are then performed separately. The output Y is obtained. Output Y is then processed again using the alltoall method to obtain partial sequences of all heads.

[0195] The specific process of ME-SP is as follows: First, the input data X is divided into multiple parts along the sequence on an average basis, and these parts are distributed to different processors for parallel processing. The input data X is processed separately with the weight coefficient matrices Wq, Wk, and Wv, resulting in the QKV of the average sequence split on each processor. To calculate the output Y of the average sequence split, the KV sequence needs to be complete. Therefore, within the parallel sequence group, the KV is allgathered to obtain the complete sequence, and the attention score for the row split and the output Y of the average sequence split are calculated.

[0196] Optionally, common communication operations for CE-SP include alltoall, etc. Common communication operations for ME-SP include allgather, ring, etc.

[0197] In both `alltoall` and `allgather`, each node sends messages to any other node, and each node receives messages from any other node. The difference between `alltoall` and `allgather` is that in `alltoall`, each node splits the data and sends each piece of data separately to each other. In `allgather`, each node sends the complete data to all nodes. Alternatively, for each computing card, in `alltoall`, before communication, the card's local data is split into multiple sub-data, and the corresponding sub-data is sent according to the position or order of other computing cards in the communication group. The card then receives the corresponding sub-data from other computing cards. In `allgather`, before communication, data is sent to other computing cards, and the card receives the corresponding data from other computing cards.

[0198] For example, taking a computing cluster consisting of XPU1, XPU2, XPU3, and XPU4 arranged in sequence, an example of all-to-all is shown in Figure 1B. Before all-to-all, XPU1 stores data A1, A2, A3, and A4; XPU2 stores data B1, B2, B3, and B4; XPU3 stores data C1, C2, C3, and C4; and XPU4 stores data D1, D2, D3, and D4. After all-to-all, XPU1 stores data A1, B1, C1, and D1; XPU2 stores data A2, B2, C2, and D2; XPU3 stores data A3, B3, C3, and D3; and XPU4 stores data A4, B4, C4, and D4. Alternatively, it can be understood that each XPU divides its local data into four sub-data sets. XPU1 divides its local data into four sub-data segments (A1, A2, A3, and A4), and sends the second sub-data segment (A2) to the second computing card (XPU2), the third sub-data segment (A3) to the third computing card (XPU3), and the fourth sub-data segment (A4) to the fourth computing card (XPU4). Correspondingly, XPU1 receives the first sub-data segment A2 sent by XPU2, receives the first sub-data segment A3 from the local data of the third computing card sent by XPU3, and receives the first sub-data segment A4 sent by XPU4.

[0199] For example, taking a computing cluster consisting of XPU1, XPU2, XPU3, and XPU4 arranged in sequence, one example of an allgather is shown in Figure 1C. Before the allgather, XPU1 stores data A, XPU2 stores data B, XPU3 stores data C, and XPU4 stores data D. After the allgather, XPU1 stores data A, B, C, and D; XPU2 stores data A, B, C, and D; XPU3 stores data A, B, C, and D; and XPU4 stores data A, B, C, and D. Alternatively, it can be understood that each XPU sends its local data to other XPUs. XPU1 sends local data A to other XPUs and receives data B sent by XPU2, data C sent by XPU3, and data D sent by XPU4.

[0200] For ease of description, the Model Parallelism (MP) strategy will be referred to as the first parallel strategy, and the communication operations involved in the Model Parallelism strategy will be referred to as the first communication operation. The Communication Efficient Sequential Parallelism (CE-SP) strategy will be referred to as the second parallel strategy, and the communication operations involved in the Communication Efficient Sequential Parallelism strategy will be referred to as the second communication operation. The Memory Efficient Sequential Parallelism (ME-SP) strategy will be referred to as the third parallel strategy, and the communication operations involved in the Memory Efficient Sequential Parallelism strategy will be referred to as the third communication operation.

[0201] 6. Row cutting and column cutting

[0202] In this application, row cutting can be understood as dividing the rows of a matrix, typically by an average division. Similarly, column cutting can be understood as dividing the columns of a matrix, also typically by an average division.

[0203] In this context, the "Pth part" in the matrix after row partitioning can refer to either the Pth part from top to bottom or the Pth part from bottom to top. Similarly, the "Qth part" in the matrix after column partitioning can refer to either the Qth part from left to right or the Qth part from right to left.

[0204] For consistency, the Pth part after row partitioning will be understood as the Pth part of the matrix from top to bottom after row partitioning. The Qth part after column partitioning will be understood as the Qth part of the matrix from left to right after column partitioning.

[0205] Training long sequences is a challenge in the field of NLP, involving the processing and generation of long text sequences. Processing long sequences requires significant memory and computational resources. In Transformer architecture models, the memory requirements of the self-attention layer are proportional to the square of the sequence length, meaning that increasing the sequence length rapidly increases the computational resource requirements.

[0206] Currently, commonly used mainstream training acceleration libraries (such as DeepSpeed, Megatron, and Colossal-AI) have proposed solutions such as model parallelism and sequence parallelism, which focus on reducing the memory usage of activation values ​​or models, so that the memory usage of activation values ​​or models can decrease linearly with the number of GPUs. However, current parallel strategies usually introduce large communication overhead, and the degree of parallelism is limited by the model structure, so they cannot be scaled to ultra-large-scale clusters.

[0207] Therefore, when using the Transformer architecture for model process data processing, improving data processing efficiency and reducing communication overhead are urgent technical problems to be solved.

[0208] To address the aforementioned technical problems, this application provides a data processing method and related equipment. Compared to existing solutions that use a single parallel strategy for data processing, the solution provided in this application introduces multiple communication operations corresponding to multiple parallel strategies. This not only improves data processing efficiency through multiple communication operations but also reduces the communication overhead introduced by each parallel strategy by limiting the execution order of QKV calculations, all-to-all operations, and parallel strategies, thereby reducing the overall communication overhead.

[0209] Before describing the data processing method and related equipment of the embodiments of this application in conjunction with the accompanying drawings, the system architecture provided in the embodiments of this application will be explained first.

[0210] Referring to Figure 2, this application embodiment provides a system architecture 200. As shown in system architecture 200, a data acquisition device 260 is used to acquire training data. The training data in this application embodiment may include at least one of the following: images, text, audio, video, etc. The training data is stored in a database 230, and a training device 220 trains a target model / rule 201 based on the training data maintained in the database 230. This target model / rule 201 can be used to implement the data processing method provided in this application embodiment. This data processing method can be applied to the following data processing tasks: classification tasks, segmentation tasks, detection tasks, or data generation tasks, etc. The target model / rule 201 in this application embodiment may specifically include self-attention networks, multilayer perceptrons, pooling layers, etc. It should be noted that in actual applications, the training data maintained in the database 230 may not all come from the acquisition by the data acquisition device 260; it may also be received from other devices. It should also be noted that the training device 220 may not necessarily train the target model / rule 201 entirely based on the training data maintained by the database 230. It may also obtain training data from the cloud or other places for model training. The above description should not be construed as a limitation on the embodiments of this application.

[0211] The target model / rule 201 in this embodiment can refer to a multi-head attention network, which includes an encoder and a decoder. Alternatively, the multi-head attention network may include a decoder, etc., without limitation here. The decoder includes at least a self-attention module and a linear module. It is understood that in practical applications, the multi-head attention network may also include other modules, such as a normalization module, a fully connected layer, an MLP, a feedforward neural network (FNN), or a feedforward network (FFN), etc., without limitation here.

[0212] Understandably, the input data may differ depending on the target model / rule 201. For example, if the target model / rule 201 includes an encoder and a decoder, the input data may include at least one of the following: the image to be processed, text, audio, video, etc. As another example, if the target model / rule 201 includes a decoder, the input data may be the sequence or word vectors corresponding to the above data.

[0213] The target model / rule 201 trained by training device 220 can be applied to different systems or devices, such as execution device 210 shown in Figure 2. Execution device 210 can be a terminal, such as a mobile terminal, tablet computer, laptop computer, augmented reality (AR) device / virtual reality (VR) device, in-vehicle terminal, etc. Of course, in addition to the aforementioned physical devices, execution device 210 can also be implemented by at least one computing instance in a virtual machine or container. When execution device 210 is implemented by a virtual machine or container, execution device 210 actually exists in the form of a cloud computing product and can provide cloud services. Similarly, training device 220 can be implemented by at least one computing instance among cloud devices, virtual machines, and containers. When training device 220 is implemented by a virtual machine or container, training device 220 actually exists in the form of a cloud computing product and can provide cloud services.

[0214] Furthermore, execution device 210 can be implemented by multiple computing instances of the same type. For example, execution device 210 can be implemented by multiple physical hosts, multiple virtual machines, or multiple containers. Similarly, training device 220 can be implemented by multiple computing instances of the same type. For example, training device 220 can be implemented by multiple physical hosts, multiple virtual machines, or multiple containers.

[0215] For example, training device 220 includes multiple computing nodes, which can be divided into different communication groups according to multiple parallel strategies. Within each communication group, communication operations according to the corresponding parallel strategy are executed, thereby enabling multiple computing nodes to train the model in parallel. Similarly, execution device 210 includes multiple computing nodes, which can be divided into different communication groups according to multiple parallel strategies. Within each communication group, communication operations according to the corresponding parallel strategy are executed, thereby enabling multiple computing nodes to process data in parallel through the model.

[0216] It should be noted that, in both execution device 210 and training device 220, multiple compute instances can be distributed within the same region or in different regions. Furthermore, multiple compute instances can be distributed within the same availability zone (AZ) or in different AZs, with each AZ comprising one or more geographically proximate data centers. Typically, a region can include multiple AZs.

[0217] Similarly, multiple compute instances can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0218] In Figure 2, the execution device 210 is configured with an I / O interface 212 for data interaction with external devices. The user can input data to the I / O interface 212 through the client device 240. The input data can be entered by the user, uploaded by the user through the camera device, or it can come from the database. The specifics are not limited here.

[0219] The preprocessing module 213 is used to preprocess the input data received by the I / O interface 212. In this embodiment, the preprocessing module 213 can be used to split the input data into sub-data sets. For example, if the input data is an image, the preprocessing module 213 can split the image into multiple image blocks.

[0220] During the preprocessing of input data by the execution device 210, or during the calculation module 211 of the execution device 210 performing calculations and other related processes, the execution device 210 can call data, code, etc. in the data storage system 250 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 250.

[0221] Finally, I / O interface 212 returns the processing result, such as the result corresponding to the target task mentioned above, to client device 240, thereby providing it to the user.

[0222] It is worth noting that the training device 220 can generate corresponding target models / rules 201 based on different training data for different objectives or tasks. The corresponding target models / rules 201 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0223] In the scenario shown in Figure 2, the user can manually provide input data, which can be done through the interface provided by I / O interface 212. Alternatively, the client device 240 can automatically send input data to I / O interface 212. If user authorization is required for the client device 240 to automatically send input data, the user can set the corresponding permissions in the client device 240. The user can view the output results of the execution device 210 on the client device 240, which can be presented in various forms such as display, sound, or animation. The client device 240 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 212 as new sample data and storing them in the database 230. Alternatively, data can be collected directly from the I / O interface 212 without going through the client device 240, using the input data and output results of the input I / O interface 212 as new sample data and storing them in the database 230.

[0224] It is worth noting that Figure 2 is merely a schematic diagram of a system architecture provided in an embodiment of this application, and the positional relationships between the devices, components, modules, etc., shown in the figure do not constitute any limitation. For example, in Figure 2, the data storage system 250 is an external memory relative to the execution device 210; in other cases, the data storage system 250 can also be placed within the execution device 210. Furthermore, the functions of the various devices shown in Figure 2 can also be integrated into the same device. For example, the client device 240 and the execution device 210 can belong to the same device. As another example, the training device 220 and the execution device 210 can belong to the same device.

[0225] The following section introduces several data processing systems involved in this application.

[0226] Figure 3A is a schematic diagram of a data processing system provided in an embodiment of this application. This data processing system includes a terminal device (Figure 3A only uses a mobile phone as an example) and a cloud device. It is understood that the terminal device can be not only a mobile phone, but also a tablet computer, portable game console, PDA, laptop computer, ultra-mobile personal computer (UMPC), handheld computer, netbook, in-vehicle media playback device, wearable electronic device, virtual reality (VR) terminal device, augmented reality (AR) terminal device, vehicle, in-vehicle terminal, aircraft terminal, intelligent robot, and other terminal devices. The terminal device is the initiator of data processing; as the initiator of data processing requests, requests are typically initiated by the user through the terminal device.

[0227] The aforementioned cloud devices can be cloud servers, network servers, application servers, management servers, or other devices or servers with data processing capabilities. Cloud devices receive data processing requests from terminal devices through interactive interfaces, and then perform data processing such as machine learning, deep learning, search, data processing, and decision-making through storage devices and data processing processors. The storage device in the cloud device can be a general term, including local storage and a database storing historical data. The database can be located on the cloud device or on other network servers.

[0228] In the data processing system shown in Figure 3A, the terminal device can receive user instructions. For example, the terminal device can obtain the user-input / selected data to be processed, and then send a processing request to the cloud device, so that the cloud device can perform data processing on the data to be processed obtained by the terminal device, thereby obtaining the result corresponding to the data to be processed.

[0229] For example, the cloud device is a computing cluster, which includes multiple computing nodes. The terminal device can obtain user-inputted data to be processed, and then send a request to the computing cluster. The cluster then splits the data into multiple sub-data items and distributes these sub-data items to multiple computing nodes. These sub-data items are then processed in parallel by the computing nodes to obtain multiple results corresponding to each sub-data item. Finally, the computing cluster calculates the final result of the processed data based on the multiple results and sends the final result back to the terminal device. The terminal device then presents this result for the user to view and use.

[0230] It is understandable that the process of splitting the data to be processed, obtaining multiple results, and obtaining the final result can be executed by the terminal device, the cloud device, or both the terminal device and the cloud device. No specific restrictions are made here.

[0231] In Figure 3A, the cloud device can execute the data processing method of the embodiments of this application. In this case, the cloud device can be the training device 220 in Figure 2 above, and the terminal device can be the execution device 210 or the client device 240 in Figure 2 above.

[0232] Figure 3B is another structural schematic diagram of the data processing system provided in the embodiment of this application. In Figure 3B, the terminal device (in Figure 3B, only the terminal device is a mobile phone as an example) is directly used as a data processing device. That is, the terminal device can directly obtain the data to be processed and process it directly by the hardware of the terminal device itself. The specific process is similar to that in Figure 3A, and can be referred to the above description, which will not be repeated here.

[0233] Optionally, in the data processing system shown in Figure 3B, the terminal device can receive user instructions. For example, the terminal device can obtain the data to be processed input by the user in the terminal device, and then the terminal device itself performs data processing on the data to be processed, thereby obtaining the result corresponding to the data to be processed, and presenting the result for the user to view and use.

[0234] For example, the terminal device is a computing cluster, which includes multiple computing nodes. The terminal device can acquire user-input data to be processed, then split the data into multiple sub-data, and distribute the corresponding sub-data to the multiple computing nodes. Thus, the multiple computing nodes process their respective sub-data in parallel to obtain multiple results corresponding to the sub-data. Finally, the computing cluster calculates the final result of the data to be processed based on the multiple results and presents the result for the user to view and use.

[0235] In Figure 3B, the terminal device itself can execute the data processing method of the embodiments of this application.

[0236] The terminal devices in Figures 3A and 3B can be the client device 240 or the execution device 210 in Figure 2. The data processing device in Figure 3A can be the execution device 210 in Figure 2. The data storage system 250 can store the data to be processed by the execution device 210. The data storage system 250 can be integrated on the execution device 210 or set up on the cloud or other network servers.

[0237] The processors in Figures 3A and 3B can perform data training / machine learning / deep learning through neural network models, and use the trained or learned models to perform data processing on the problem, thereby obtaining the corresponding results.

[0238] It is understandable that data processing devices (such as terminal devices and / or cloud devices) can be implemented not only as physical devices as mentioned above, but also as at least one computing instance in a virtual machine or container. When a data processing device is implemented by a virtual machine or container, it actually exists in the form of a cloud computing product, capable of providing cloud services. Furthermore, a data processing device can be implemented by multiple computing instances of the same type. For example, a data processing device can be implemented by multiple physical hosts, or by multiple virtual machines, or by multiple containers.

[0239] It should be noted that multiple compute instances can be distributed within the same region or across different regions. Furthermore, multiple compute instances can be distributed within the same Availability Zone (AZ) or across different AZs, with each AZ comprising one or more geographically proximate data centers. Typically, a region can include multiple AZs.

[0240] Similarly, multiple compute instances can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to achieve interconnection between VPCs.

[0241] For example, the data processing device is a computing cluster, which includes multiple computing nodes. The data processing device acquires the data to be processed, then splits the data into multiple sub-data, and distributes the corresponding sub-data to the multiple computing nodes. Thus, the multiple computing nodes process their respective sub-data in parallel to obtain multiple results corresponding to the sub-data. Finally, the computing cluster calculates the final result of the data to be processed based on the multiple results and presents the result for the user to view and use.

[0242] For example, an example of a computing cluster is shown in Figure 4. The computing cluster includes aggregation switching device 1, aggregation switching device 2, access switching device 1, access switching device 2, access switching device 3, access switching device 4, access switching device 5, access switching device 6, computing card 1, computing card 2, computing card 3, computing card 4, computing card 5, computing card 6, computing card 7, and computing card 8.

[0243] In this computing cluster, computing cards 1 through 4 are each connected to any one of the access switching devices 1, 2, and 3. Computing cards 5 through 8 are each connected to any one of the access switching devices 4, 5, and 6. Access switching devices 1 through 6 are each connected to either aggregation switching device 1 or aggregation switching device 2.

[0244] Furthermore, computing node 1 includes: access switching devices 1 to 3, and computing cards 1 to 4. Computing node 2 includes: access switching devices 4 to 6, and computing cards 5 to 8.

[0245] It should be noted that the number of computing cards, access switching devices, and aggregation switching devices in Figure 4 is just an example. In actual applications, computing clusters can include a greater number of computing cards, access switching devices, and aggregation switching devices.

[0246] In this application, the computing card in the computing node may include at least one of the following: a neural network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU), etc., for executing communication operations corresponding to parallel strategies. For ease of description, this application only uses an XPU as an example for illustrative description.

[0247] The data processing method provided in this application is applied in a computing cluster deploying a multi-head attention network, enabling parallel data processing of the multi-head attention network. Specifically, the computing cluster includes N computing cards, where N is a positive integer greater than 2. These N computing cards are used to execute multiple parallel strategies, thereby achieving parallel training or data processing through the multi-head attention network.

[0248] The multi-head attention network includes self-attention modules and linear modules. For example, the multi-head attention network is used to perform at least one of the following tasks: data analysis, data generation, data format conversion, data recognition, data classification, etc. Data may include at least one of the following: text data, image data, audio data, or video data, etc., without specific limitations here. It is understood that the data processing procedures of the multi-head attention network involved in this application can be applied to at least one of the following processes: network inference process, network training process, etc., without specific limitations here.

[0249] The multiple parallel strategies in this application include at least two of the following: a first parallel strategy, a second parallel strategy, and a third parallel strategy. The first parallel strategy can also be called a model parallel strategy, the second parallel strategy can also be called a communication-efficient sequence parallel strategy, and the third parallel strategy can also be called a memory / video memory-efficient sequence parallel strategy. That is, the multiple parallel strategies include at least two of the following: a model parallel strategy, a communication-efficient sequence parallel strategy, and a memory-efficient sequence parallel strategy.

[0250] Furthermore, depending on the specific circumstances of N computing cards executing multiple parallel strategies, the steps provided in this application can take various forms, which are described below.

[0251] In the first scenario, the multiple parallel strategies include at least a second and a third parallel strategy. That is, the multiple parallel strategies include at least: a communication-efficient sequential parallel strategy and a memory-efficient sequential parallel strategy.

[0252] Please refer to Figure 5, which is a flowchart illustrating the first scenario of the data processing method provided in this application. This method can be executed by the cloud device in Figure 3A (i.e., the data processing device includes multiple cloud devices, each including at least one computing card), or by the terminal device in Figure 3B (i.e., the data processing device includes multiple terminal devices, each including at least one computing card), or jointly by the terminal device and the cloud device in Figure 3A (i.e., the data processing device includes both terminal devices and cloud devices). This data processing method includes, but is not limited to, steps 501 to 503.

[0253] Step 501: The first calculation card performs QKV calculation.

[0254] Step 502: The first calculation card performs an alltoall operation based on the result of the QKV calculation.

[0255] Step 503: The first computing card performs parallel operations based on the results of the all-to-all operation. The parallel operations are used to obtain the data processing results of the multi-head attention network. The parallel operations are either all-gather operations or ring operations.

[0256] Furthermore, steps 501 to 503 in the embodiment shown in Figure 5 can have various forms depending on the parallel strategy or the number of communication operations, which will be described below:

[0257] For ease of understanding, the most comprehensive embodiment will be described first, followed by descriptions of other embodiments.

[0258] 1. Multiple parallel strategies also include a first parallel strategy, that is, multiple parallel strategies include a first parallel strategy, a second parallel strategy, and a third parallel strategy. For example, multiple parallel strategies include: a model parallel strategy, a communication-efficient sequence parallel strategy, and a memory-efficient sequence parallel strategy.

[0259] In this case, N = A * B * C; A represents the parallelism of the first parallel strategy, B represents the parallelism of the second parallel strategy, and C represents the parallelism of the third parallel strategy. A * B is related to the number of attention heads in the multi-head attention network. The first parallel strategy has B * C first-group computation cards, the second parallel strategy has A * C second-group computation cards, and the third parallel strategy has A * B third-group computation cards. A, B, and C are positive integers greater than 1. The first computation card is any one of the N computation cards. The first computation card is ranked Xth in its first group, Yth in its second group, and Zth in its third group. X is greater than 0 and less than or equal to A; Y is greater than 0 and less than or equal to B; Z is greater than 0 and less than or equal to C. For example, A * B * T is the number of attention heads in the multi-head attention network, and T is a positive integer greater than or equal to 1.

[0260] Please refer to Figure 6A, which is a flowchart illustrating a data processing method in the first scenario provided in this application. This method can be executed by the cloud device in Figure 3A (i.e., the data processing device includes multiple cloud devices, each including at least one computing card), or by the terminal device in Figure 3B (i.e., the data processing device includes multiple terminal devices, each including at least one computing card), or jointly by the terminal device and the cloud device in Figure 3A (i.e., the data processing device includes both terminal devices and cloud devices). This data processing method includes, but is not limited to, steps 601 to 604.

[0261] Step 601: The first calculation card performs QKV calculation based on the first submatrix and the second submatrix to obtain the third submatrix.

[0262] The first calculation card first obtains the first submatrix and the second submatrix, and then performs QKV calculation based on the first submatrix and the second submatrix to obtain the third submatrix.

[0263] In this application, the first computing card can obtain the first sub-matrix in various ways. For example, in the case of a multi-head attention network including an encoder and a decoder, it can be obtained by splitting the feature matrix obtained after inputting input data into the encoder. Alternatively, it can be obtained by receiving data from other devices (e.g., devices responsible for data distribution in a computing cluster). Another example is selection from a database, etc., and the specific method is not limited here.

[0264] The first submatrix is ​​the B*(Z-1)+Yth submatrix from top to bottom after the feature matrix is ​​divided into B*C equal rows; the feature matrix is ​​the feature matrix corresponding to the input data of the multi-head attention network; the second submatrix is ​​the Xth submatrix from left to right after the complete weight matrix is ​​divided into A equal columns; the third submatrix is ​​the B*(Z-1)+Yth submatrix from top to bottom after the rows of the complete QKV matrix are divided into B*C equal rows; the third submatrix is ​​the Xth submatrix from left to right after the columns of the complete QKV matrix are divided into A equal columns; the complete QKV matrix is ​​related to the feature matrix and the complete weight matrix.

[0265] This step can also be understood as the calculation process of QKV. Furthermore, the complete weight matrix mentioned above includes the complete Wq matrix, complete Wk matrix, and complete Wv matrix corresponding to the input data. Correspondingly, the complete QKV matrix includes the complete Q matrix, complete K matrix, and complete V matrix. In addition, the complete QKV matrix is ​​related to the feature matrix and the complete weight matrix; it can be understood that the complete QKV matrix is ​​obtained by multiplying the feature matrix and the complete weight matrix.

[0266] To make it easier to understand, the following example illustrates this step:

[0267] For example, the process of this step can be shown in Figure 6B, with A=4, B=3, C=2, X=Y=Z=1 (that is, the first computing card is ranked first in the first group of computing cards, the first computing card is ranked first in the second group of computing cards, and the first computing card is ranked first in the third group of computing cards), the first communication operation corresponding to the first parallel strategy is allreduce, the second communication operation corresponding to the second parallel strategy is alltoall, the third communication operation corresponding to the third parallel strategy is allgather / ring, the feature matrix is ​​6 rows and 12 columns (i.e., 6×12), and the complete weight matrix is ​​12 rows and 12 columns (i.e., 12×12) as an example.

[0268] The feature matrix is ​​a 6×12 matrix, the complete weight matrix is ​​a 12×12 matrix, and the complete QKV matrix is ​​a 6×12 matrix. The first submatrix is ​​the B*(Z-1)+Yth submatrix from top to bottom after the feature matrix is ​​divided into B*C equal rows (i.e., the first submatrix from top to bottom after 6 divisions), which is a 1×12 matrix. The second submatrix is ​​the Xth submatrix from left to right after the complete weight matrix is ​​divided into A equal columns (i.e., the first submatrix from left to right after 4 divisions), which is a 12×3 matrix. The third submatrix is ​​the B*(Z-1)+Yth column from top to bottom after the rows of the complete QKV matrix are divided into B*C parts (i.e., the first column from top to bottom after 6 parts). The columns of the third submatrix are the Xth column from left to right after the columns of the complete QKV matrix are divided into A parts (i.e., the first column from left to right after 4 parts). In other words, the third submatrix is ​​a 1×3 matrix.

[0269] Step 602: Within the second group of computing cards to which the first computing card belongs, the first computing card obtains the fourth submatrix based on the second communication operation involved in the third submatrix and the second parallel strategy.

[0270] After the first computing card obtains the third submatrix, within the second group of computing cards to which the first computing card belongs, the first computing card obtains the fourth submatrix based on the second communication operation involving the third submatrix and the second parallel strategy.

[0271] The columns of the fourth submatrix are the (X-1)*B+Yth parts from left to right after the columns of the complete QKV matrix are divided into A*B parts, and the rows of the fourth submatrix are the Zth parts from top to bottom after the rows of the complete QKV matrix are divided into C parts.

[0272] This step can also be understood as follows: before the second communication operation, parallel processing is performed by cutting rows; after the second communication operation, parallel processing is performed by cutting columns.

[0273] For example, continuing the above example, the process of this step can be as shown in Figure 7. The third submatrix is ​​a 1×3 matrix. The first computing card obtains the fourth submatrix within its second set of computing cards based on the second communication operation involving the third submatrix and the second parallel strategy. The columns of the fourth submatrix are the (X-1)*B+Yth parts from left to right after the columns of the complete QKV matrix are divided into A*B parts (i.e., the first part from left to right after 12 parts). The rows of the fourth submatrix are the Zth parts from top to bottom after the rows of the complete QKV matrix are divided into C parts (i.e., the first part from top to bottom after 2 parts). That is, the fourth submatrix is ​​a 3×1 matrix.

[0274] Step 603: Within the third group of computing cards to which the first computing card belongs, the first computing card obtains the first output matrix from the attention module based on the third communication operation involved in the fourth sub-matrix and the third parallel strategy.

[0275] After the first computing card obtains the fourth submatrix, the first output matrix from the attention module is obtained in the third group of computing cards to which the first computing card belongs, based on the fourth submatrix and the third communication operation involved in the third parallel strategy.

[0276] The multi-head attention network includes a self-attention module and a linear module. The first output matrix has the same number of rows and columns as the fourth sub-matrix, but their contents are different.

[0277] This step has several variations depending on the third parallel strategy.

[0278] In one possible implementation, within the third group of computing cards to which the first computing card belongs, the first computing card performs a third communication operation based on the KV matrix in the fourth submatrix to obtain a fifth submatrix. The fifth submatrix is ​​the (X-1)*B+Yth part from left to right of the KV matrix in the complete QKV matrix after it has been divided into A*B parts by average columns. The first computing card performs attention calculations based on the Q matrix in the fourth submatrix and the fifth submatrix to obtain a first output matrix.

[0279] In this approach, the third communication operation corresponding to the third parallel strategy can refer to allgather. This step in this approach can also be understood as performing an overall attention calculation after passing the fourth submatrix from each computing card.

[0280] For example, continuing with the previous example of a 3×1 fourth submatrix, this method can be illustrated as shown in Figure 8. The first computing card performs a third communication operation with other computing cards in its third group to obtain the fifth submatrix from the KV matrix in the fourth submatrix. The fifth submatrix is ​​the (X-1)*B+Yth part from left to right (i.e., the first part from left to right after 12 parts) of the KV matrix in the complete QKV matrix, which has been divided into A*B parts by average columns. That is, the fifth submatrix is ​​a 6×1 matrix. After obtaining the fourth and fifth submatrixes, the first computing card performs attention calculations on the Q matrix in the fourth submatrix and the KV matrix in the fifth submatrix to obtain the first output matrix. The first output matrix has the same number of rows and columns as the fourth submatrix, but different content. That is, the first output matrix is ​​a 3×1 matrix.

[0281] In another possible implementation, the first computing card performs attention calculation on the third output matrix based on the fourth submatrix. Within the third group of computing cards to which the first computing card belongs, the first computing card performs a third communication operation with the second computing card based on the fourth submatrix to obtain a sixth submatrix. The second computing card is located within the third group of computing cards to which the first computing card belongs. The first computing card performs attention calculation on the fourth output matrix based on the sixth submatrix. The first computing card obtains the first output matrix based on the third and fourth output matrices.

[0282] Understandably, if the third set of computing cards includes two computing cards, the first computing card obtains the first output matrix based on the third and fourth output matrices. Specifically, this can be achieved by accumulating the third and fourth output matrices to obtain the first output matrix. If the third set of computing cards includes more than two computing cards, the first computing card cyclically executes the following processes: calculating the third output matrix, performing the third communication operation, and calculating the fourth output matrix, until the first output matrix is ​​obtained through multiple accumulations.

[0283] In this approach, the third communication operation corresponding to the third parallel strategy can refer to the ring. This step in this approach can also be understood as calculating attention once, passing it once, and accumulating it once.

[0284] For example, continuing with the example of the fourth submatrix being 3×1, this method can be as shown in Figure 9. The first computing card performs attention calculation on the 3×1 fourth submatrix to obtain a 3×1 third output matrix. The first computing card obtains a 3×1 sixth submatrix according to the third communication operation, and then performs attention calculation on the 3×1 sixth submatrix to obtain a 3×1 fourth output matrix. The 3×1 third output matrix and the 3×1 fourth output matrix are then added together to obtain a 3×1 intermediate matrix. This process of attention calculation, third communication operation, and accumulation is repeated multiple times to obtain a 3×1 first output matrix.

[0285] For example, taking the third group of computing cards, which includes five computing cards, the method shown in Figure 8 involves the first computing card directly obtaining partial data from the four fourth sub-matrices of the other four computing cards through a third communication operation to obtain the fifth sub-matrix. The method shown in Figure 9 involves the first computing card performing attention calculations based on its local fourth sub-matrix to obtain the third output matrix. After performing a third communication operation with one of the computing cards, the first computing card obtains the sixth sub-matrix, and then performs attention calculations based on the sixth sub-matrix to obtain the fourth output matrix. Then, the third and fourth output matrices are summed to obtain an intermediate output matrix. This process is repeated four times, and the first output matrix is ​​obtained after four summations.

[0286] Step 604: The first computing card obtains the second output matrix of the linear module based on the first output matrix, the second communication operation, and the first communication operation involved in the first parallel strategy.

[0287] After the first computing card obtains the first output matrix, it obtains the second output matrix of the linear module based on the first output matrix, the second communication operation, and the first communication operation involved in the first parallel strategy.

[0288] The second output matrix has the same number of rows and columns as the first sub-matrix, but different content. The second output matrix is ​​used to obtain the data processing result obtained by the input data through the multi-head attention network.

[0289] Optionally, within the second group of computing cards to which the first computing card belongs, the first computing card performs a second communication operation based on the first output matrix to obtain a seventh submatrix. The first computing card obtains an eighth submatrix based on the mapping matrix between the seventh submatrix and the average row-cut matrix. Within the first group of computing cards to which the first computing card belongs, the first computing card performs a first communication operation based on the eighth submatrix to obtain a second output matrix.

[0290] The seventh submatrix has the same number of rows as the B*(Z-1)+Yth row segment from top to bottom after the rows of the complete QKV matrix are divided into B*C parts, but its content is different. The seventh submatrix has the same number of columns as the Xth column segment from left to right after the columns of the complete QKV matrix are divided into A parts, but its content is different. The eighth submatrix has the same number of rows and columns as the first submatrix, but its content is different, and the eighth submatrix also has the same number of rows and columns as the second output matrix, but its content is different. The mapping matrix has the same number of rows as the complete weight matrix, and the mapping matrix has the same number of columns as the complete weight matrix. Alternatively, it can be understood that the mapping matrix, after being averaged by rows, is the Xth segment from top to bottom after the complete mapping matrix has been averaged by rows into A parts.

[0291] For example, continuing the above example, as shown in Figure 10, after the first computing card obtains the 3×1 first output matrix, a second communication operation is performed in the second set of computing cards to obtain a 1×3 seventh submatrix, and then a 1×12 eighth submatrix is ​​obtained through a linear module (e.g., multiplied by a mapping matrix that is averaged by row slicing). Finally, a first communication operation is performed in the first set of computing cards to obtain a 1×12 second output matrix.

[0292] In this application, by introducing multiple parallel strategies and defining the execution order between them (first parallel strategy starts, second parallel strategy starts, third parallel strategy starts and ends, second parallel strategy ends, first parallel strategy ends), the communication overhead introduced by each parallel strategy can be reduced, thereby reducing the total communication overhead and improving data processing efficiency. Furthermore, the scheme provided in this application can also achieve communication computation masking. For example, the second communication operation of the Q matrix can mask the computation of the K and V matrices; the second and third communication operations of the K matrix can mask the computation of the V matrix; and the second and third communication operations of the V matrix can mask the computation of the attention score (i.e., the process of obtaining the second output matrix), thereby increasing throughput. In addition, compared to existing two-dimensional model parallel schemes (i.e., the input sequence cannot be segmented, and the length of the input sequence is limited), this application, through multi-dimensional parallelism, can not only achieve sequence segmentation but also model parallelism, thereby improving the data processing efficiency of multi-head attention networks.

[0293] To facilitate understanding of multiple parallel strategies, including the first, second, and third parallel strategies, the following example illustrates the first case: the first parallel strategy is a model-parallel strategy with the first communication operation being allreduce; the second parallel strategy is CE-SP with the second communication operation being alltoall; and the third parallel strategy is ME-SP with the third communication operation being allgather.

[0294] For example, consider a cluster with A = B = C = 4 and N = 64. This means the cluster consists of 64 computing cards, evenly distributed across the XYZ space as shown in Figure 11.

[0295] This example can also be understood as an example of a 3D sequence parallelism scheme. Assume the multi-head attention network has 16 attention heads, meaning the total sequence parallelism is 16 (of which CE-SP has 4 sequence parallelism and ME-SP has 4 sequence parallelism). The model parallelism is 4.

[0296] The data processing method for the first case includes the following steps:

[0297] Step 1: The input data is divided into 16 parts in the sequence dimension and placed on 16 XPUs in the XZ plane in Figure 11. The input data on the 4 XPUs in the Y axis is the same. Each part of the data is matrix multiplied with the 4 parts Wq, Wk, Wv in the Y axis to obtain Q, K, V, where the head is divided into 1 / 4 and the sequence is divided into 1 / 16.

[0298] Step 2: Q, K, V are all-to-all executed on the CE-SP sequence parallel group, i.e., on the 4 cards of the Z-axis, to obtain 1 head and the sequence is divided into 1 / 4 Q, K, V.

[0299] Step 3: Perform allgather on K and V within the ME-SP sequence parallel group, i.e., on the 4 cards of the X-axis, to obtain a K and V sequence with 1 head and complete sequence.

[0300] Step 4: Multiply the Q matrix (which is divided into 1 / 4 segments) with the complete K matrix of the sequence to obtain the attention score with an average row division of 1 / 4.

[0301] Step 5: Calculate the softmax of the attention score after the row is divided into 1 / 4, and multiply it with the complete V matrix of the sequence to obtain the output after the sequence is divided into 1 / 4.

[0302] Step 6: The output of the sequence divided into 1 / 4 is executed alltoall in the CE-SP sequence parallel group, that is, on the 4 cards of the Z-axis, to obtain 4 heads and the output of the sequence divided into 1 / 16.

[0303] Step 7: Multiply the output of the sequence divided into 1 / 16 and column divided into 1 / 4 by the mapping matrix of the average row divided into 1 / 4 to obtain the partial output. Execute allreduce within the model parallel group, i.e., on the 4 cards of the Y-axis, to obtain the output of the sequence divided into 1 / 16 (i.e., the second output matrix).

[0304] In this process, computational and communication operations that do not have dependencies can mask each other. For example, Q's alltoall operation can mask the computations of K and V; similarly, K's alltoall and allgather operations can mask the computations of V; and V's alltoall and allgather operations can mask the computations in steps 4 and 5, thereby increasing throughput.

[0305] 2. The multiple parallel strategies do not include the first parallel strategy; that is, the multiple parallel strategies include the second and third parallel strategies. Specifically, the multiple parallel strategies include: a communication-efficient sequential parallel strategy and a memory-efficient sequential parallel strategy.

[0306] In this case, N = B * C; B represents the parallelism of the second parallel strategy, and C represents the parallelism of the third parallel strategy. B is related to the number of attention heads in the multi-head attention network. The second parallel strategy has C second-group computation cards in parallel, and the third parallel strategy has B third-group computation cards in parallel. B and C are positive integers greater than 1. The first computation card is any one of the N computation cards. The first computation card is ranked Y-th in its second-group and Z-th in its third-group. Y is greater than 0 and less than or equal to B; Z is greater than 0 and less than or equal to C. For example, B * T is the number of attention heads in the multi-head attention network, and T is a positive integer greater than or equal to 1.

[0307] Please refer to Figure 12, which is another flowchart illustrating the data processing method in the first scenario provided in this application. This method can be executed by the cloud device in Figure 3A (i.e., the data processing device includes multiple cloud devices, each including at least one computing card), or by the terminal device in Figure 3B (i.e., the data processing device includes multiple terminal devices, each including at least one computing card), or jointly by the terminal device and the cloud device in Figure 3A (i.e., the data processing device includes both terminal devices and cloud devices). This data processing method includes, but is not limited to, steps 1201 to 1204.

[0308] Step 1201: The first calculation card performs QKV calculation based on the first submatrix and the complete weight matrix to obtain the third submatrix.

[0309] The first calculation card first obtains the first submatrix and the complete weight matrix, and then performs QKV calculation based on the first submatrix and the complete weight matrix to obtain the third submatrix.

[0310] In this application, the first computing card can obtain the first sub-matrix in various ways. For example, in the case of a multi-head attention network including an encoder and a decoder, it can be obtained by splitting the feature matrix obtained after inputting input data into the encoder. Alternatively, it can be obtained by receiving data from other devices (e.g., devices responsible for data distribution in a computing cluster). Another example is selection from a database, etc., and the specific method is not limited here.

[0311] The first submatrix is ​​the B*(Z-1)+Yth submatrix from top to bottom after the feature matrix has been divided into B*C rows on average; the feature matrix is ​​the feature matrix corresponding to the input data of the multi-head attention network. The third submatrix is ​​the B*(Z-1)+Yth submatrix from top to bottom after the rows of the complete QKV matrix have been divided into B*C rows. The columns of the third submatrix are the columns of the complete QKV matrix. The complete QKV matrix is ​​related to the feature matrix and the complete weight matrix.

[0312] This step can also be understood as the calculation process of QKV. Furthermore, the complete weight matrix mentioned above includes the complete Wq matrix, complete Wk matrix, and complete Wv matrix corresponding to the input data. Correspondingly, the complete QKV matrix includes the complete Q matrix, complete K matrix, and complete V matrix. In addition, the complete QKV matrix is ​​related to the feature matrix and the complete weight matrix; it can be understood that the complete QKV matrix is ​​obtained by multiplying the feature matrix and the complete weight matrix.

[0313] To make it easier to understand, the following example illustrates this step:

[0314] For example, the process of this step can be shown in Figure 13, with B=3, C=2, Y=Z=1 (that is, the first computing card is ranked first in the second group of computing cards and the first computing card is ranked first in the third group of computing cards), the second communication operation corresponding to the second parallel strategy is alltoall, the third communication operation corresponding to the third parallel strategy is allgather / ring, the feature matrix is ​​6 rows and 12 columns (i.e., 6×12), and the complete weight matrix is ​​12 rows and 12 columns (i.e., 12×12).

[0315] The feature matrix is ​​a 6×12 matrix, the complete weight matrix is ​​a 12×12 matrix, and the complete QKV matrix is ​​a 6×12 matrix. The first submatrix is ​​the B*(Z-1)+Yth submatrix from top to bottom after the feature matrix has been divided into B*C equal rows (i.e., the first of 6 parts), meaning the first submatrix is ​​a 1×12 matrix. The third submatrix is ​​the B*(Z-1)+Yth submatrix from top to bottom after the rows of the complete QKV matrix have been divided into B*C equal rows (i.e., the first of 6 parts), and the columns of the third submatrix are the same as the columns of the complete QKV matrix, meaning the third submatrix is ​​a 1×12 matrix.

[0316] Step 1202: Within the second group of computing cards to which the first computing card belongs, the first computing card obtains the fourth submatrix based on the second communication operation involved in the third submatrix and the second parallel strategy.

[0317] After the first computing card obtains the third submatrix, within the second group of computing cards to which the first computing card belongs, the first computing card uses the third submatrix and the second communication operation to obtain the fourth submatrix.

[0318] The columns of the fourth submatrix are the Yth columns from left to right after the columns of the complete QKV matrix are divided into B parts, and the rows of the fourth submatrix are the Zth rows from top to bottom after the rows of the complete QKV matrix are divided into C parts.

[0319] This step can also be understood as follows: before the second communication operation, parallel processing is performed by cutting rows; after the second communication operation, parallel processing is performed by cutting columns.

[0320] For example, continuing the above example, the process of this step can be as shown in Figure 14. The third submatrix is ​​a 1×12 matrix. The first computing card obtains the fourth submatrix based on the third submatrix and the second communication operation within its corresponding second computing card. The columns of the fourth submatrix are the Yth part from left to right (i.e., the first of 3 parts) after the columns of the complete QKV matrix are divided into B parts. The rows of the fourth submatrix are the Zth part from top to bottom (i.e., the first of 2 parts) after the rows of the complete QKV matrix are divided into C parts. Therefore, the fourth submatrix is ​​a 3×4 matrix.

[0321] Step 1203: Within the third group of computing cards to which the first computing card belongs, the first computing card obtains the first output matrix from the attention module based on the third communication operation involved in the fourth sub-matrix and the third parallel strategy.

[0322] After the first computing card obtains the fourth submatrix, within the third group of computing cards to which the first computing card belongs, the first computing card obtains the first output matrix from the attention module based on the fourth submatrix and the third communication operation.

[0323] The multi-head attention network includes a self-attention module and a linear module. The first output matrix has the same number of rows and columns as the fourth sub-matrix, but their contents are different.

[0324] This step has several variations depending on the third parallel strategy.

[0325] In one possible implementation, within the third group of computing cards to which the first computing card belongs, the first computing card performs a third communication operation based on the KV matrix in the fourth submatrix to obtain a fifth submatrix. The fifth submatrix is ​​the Y-th part from left to right of the KV matrix in the complete QKV matrix after it has been divided into B parts by average columns. The first computing card performs attention calculations based on the Q matrix in the fourth submatrix and the fifth submatrix to obtain a first output matrix.

[0326] In this approach, the third communication operation corresponding to the third parallel strategy can refer to allgather. This step in this approach can also be understood as performing an overall attention calculation after passing the fourth submatrix from each computing card.

[0327] For example, continuing the example of a 3×4 fourth submatrix, this method can be illustrated as shown in Figure 15. The first computing card performs a third communication operation with other computing cards in its third group to obtain the fifth submatrix from the KV matrix in the fourth submatrix. The fifth submatrix is ​​the Yth part from left to right (i.e., the first part from left to right after dividing the KV matrix in the complete QKV matrix into B parts by average column division) . That is, the fifth submatrix is ​​a 6×4 matrix. After obtaining the fourth and fifth submatrixes, the first computing card performs attention calculations on the Q matrix in the fourth submatrix and the KV matrix in the fifth submatrix to obtain the first output matrix. The first output matrix has the same number of rows and columns as the fourth submatrix, but different content. That is, the first output matrix is ​​a 3×4 matrix.

[0328] In another possible implementation, the first computing card performs attention calculation on the third output matrix based on the fourth submatrix. Within the third group of computing cards to which the first computing card belongs, the first computing card performs a third communication operation with the second computing card based on the fourth submatrix to obtain a sixth submatrix. The second computing card is located within the third group of computing cards to which the first computing card belongs. The first computing card performs attention calculation on the fourth output matrix based on the sixth submatrix. The first computing card obtains the first output matrix based on the third and fourth output matrices.

[0329] Understandably, if the third set of computing cards includes two computing cards, the first computing card obtains the first output matrix based on the third and fourth output matrices. Specifically, this can be achieved by accumulating the third and fourth output matrices to obtain the first output matrix. If the third set of computing cards includes more than two computing cards, the first computing card cyclically executes the following processes: calculating the third output matrix, performing the third communication operation, and calculating the fourth output matrix, until the first output matrix is ​​obtained through multiple accumulations.

[0330] In this approach, the third communication operation corresponding to the third parallel strategy can refer to the ring. This step in this approach can also be understood as calculating attention once, passing it once, and accumulating it once.

[0331] For example, continuing with the previous example of a 3×4 fourth submatrix, this method can be illustrated as shown in Figure 16. The first computing card performs attention calculation on the 3×4 fourth submatrix to obtain a 3×4 third output matrix. The first computing card then obtains a 3×4 sixth submatrix based on the third communication operation, and performs attention calculation on the 3×4 sixth submatrix to obtain a 3×4 fourth output matrix. The 3×4 third output matrix and the 3×4 fourth output matrix are then added together to obtain a 3×4 intermediate matrix. This process of attention calculation, third communication operation, and accumulation is repeated multiple times to obtain a 3×4 first output matrix.

[0332] Step 1204: The first computing card obtains the second output matrix of the linear module based on the first output matrix and the second communication operation.

[0333] After the first computing card obtains the first output matrix, it obtains the second output matrix of the linear module based on the first output matrix and a second communication operation.

[0334] The second output matrix has the same number of rows and columns as the first sub-matrix, but different content. The second output matrix is ​​used to obtain the data processing result obtained by the input data through the multi-head attention network.

[0335] Optionally, within the second group of computing cards to which the first computing card belongs, the first computing card performs a second communication operation based on the first output matrix to obtain the seventh submatrix. The first computing card then obtains the second output matrix based on the seventh submatrix and the mapping matrix.

[0336] The seventh submatrix has the same number of rows as the B*(Z-1)+Yth row group from top to bottom after the rows of the complete QKV matrix are divided into B*C parts, but its content is different. The seventh submatrix also has the same number of columns as the complete QKV matrix, but its content is different. The eighth submatrix has the same number of rows and columns as the first submatrix, but its content is different, and it also has the same number of rows and columns as the second output matrix, but its content is different. The mapping matrix has the same number of rows as the complete weight matrix, and the mapping matrix has the same number of columns as the complete weight matrix.

[0337] For example, continuing the above example, as shown in Figure 17, after the first computing card obtains the 3×4 first output matrix, a second communication operation is performed in the second set of computing cards to obtain the 1×12 seventh submatrix, and then the 1×12 second output matrix is ​​obtained through a linear module (e.g., multiplied by a mapping matrix).

[0338] In this application, by introducing multiple parallel strategies and limiting the execution order between them (starting with the second parallel strategy, ending with the third parallel strategy, and ending with the second parallel strategy), the communication overhead introduced by each parallel strategy can be reduced, thereby lowering the total communication overhead and improving data processing efficiency. Furthermore, the scheme provided in this application can also achieve communication computation masking. For example, the second communication operation of the Q matrix can mask the computation of the K and V matrices; the second and third communication operations of the K matrix can mask the computation of the V matrix; and the second and third communication operations of the V matrix can mask the computation of the attention score (i.e., the process of obtaining the second output matrix), thereby increasing throughput. In addition, compared to existing two-dimensional model parallel schemes (i.e., the input sequence cannot be segmented, and the length of the input sequence is limited), this application, through multi-dimensional parallelism, can not only achieve sequence segmentation but also model parallelism, thereby improving the data processing efficiency of multi-head attention networks.

[0339] To facilitate understanding of the case where multiple parallel strategies include a second parallel strategy and a third parallel strategy, the following example illustrates the second case, where the second parallel strategy is CE-SP and the second communication operation is alltoall, and the third parallel strategy is ME-SP and the third communication operation is allgather.

[0340] For example, as shown in Figure 18, with B=C=2 and N=4, the computing cluster includes four computing cards (computing card 1, computing card 2, computing card 3, and computing card 4).

[0341] This example can also be understood as an example of a 2D sequence parallelism scheme. Assume that the number of attention heads in the multi-head attention network is 2, that is, the total sequence parallelism is 4 (of which, the sequence parallelism of CE-SP is 2 and the sequence parallelism of ME-SP is 2).

[0342] The data processing method for the second case includes the following steps:

[0343] Step 1: The input data is divided into 4 parts along the sequence dimension and placed on 4 computing cards. The four parts of data are multiplied by the complete Wq, Wk, and Wv on the computing cards to obtain 2 heads and the sequence is divided into 1 / 4 Q, K, and V.

[0344] Step 2: Q, K, V are executed alltoall within the CE-SP sequence parallel group, i.e. (Computing Card 1, Computing Card 2) and (Computing Card 3, Computing Card 4), resulting in one head and the sequence being split into 1 / 2 Q, K, V. (,) represents the communication group.

[0345] Step 3: K and V are allgathered within the ME-SP sequence parallel group, i.e. (computation card 1, computation card 3) and (computation card 2, computation card 4), to obtain K and V with one head and a complete sequence.

[0346] Step 4: Multiply the Q matrix (which is half the length of the sequence) with the K matrix (which is the complete sequence matrix) to obtain the attention score with an average row split of half.

[0347] Step 5: Calculate the softmax of the attention score after the row is split into 1 / 2, and multiply it with the complete V matrix of the sequence to obtain the output after the sequence is split into 1 / 2.

[0348] Step 6: The output of the sequence divided into 1 / 2 is executed alltoall within the CE-SP sequence parallel group, i.e. (computing card 1, computing card 2) and (computing card 3, computing card 4), to obtain two heads. The output of the sequence divided into 1 / 4 is obtained.

[0349] Step 7: The output of the sequence is split into 1 / 4 and multiplied with the complete projection model matrix to obtain the attention output.

[0350] In this process, on the one hand, computational and communication operations that do not have dependencies can mask each other. For example, the all-to-all operation of Q can mask the computations of K and V, and the all-to-all and all-gather operations of K can mask the computations of V. Similarly, the all-to-all and all-gather operations of V can mask the computations in steps 4 and 5, thereby increasing throughput. On the other hand, all-to-all rearranges QKV along the head dimension, resulting in better communication. Furthermore, segmentation along the sequence dimension reduces the computational cost of attention weights on a single computing card, leading to better memory utilization.

[0351] The second scenario involves multiple parallel strategies, including a first parallel strategy and a third parallel strategy. That is, multiple parallel strategies include: a model-parallel strategy and a memory-efficient sequence parallel strategy. For example, the multiple communication operations corresponding to these parallel strategies include: parallel operations and a full reduction (allreduce) operation, where the parallel operation is either a full aggregation (allgather) operation or a ring operation.

[0352] Please refer to Figure 19A, a flowchart illustrating a second scenario of the data processing method provided in this application. This method can be executed by the cloud device in Figure 3A (i.e., the data processing device includes multiple cloud devices, each including at least one computing card), or by the terminal device in Figure 3B (i.e., the data processing device includes multiple terminal devices, each including at least one computing card), or jointly by the terminal device and the cloud device in Figure 3A (i.e., the data processing device includes both terminal devices and cloud devices). This data processing method includes, but is not limited to, steps 19011 to 19031.

[0353] Step 19011: The first calculation card performs QKV calculation;

[0354] Step 19021: The first computing card performs parallel operations based on the results of the QKV calculation;

[0355] Step 19031: The first computing card performs an allreduce operation based on the results of the parallel operation. The allreduce operation is used to obtain the data processing results of the multi-head attention network.

[0356] Furthermore, for ease of understanding, the various steps in the embodiment shown in FIG19A will be described below with reference to FIG19B to FIG23.

[0357] In this case, N = A * C; A represents the parallelism of the first parallel strategy, and C represents the parallelism of the third parallel strategy. A is related to the number of attention heads in the multi-head attention network. The first parallel strategy has C first-group computation cards in parallel, and the third parallel strategy has A third-group computation cards in parallel. A and C are positive integers greater than 1. The first computation card is any one of the N computation cards. The first computation card is ranked Xth in its first group and Zth in its third group. X is greater than 0 and less than or equal to A; Z is greater than 0 and less than or equal to C. For example, A * T is the number of attention heads in the multi-head attention network, and T is a positive integer greater than or equal to 1.

[0358] Please refer to Figure 19B, which is another flowchart illustrating the second scenario of the data processing method provided in this application. This method can be executed by the cloud device in Figure 3A (i.e., the data processing device includes multiple cloud devices, each including at least one computing card), or by the terminal device in Figure 3B (i.e., the data processing device includes multiple terminal devices, each including at least one computing card), or jointly by the terminal device and the cloud device in Figure 3A (i.e., the data processing device includes both terminal devices and cloud devices). This data processing method includes, but is not limited to, steps 19012 to 19032.

[0359] Step 19012: The first calculation card performs QKV calculation based on the first submatrix and the second submatrix to obtain the third submatrix.

[0360] The first calculation card first obtains the first submatrix and the second submatrix, and then performs QKV calculation based on the first submatrix and the second submatrix to obtain the third submatrix.

[0361] In this application, the first computing card can obtain the first sub-matrix in various ways. For example, in the case of a multi-head attention network including an encoder and a decoder, it can be obtained by splitting the feature matrix obtained after inputting input data into the encoder. Alternatively, it can be obtained by receiving data from other devices (e.g., devices responsible for data distribution in a computing cluster). Another example is selection from a database, etc., and the specific method is not limited here.

[0362] The first submatrix is ​​the Zth submatrix from top to bottom after the feature matrix is ​​divided into C equal rows; the feature matrix is ​​the feature matrix corresponding to the input data of the multi-head attention network; the second submatrix is ​​the Xth submatrix from left to right after the complete weight matrix is ​​divided into A equal columns; the third submatrix is ​​the Zth submatrix from top to bottom after the rows of the complete QKV matrix are divided into C equal rows; the third submatrix is ​​the Xth submatrix from left to right after the columns of the complete QKV matrix are divided into A equal columns; the complete QKV matrix is ​​related to the feature matrix and the complete weight matrix.

[0363] This step can also be understood as the calculation process of QKV. Furthermore, the complete weight matrix mentioned above includes the complete Wq matrix, complete Wk matrix, and complete Wv matrix corresponding to the input data. Correspondingly, the complete QKV matrix includes the complete Q matrix, complete K matrix, and complete V matrix. In addition, the complete QKV matrix is ​​related to the feature matrix and the complete weight matrix; it can be understood that the complete QKV matrix is ​​obtained by multiplying the feature matrix and the complete weight matrix.

[0364] To make it easier to understand, the following example illustrates this step:

[0365] For example, the process of this step can be shown in Figure 20, with A=4, C=2, X=Z=1 (that is, the first computing card is ranked first in the first group of computing cards and the first computing card is ranked first in the third group of computing cards), the first communication operation corresponding to the first parallel strategy is allreduce, the third communication operation corresponding to the third parallel strategy is allgather / ring, the feature matrix is ​​6 rows and 12 columns (i.e., 6×12), and the complete weight matrix is ​​12 rows and 12 columns (i.e., 12×12).

[0366] The feature matrix is ​​a 6×12 matrix, the complete weight matrix is ​​a 12×12 matrix, and the complete QKV matrix is ​​a 6×12 matrix. The first submatrix is ​​the Zth submatrix from top to bottom after the feature matrix is ​​divided into C equal rows (i.e., the first submatrix from top to bottom after 2 divisions), which is a 3×12 matrix. The second submatrix is ​​the Xth submatrix from left to right after the complete weight matrix is ​​divided into A equal columns (i.e., the first submatrix from left to right after 4 divisions), which is a 12×3 matrix. The third submatrix consists of the Zth submatrix from top to bottom after the rows of the complete QKV matrix are divided into C equal rows (i.e., the first submatrix from top to bottom after 2 divisions), and the columns of the third submatrix are the Xth submatrix from left to right after the columns of the complete QKV matrix are divided into A equal columns (i.e., the first submatrix from left to right after 4 divisions), which is a 3×3 matrix.

[0367] Step 19022: Within the third group of computing cards to which the first computing card belongs, the first computing card obtains the first output matrix from the attention module based on the third submatrix and the third communication operation involved in the third parallel strategy.

[0368] After the first computing card obtains the fourth submatrix, the first output matrix from the attention module is obtained in the third group of computing cards to which the first computing card belongs, based on the third submatrix and the third communication operation involved in the third parallel strategy.

[0369] The multi-head attention network includes a self-attention module and a linear module. The first output matrix has the same number of rows and columns as the third sub-matrix, but their contents are different.

[0370] This step has several variations depending on the third parallel strategy.

[0371] In one possible implementation, within the third group of computing cards to which the first computing card belongs, the first computing card performs a third communication operation based on the KV matrix in the third submatrix to obtain a fifth submatrix. The fifth submatrix is ​​the Xth part from left to right of the KV matrix in the complete QKV matrix after it has been divided into A parts by average columns. The first computing card performs attention calculations based on the Q matrix in the third submatrix and the fifth submatrix to obtain a first output matrix.

[0372] In this approach, the third communication operation corresponding to the third parallel strategy can refer to allgather. This step in this approach can also be understood as performing an overall attention calculation after passing the third submatrix of each computing card.

[0373] For example, continuing the example of a 3×3 third submatrix, this method can be illustrated as shown in Figure 21. The first computing card performs a third communication operation with other computing cards in its third group to obtain the fifth submatrix from the KV matrix in the third submatrix. The fifth submatrix is ​​the Xth part from left to right (i.e., the first part from left to right after dividing the KV matrix in the complete QKV matrix into A parts by average columns) (i.e., the first part from left to right after dividing into 4 parts). That is, the fifth submatrix is ​​a 6×3 matrix. After obtaining the third and fifth submatrixes, the first computing card performs attention calculations on the Q matrix in the third submatrix and the KV matrix in the fifth submatrix to obtain the first output matrix. The first output matrix has the same number of rows and columns as the third submatrix, but different content. That is, the first output matrix is ​​a 3×3 matrix.

[0374] In another possible implementation, the first computing card performs attention calculation on the third output matrix based on the third submatrix. Within the third group of computing cards to which the first computing card belongs, the first computing card performs a third communication operation with the second computing card based on the third submatrix to obtain a sixth submatrix. The second computing card is located within the third group of computing cards to which the first computing card belongs. The first computing card performs attention calculation on the fourth output matrix based on the sixth submatrix. The first computing card obtains the first output matrix based on the third and fourth output matrices.

[0375] Understandably, if the third set of computing cards includes two computing cards, the first computing card obtains the first output matrix based on the third and fourth output matrices. Specifically, this can be achieved by accumulating the third and fourth output matrices to obtain the first output matrix. If the third set of computing cards includes more than two computing cards, the first computing card cyclically executes the following processes: calculating the third output matrix, performing the third communication operation, and calculating the fourth output matrix, until the first output matrix is ​​obtained through multiple accumulations.

[0376] In this approach, the third communication operation corresponding to the third parallel strategy can refer to the ring. This step in this approach can also be understood as calculating attention once, passing it once, and accumulating it once.

[0377] For example, continuing with the previous example of a 3×3 third submatrix, this method can be illustrated as shown in Figure 22. The first computing card performs attention calculation on the 3×3 third submatrix to obtain a 3×3 third output matrix. The first computing card then obtains a 3×3 sixth submatrix based on the third communication operation, and performs attention calculation on the 3×3 sixth submatrix to obtain a 3×3 fourth output matrix. The 3×3 third output matrix and the 3×3 fourth output matrix are then added together to obtain a 3×3 intermediate matrix. This process of attention calculation, third communication operation, and accumulation is repeated multiple times to obtain a 3×3 first output matrix.

[0378] Step 19032: The first computing card obtains the second output matrix of the linear module based on the first communication operation involved in the first output matrix and the first parallel strategy.

[0379] After the first computing card obtains the first output matrix, it obtains the second output matrix of the linear module based on the first output matrix and the first communication operation involved in the first parallel strategy.

[0380] The second output matrix has the same number of rows and columns as the first sub-matrix, but different content. The second output matrix is ​​used to obtain the data processing result obtained by the input data through the multi-head attention network.

[0381] Optionally, the first computing card obtains the eighth submatrix based on the mapping matrix between the first output matrix and the average row-cut matrix. Within the first group of computing cards to which the first computing card belongs, the first computing card performs a first communication operation based on the eighth submatrix to obtain the second output matrix.

[0382] The eighth submatrix has the same number of rows and columns as the first submatrix, but its content differs. Similarly, the eighth submatrix has the same number of rows and columns as the second output matrix, but its content differs. The number of rows in the mapping matrix is ​​the same as the number of columns in the complete weight matrix, and the number of columns in the mapping matrix is ​​the same as the number of rows in the complete weight matrix. Alternatively, it can be understood that the mapping matrix, after being averaged across rows, is the Xth part from top to bottom of the complete mapping matrix after it has been averaged across rows into A parts.

[0383] For example, continuing the above example, as shown in Figure 23, after the first computing card obtains the 3×3 first output matrix, it obtains the 3×12 eighth submatrix through a linear module (e.g., multiplied by a mapping matrix that has been averaged and row-cut). A first communication operation is then performed within the first set of computing cards to obtain the 3×12 second output matrix.

[0384] In this application, by introducing multiple parallel strategies and limiting the execution order between them (starting with the first parallel strategy, ending with the third parallel strategy, and ending with the first parallel strategy), the communication overhead introduced by each parallel strategy can be reduced, thereby lowering the total communication overhead and improving data processing efficiency. Furthermore, the scheme provided in this application can also achieve communication computation masking. For example, the third communication operation of the V matrix can mask the calculation of the attention score (i.e., the process of obtaining the second output matrix), thereby increasing throughput. In addition, compared to existing two-dimensional model parallel schemes (i.e., the input sequence cannot be segmented, and the length of the input sequence is limited), this application, through multi-dimensional parallelism, can not only achieve sequence segmentation but also model parallelism, thereby improving the data processing efficiency of multi-head attention networks.

[0385] The third scenario involves multiple parallel strategies, including both the first and second parallel strategies. That is, multiple parallel strategies include model parallelism and efficient communication sequence parallelism. For example, the multiple communication operations corresponding to these parallel strategies might include all-to-all operations and all-reduce operations.

[0386] Please refer to Figure 24A, a flowchart illustrating a third scenario of the data processing method provided in this application. This method can be executed by the cloud device in Figure 3A (i.e., the data processing device includes multiple cloud devices, each including at least one computing card), or by the terminal device in Figure 3B (i.e., the data processing device includes multiple terminal devices, each including at least one computing card), or jointly by the terminal device and the cloud device in Figure 3A (i.e., the data processing device includes both terminal devices and cloud devices). This data processing method includes, but is not limited to, steps 24011 to 24041.

[0387] Step 24011: The first calculation card performs QKV calculation;

[0388] Step 24021: The first calculation card performs an alltoall operation based on the result of the QKV calculation;

[0389] Step 24031: The first computing card performs attention calculation based on the result of the alltoall operation;

[0390] Step 24041: The first computing card performs an allreduce operation based on the result of the attention calculation. The allreduce operation is used to obtain the data processing result of the multi-head attention network.

[0391] Furthermore, for ease of understanding, the various steps in the embodiment shown in FIG24A will be described below with reference to FIG24B to FIG27.

[0392] In this case, N = A * B; A represents the parallelism of the first parallel strategy, B represents the parallelism of the second parallel strategy, and A * B is related to the number of attention heads in the multi-head attention network. The first parallel strategy has B first-group computation cards in parallel, and the second parallel strategy has A second-group computation cards in parallel. A and B are positive integers greater than 1. The first computation card is any one of the N computation cards, and its ranking in the first group is X-th, and its ranking in the second group is Y-th. X is greater than 0 and less than or equal to A; Y is greater than 0 and less than or equal to B. For example, A * B * T is the number of attention heads in the multi-head attention network, and T is a positive integer greater than or equal to 1.

[0393] Please refer to Figure 24B, another flowchart illustrating the third scenario of the data processing method provided in this application. This method can be executed by the cloud device in Figure 3A (i.e., the data processing device includes multiple cloud devices, each cloud device including at least one computing card), or by the terminal device in Figure 3B (i.e., the data processing device includes multiple terminal devices, each terminal device including at least one computing card), or by a combination of the terminal device and the cloud device in Figure 3A (i.e., the data processing device includes both the terminal device and the cloud device). This data processing method includes, but is not limited to, steps 24012 to 24042.

[0394] Step 24012: The first calculation card performs QKV calculation based on the first submatrix and the second submatrix to obtain the third submatrix.

[0395] The first calculation card first obtains the first submatrix and the second submatrix, and then performs QKV calculation based on the first submatrix and the second submatrix to obtain the third submatrix.

[0396] In this application, the first computing card can obtain the first sub-matrix in various ways. For example, in the case of a multi-head attention network including an encoder and a decoder, it can be obtained by splitting the feature matrix obtained after inputting input data into the encoder. Alternatively, it can be obtained by receiving data from other devices (e.g., devices responsible for data distribution in a computing cluster). Another example is selection from a database, etc., and the specific method is not limited here.

[0397] The first submatrix is ​​the Y-th submatrix from top to bottom after the feature matrix is ​​divided into B parts by average rows; the feature matrix is ​​the feature matrix corresponding to the input data of the multi-head attention network; the second submatrix is ​​the X-th submatrix from left to right after the complete weight matrix is ​​divided into A parts by average columns; the third submatrix is ​​the Y-th submatrix from top to bottom after the rows of the complete QKV matrix are divided into B parts by average rows; the third submatrix is ​​the X-th submatrix from left to right after the columns of the complete QKV matrix are divided into A parts by average columns; the complete QKV matrix is ​​related to the feature matrix and the complete weight matrix.

[0398] This step can also be understood as the calculation process of QKV. Furthermore, the complete weight matrix mentioned above includes the complete Wq matrix, complete Wk matrix, and complete Wv matrix corresponding to the input data. Correspondingly, the complete QKV matrix includes the complete Q matrix, complete K matrix, and complete V matrix. In addition, the complete QKV matrix is ​​related to the feature matrix and the complete weight matrix; it can be understood that the complete QKV matrix is ​​obtained by multiplying the feature matrix and the complete weight matrix.

[0399] To make it easier to understand, the following example illustrates this step:

[0400] For example, the process of this step can be shown in Figure 25, with A=4, B=3, X=Y=1 (that is, the first computing card is ranked 1st in the first group of computing cards and the first computing card is ranked 1st in the second group of computing cards), the first communication operation corresponding to the first parallel strategy is allreduce, the second communication operation corresponding to the second parallel strategy is alltoall, the feature matrix is ​​6 rows and 12 columns (i.e., 6×12), and the complete weight matrix is ​​12 rows and 12 columns (i.e., 12×12).

[0401] The feature matrix is ​​a 6×12 matrix, the complete weight matrix is ​​a 12×12 matrix, and the complete QKV matrix is ​​a 6×12 matrix. The first submatrix is ​​the Y-th submatrix from top to bottom after the feature matrix is ​​divided into B equal rows (i.e., the first submatrix from top to bottom after being divided into 3 rows), which is a 2×12 matrix. The second submatrix is ​​the X-th submatrix from left to right after the complete weight matrix is ​​divided into A equal columns (i.e., the first submatrix from left to right after being divided into 4 columns), which is a 12×3 matrix. The third submatrix consists of the Y-th submatrix from top to bottom after the rows of the complete QKV matrix are divided into B equal rows (i.e., the first submatrix from top to bottom after being divided into 3 rows), and the columns of the third submatrix are the X-th submatrix from left to right after the columns of the complete QKV matrix are divided into A equal columns (i.e., the first submatrix from left to right after being divided into 4 columns), which is a 2×3 matrix.

[0402] Step 24022: Within the second group of computing cards to which the first computing card belongs, the first computing card obtains the fourth submatrix based on the second communication operation involved in the third submatrix and the second parallel strategy.

[0403] After the first computing card obtains the third submatrix, within the second group of computing cards to which the first computing card belongs, the first computing card obtains the fourth submatrix based on the second communication operation involving the third submatrix and the second parallel strategy.

[0404] The columns of the fourth submatrix are the (X-1)*B+Yth parts from top to bottom after the columns of the complete QKV matrix are divided into A*B parts, and the rows of the fourth submatrix are the rows of the complete QKV matrix.

[0405] This step can also be understood as follows: before the second communication operation, parallel processing is performed by cutting rows; after the second communication operation, parallel processing is performed by cutting columns.

[0406] For example, continuing the above example, the process of this step can be shown in Figure 26. The third submatrix is ​​a 2×3 matrix. The first computing card obtains the fourth submatrix within its respective second group of computing cards based on the third submatrix and the second communication operation involved in the second parallel strategy. The columns of the fourth submatrix are the (X-1)*B+Yth parts from left to right after the columns of the complete QKV matrix are divided into A*B parts (i.e., the first part from left to right after 12 parts). The rows of the fourth submatrix are the rows of the complete QKV matrix (i.e., 6 rows), that is, the fourth submatrix is ​​a 6×1 matrix.

[0407] Step 24032: The first computing card performs attention calculation based on the fourth submatrix to obtain the first output matrix of the self-attention module.

[0408] After the first computing card obtains the fourth sub-matrix, it performs attention calculations based on the fourth sub-matrix to obtain the first output matrix of the self-attention module.

[0409] The multi-head attention network includes a self-attention module and a linear module. The first output matrix has the same number of rows and columns as the fourth sub-matrix, but their contents are different.

[0410] Specifically, the first computation card performs attention calculations based on the QKV in the fourth submatrix of 6×1 to obtain the first output matrix of 6×1.

[0411] Step 24042: The first computing card obtains the second output matrix of the linear module based on the first output matrix, the second communication operation, and the first communication operation involved in the first parallel strategy.

[0412] After the first computing card obtains the first output matrix, it obtains the second output matrix of the linear module based on the first output matrix, the second communication operation, and the first communication operation involved in the first parallel strategy.

[0413] The second output matrix has the same number of rows and columns as the first sub-matrix, but different content. The second output matrix is ​​used to obtain the data processing result obtained by the input data through the multi-head attention network.

[0414] Optionally, within the second group of computing cards to which the first computing card belongs, the first computing card performs a second communication operation based on the first output matrix to obtain a seventh submatrix. The first computing card obtains an eighth submatrix based on the mapping matrix between the seventh submatrix and the average row-cut matrix. Within the first group of computing cards to which the first computing card belongs, the first computing card performs a first communication operation based on the eighth submatrix to obtain a second output matrix.

[0415] The seventh submatrix has the same number of rows as the Yth submatrix (from top to bottom) after the rows of the complete QKV matrix are divided into B parts, but its content is different. The seventh submatrix has the same number of columns as the Xth submatrix (from left to right) after the columns of the complete QKV matrix are divided into A parts, but its content is different. The eighth submatrix has the same number of rows and columns as the first submatrix, but its content is different, and the eighth submatrix also has the same number of rows and columns as the second output matrix, but its content is different. The mapping matrix has the same number of rows as the complete weight matrix, and the mapping matrix has the same number of columns as the complete weight matrix. Alternatively, it can be understood that the mapping matrix, after being averaged by rows, is the Xth submatrix (from top to bottom) after the complete mapping matrix has been averaged by rows into A parts.

[0416] For example, continuing the above example, as shown in Figure 27, after the first computing card obtains the 6×1 first output matrix, a second communication operation is performed in the second set of computing cards to obtain a 2×3 seventh submatrix. Then, a 2×12 eighth submatrix is ​​obtained through a linear module (e.g., multiplied by a mapping matrix that is averaged and row-cut). Finally, a first communication operation is performed in the first set of computing cards to obtain a 2×12 second output matrix.

[0417] In this application, by introducing multiple parallel strategies and limiting the execution order between them (first parallel strategy starts, second parallel strategy starts, second parallel strategy ends, first parallel strategy ends), the communication overhead introduced by each parallel strategy can be reduced, thereby reducing the total communication overhead and improving data processing efficiency. Furthermore, the scheme provided in this application can also achieve communication computation masking. For example, the second communication operation of the Q matrix can mask the computation of the K and V matrices, the second communication operation of the K matrix can mask the computation of the V matrix, and the second communication operation of the V matrix can mask the computation of the attention score (i.e., the process of obtaining the second output matrix), thereby increasing throughput. In addition, compared to existing two-dimensional model parallel schemes (i.e., the input sequence cannot be segmented, and the length of the input sequence is limited), this application, through multi-dimensional parallelism, can not only achieve sequence segmentation but also model parallelism, thereby improving the data processing efficiency of multi-head attention networks.

[0418] The fourth scenario involves multiple parallel strategies, including a first parallel strategy, a second parallel strategy, and a third parallel strategy. The first parallel strategy is a high-dimensional (row-slicing and column-slicing) model parallel strategy. That is, the multiple parallel strategies include at least: a high-dimensional (row-slicing and column-slicing) model parallel strategy (MP), a CE-SP communication efficient sequence parallel strategy, and a ME-SP memory efficient sequence parallel strategy. The ME-SP memory efficient sequence parallel strategy is also known as a video memory sequence parallel strategy.

[0419] Please refer to Figure 28, a flowchart illustrating the fourth scenario of the data processing method provided in this application. This method can be executed by the cloud device in Figure 3A (i.e., the data processing device includes multiple cloud devices, each cloud device includes at least one computing card, and every other cloud device includes at least one computing card), or it can be executed jointly by the terminal device and the cloud device in Figure 3A (i.e., the data processing device includes both the terminal device and the cloud device). This data processing method includes, but is not limited to, steps 2801-2803.

[0420] Step 2801: The first computing card performs an allgather operation based on the first data to obtain the second data. The first data is a submatrix of the feature matrix of the input data of the multi-head attention network.

[0421] Step 2802: The first computing card performs QKV calculation based on the second data and the first sub-model. The first sub-model is a sub-matrix obtained by row-cutting and column-cutting the weight matrix of the multi-head attention network.

[0422] Step 2803: The first computing card performs a reducescatter operation based on the QKV calculation result to obtain the third data, which is used to obtain the processing result of the first data.

[0423] Prior to step 2801, the method further includes:

[0424] The feature matrix of the input data of the multi-head attention network is segmented along the sequence dimension to obtain the C sub-sequence data.

[0425] The first data is obtained by averaging the column cuts of each of the C subsequence data.

[0426] N = A * B * C; A represents the parallelism of the first parallel strategy, B represents the parallelism of the second parallel strategy, and C represents the parallelism of the third parallel strategy. A, B, and C are positive integers greater than 1. The method further includes: performing row and column slicing on the weight matrix of the multi-head attention network to obtain the A sub-models of the multi-head attention network, wherein the first sub-model is any one of the A sub-models of the multi-head attention network.

[0427] For example, please refer to Figure 29. Taking A=4, B=1, C=2 as an example, N=A*B*C=8; that is, the parallelism of MP is 4, the parallelism of CE-SP is 1, the parallelism of ME-SP is 2, and the computing cluster includes 8 computing cards, namely computing card 1, computing card 2, computing card 3, computing card 4, computing card 5, computing card 6, computing card 7 and computing card 8.

[0428] The input data of the multi-head attention network is split along the sequence dimension to obtain two sub-sequences, each of which is half of the input data. In the scheme shown in Figure 29, the input data of the multi-head attention network is "This", "is", "a", "seq". Since C=2, the first sub-sequence data is "This" and "is", and the second sub-sequence data is "a" and "seq".

[0429] Calculation cards 1-4 store the first subsequence, and calculation cards 2-8 store the second subsequence.

[0430] In this embodiment, the input data, subsequence data, first data, second data, third data, fourth data, fifth data, sixth data, and seventh data all represent the feature matrix of the data.

[0431] Each subsequence data is divided into four equal parts, resulting in four first data points, which are placed on four separate computing cards. Each first data point represents 1 / 4 of the subsequence data. Computing cards 1-4 store the first column of the divided first subsequence, i.e., the first data point of the first column.

[0432] Perform an all gather on the first data between the first data in the first column of calculation card 1 and the first data in the second column of calculation card 3. Similarly, perform an all gather on the first data between calculation cards 2 and 4, calculation cards 5 and 7, and calculation cards 6 and 8 to obtain the second data.

[0433] Since 1 / 4 of the first data subsequence cannot be directly used for QKV calculation with the first sub-model in terms of form, while 1 / 2 of the subsequence meets the requirements for QKV calculation with the first sub-model in terms of form, it is necessary to perform an all-gather operation on the first data in adjacent columns, such as the first data in the first column and the first data in the second column, the first data in the third column and the first data in the fourth column, to obtain the second data that meets the requirements for QKV calculation with the first sub-model in terms of form.

[0434] QKV calculation is performed based on the second data and the first sub-model. The first sub-model is a sub-matrix obtained by row-slicing and column-slicing the weight matrix of the multi-head attention network.

[0435] Within the parallel MP group of the model, the QKV computation results are reduced and scattered to obtain the third data, which is used to obtain the processing results of the first data.

[0436] The purpose of performing reducescatter within the MP group is illustrated using computation card 1 and computation card 2 as examples. Computation card 1 stores the upper half of Head 1 (H1 in Figure 29) and Head 2 (H2 in Figure 34). Therefore, the QKV calculation result in computation card 1 is incomplete. Thus, a reducescatter operation needs to be performed between computation card 1 and computation card 2, which stores the lower half of Head 1 and Head 2, so that computation card 1 contains the complete QKV calculation result of Head 1. That is, computation card 1 contains the complete QKV calculation result of Head 1, and computation card 2 contains the complete QKV calculation result of Head 2, or computation card 1 contains the complete QKV calculation result of Head 2, and computation card 2 contains the complete QKV calculation result of Head 1.

[0437] Among them, calculation cards 1 and 2 form one MP group (similarly, calculation cards 3 and 4 form one MP group, calculation cards 5 and 6 form one MP group, and calculation cards 7 and 8 form one MP group). The third data consists of half of the Q matrix, half of the K matrix, and half of the V matrix.

[0438] Please refer to Figure 30. This data processing method also includes, but is not limited to, steps 3001-3505.

[0439] Step 3001: The first calculation card performs the allgather operation on the KV matrix in the third data to obtain the fourth data;

[0440] Step 3002: Perform self-attention calculation on the Q matrix in the third data and the KV matrix in the fourth data to obtain the fifth data;

[0441] Step 3003: The first calculation card performs an allgather operation on the fifth data to obtain the sixth data;

[0442] Step 3004: The sixth data is multiplied by the second sub-model to obtain the seventh data. The second sub-model is a sub-matrix obtained by row-cutting and column-cutting the weight matrix of the multi-head attention network.

[0443] Step 3005: Perform a reducescatter operation on the seventh data to obtain the processing result of the first data.

[0444] For example, please refer to Figure 29, where A=4, B=1, C=2, the computing cluster includes 8 computing cards (N=8), namely computing card 1, computing card 2, computing card 3, computing card 4, computing card 5, computing card 6, computing card 7 and computing card 8.

[0445] Within the memory-efficient sequential parallel ME-SP group, the allgather operation is performed on the KV matrix in the third data to obtain the fourth data.

[0446] The fourth set of data includes the complete K matrix and the complete V matrix.

[0447] Among them, computing card 1 and computing card 5 form one ME-SP parallel group (similarly, computing card 3 and computing card 4 form one model parallel group, computing card 5 and computing card 6 form one model parallel group, and computing card 7 and computing card 8 form one model parallel group).

[0448] The fifth data Y is obtained by performing self-attention calculation on half of the Q matrix from the third data, the complete K matrix from the fourth data, and the complete V matrix.

[0449] Perform the allgather operation on the fifth data between calculation cards 1 and 2. Similarly, perform all gather on the fifth data between calculation cards 3 and 4, 5 and 6, and 7 and 8 to obtain the sixth data.

[0450] Similarly, in order to meet the formal requirements of matrix multiplication with the second sub-model, an allgather operation is performed between the fifth data in adjacent columns;

[0451] The sixth data is multiplied by the second sub-model to obtain the seventh data. The second sub-model is a sub-matrix obtained by row-cutting and column-cutting the weight matrix of the multi-head attention network.

[0452] A reducescatter operation is performed on the seventh data between computing cards 1 and 3. Similarly, reducescatter is performed on the seventh data between computing cards 2 and 4, 5 and 7, and 6 and 8 to obtain the processing result of the first data.

[0453] In one embodiment of the fourth case, the first sub-model is a sub-matrix obtained by segmenting the multi-head attention network based on the first segmentation scheme, and the second sub-model is a sub-matrix obtained by segmenting the multi-head attention network based on the second segmentation scheme.

[0454] The first segmentation scheme and the second segmentation scheme are calculated using the first constraint, the second constraint and the third constraint;

[0455] Based on the first and second segmentation schemes, the parameter matrix of the multi-head attention network is segmented by rows and columns.

[0456] The first constraint is satisfied:

[0457] The second constraint satisfies: A = p1 * p2

[0458] The third constraint satisfies: A=p1′*p2′

[0459] Where A represents the parallelism of the first parallel strategy.

[0460] p1 is the number of rows in the first segmentation scheme.

[0461] p2 is the number of columns in the first partitioning scheme.

[0462] b, s, and h represent the batch size of the input data, the sequence length of the input data, and the hidden layer dimension, respectively.

[0463] p1' is the number of rows in the second segmentation scheme.

[0464] p2' is the number of columns in the second segmentation scheme.

[0465] By applying the first, second, and third constraints, the optimal model splitting scheme can be obtained with minimal communication cost, further reducing model occupancy and thus lowering training memory requirements. Furthermore, this scheme is more compatible with the Switch's communication architecture, efficiently utilizing the bandwidth between chip dies. For example, in this scheme, the allgather and reducescatter operators can achieve some pipeline masking before and after matrix multiplication using fusion operators. However, when the communication time exceeds the matrix multiplication computation time, communication becomes a bottleneck, resulting in cavitation latency. The two dies of the Switch have additional bandwidth, which, along with the existing hccs bandwidth, can form independent communication domains that do not interfere with each other. Therefore, the allgather and reducescatter communications in this scheme can be placed in these two communication domains respectively, thus avoiding communication blockage and completely eliminating cavitation time.

[0466] The data processing method in this application has been described above. The data processing device in this application is described below. Please refer to FIG31 for an embodiment of the data processing device in this application. The data processing device can realize the function of the data processing device in the above method embodiment, and thus can also realize the beneficial effects of the above method embodiment.

[0467] In this application, the data processing device can be the cloud device in Figure 3A, the terminal device in Figure 3B, or the computing node or computing card in Figure 4. That is, the data processing device can also be understood as a computing node in a computing cluster or a computing card within a computing node. The data processing device includes a processing module 3101. Optionally, the data processing device may further include an acquisition module 3102.

[0468] Both the acquisition module 3102 and the processing module 3101 can be implemented in software or in hardware. For example, the implementation of the processing module 3101 will be described below. Similarly, the implementation of the acquisition module 3102 can be referenced from the implementation of the processing module 3101.

[0469] As an example of a software functional unit, the processing module 3101 may include code running on a computing instance. The computing instance can be similar to the description of data processing devices and / or terminal devices in Figure 3B above; that is, a computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Related descriptions can be found in the previous descriptions and will not be repeated here.

[0470] Furthermore, as an example of a hardware functional unit, the processing module 3101 may include at least one computing device, such as a server. Alternatively, the processing module 3101 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0471] The first scenario for the data processing device: The operations performed by each module in the data processing device are similar to those performed by the first computing card in the embodiments shown in Figures 5 to 18. In this scenario, the computing cluster deploys a multi-head attention network, includes N computing cards, and performs multiple communication operations, including all-to-all operations and parallel operations. The parallel operations are either all-gather operations or ring operations. The first computing card is any one of the N computing cards.

[0472] The functions of each module in the first scenario are as follows:

[0473] Processing module 3101 is used for QKV calculation;

[0474] Processing module 3101 is also used to perform alltoall operations based on the results calculated using QKV;

[0475] The processing module 3101 is also used to perform parallel operations based on the results of the alltoall operation, and the parallel operations are used to obtain the data processing results of the multi-head attention network.

[0476] In one possible implementation, the multiple communication operations also include: a full reduction (allreduce) operation; and a processing module 3101, which is further used to perform an allreduce operation based on the results of the parallel operations, the allreduce operation being used to obtain the data processing results.

[0477] In one possible implementation, N = A * B * C; A represents the parallelism of the first parallel strategy corresponding to the allreduce operation, B represents the parallelism of the second parallel strategy corresponding to the alltoall operation, and C represents the parallelism of the third parallel strategy corresponding to the parallel operation. The second parallel strategy has multiple second sets of computing cards in parallel, and the third parallel strategy has multiple third sets of computing cards in parallel. A, B, and C are positive integers greater than 1.

[0478] Processing module 3101 is specifically used to perform an alltoall operation based on the QKV calculation result within the second group of computing cards to which the first computing card belongs;

[0479] The processing module 3101 is specifically used to perform parallel operations based on the result of the alltoall operation within the third group of computing cards to which the first computing card belongs;

[0480] The processing module 3101 is specifically used to perform alltoall and allreduce operations based on the results of parallel operations.

[0481] In one possible implementation, the first parallel strategy uses B*C first-group computing cards, the second parallel strategy uses A*C second-group computing cards, and the third parallel strategy uses A*B third-group computing cards. The first computing card is ranked Xth in its first-group, Yth in its second-group, and Zth in its third-group, where X is greater than 0 and less than or equal to A; Y is greater than 0 and less than or equal to B; and Z is greater than 0 and less than or equal to C.

[0482] The processing module 3101 is specifically used to perform QKV calculation based on the first submatrix and the second submatrix to obtain the QKV calculation result. The QKV calculation result includes a third submatrix. The first submatrix is ​​the B*(Z-1)+Yth submatrix from top to bottom after the feature matrix is ​​divided into B*C parts by average rows. The feature matrix is ​​the feature matrix corresponding to the input data of the multi-head attention network. The second submatrix is ​​the Xth submatrix from left to right after the complete weight matrix is ​​divided into A parts by average columns. The rows of the third submatrix are the B*(Z-1)+Yth submatrix from top to bottom after the rows of the complete QKV matrix are divided into B*C parts by average rows. The columns of the third submatrix are the Xth submatrix from left to right after the columns of the complete QKV matrix are divided into A parts by average columns. The complete QKV matrix is ​​related to the feature matrix and the complete weight matrix.

[0483] The processing module 3101 is specifically used to perform an alltoall operation based on the third submatrix to obtain the result of the alltoall operation. The result of the alltoall operation includes a fourth submatrix. The columns of the fourth submatrix are the (X-1)*B+Yth parts from left to right after the columns of the complete QKV matrix are divided into A*B parts on average. The rows of the fourth submatrix are the Zth parts from top to bottom after the rows of the complete QKV matrix are divided into C parts on average.

[0484] The processing module 3101 is specifically used to perform parallel operations based on the fourth sub-matrix to obtain the result of the parallel operations. The result of the parallel operations is the first output matrix of the self-attention module. The multi-head attention network includes a self-attention module and a linear module. The first output matrix has the same number of rows and columns as the fourth sub-matrix, but the content is different.

[0485] In one possible implementation, the processing module 3101 is specifically used to perform an allgather operation based on the KV matrix in the fourth submatrix to obtain the fifth submatrix, which is the (X-1)*B+Yth part from left to right after the KV matrix in the complete QKV matrix is ​​divided into A*B parts by average columns.

[0486] The processing module 3101 is specifically used to perform attention calculations based on the Q matrix in the fourth sub-matrix and the fifth sub-matrix to obtain the first output matrix.

[0487] In one possible implementation, the processing module 3101 is specifically used to perform attention calculation based on the fourth submatrix to obtain the third output matrix;

[0488] The processing module 3101 is specifically used to perform a ring operation based on the fourth submatrix and the second computing card to obtain the sixth submatrix. The second computing card is located in the third group of computing cards to which the first computing card belongs.

[0489] The processing module 3101 is specifically used to perform attention calculation based on the sixth submatrix to obtain the fourth output matrix;

[0490] The processing module 3101 is specifically used to obtain the first output matrix based on the third output matrix and the fourth output matrix.

[0491] In one possible implementation, the processing module 3101 is specifically used within the second group of computing cards to which the first computing card belongs to perform an alltoall operation based on the result of the parallel operation to obtain a seventh sub-matrix. The number of rows in the seventh sub-matrix is ​​the same as the number of rows in the B*(Z-1)+Yth part from top to bottom after the rows of the complete QKV matrix are divided into B*C parts, but the content is different. The number of columns in the seventh sub-matrix is ​​the same as the number of columns in the Xth part from left to right after the columns of the complete QKV matrix are divided into A parts, but the content is different.

[0492] Processing module 3101 is specifically used to obtain the eighth submatrix based on the seventh submatrix and the mapping matrix that is averaged and row-cut. The eighth submatrix has the same number of rows and columns as the first submatrix, but different content. The eighth submatrix also has the same number of rows and columns as the second output matrix, but different content. The number of rows in the mapping matrix is ​​the same as the number of columns in the complete weight matrix, and the number of columns in the mapping matrix is ​​the same as the number of rows in the complete weight matrix. The second output matrix has the same number of rows and columns as the first submatrix, but different content. The second output matrix is ​​the output matrix of the linear module in the multi-head attention network.

[0493] The processing module 3101 is specifically used to perform an allreduce operation based on the eighth submatrix within the first group of computing cards to which the first computing card belongs, in order to obtain the second output matrix.

[0494] In one possible implementation, N = B * C; B represents the degree of parallelism of the second parallel strategy corresponding to the alltoall operation, and C represents the degree of parallelism of the third parallel strategy corresponding to the parallel operation. The second parallel strategy has multiple second sets of computing cards in parallel, and the third parallel strategy has multiple third sets of computing cards in parallel. B and C are positive integers greater than 1.

[0495] Processing module 3101 is specifically used to perform an alltoall operation based on the QKV calculation result within the second group of computing cards to which the first computing card belongs;

[0496] The processing module 3101 is specifically used to perform parallel operations based on the result of the alltoall operation within the third group of computing cards to which the first computing card belongs;

[0497] The processing module 3101 is also used to perform an alltoall operation based on the parallel operation result within the second group of computing cards to which the first computing card belongs. The alltoall operation is used to obtain the data processing result.

[0498] In one possible implementation, the second parallel strategy has C second-group computing cards running in parallel, and the third parallel strategy has B third-group computing cards running in parallel; the first computing card is ranked Yth in its second-group computing card and Zth in its third-group computing card; Y is greater than 0 and less than or equal to B; Z is greater than 0 and less than or equal to C.

[0499] Processing module 3101 is specifically used to perform QKV calculation based on the first submatrix and the complete weight matrix to obtain the QKV calculation result. The QKV calculation result includes: the ninth submatrix. The first submatrix is ​​the B*(Z-1)+Yth submatrix from top to bottom after the feature matrix is ​​divided into B*C parts by average row division. The feature matrix is ​​the feature matrix corresponding to the input data of the multi-head attention network. The rows of the ninth submatrix are the B*(Z-1)+Yth submatrix from top to bottom after the rows of the complete QKV matrix are divided into B*C parts. The columns of the ninth submatrix are the columns of the complete QKV matrix. The complete QKV matrix is ​​related to the feature matrix and the complete weight matrix.

[0500] The processing module 3101 is specifically used to perform an alltoall operation based on the ninth submatrix to obtain the result of the alltoall operation. The result of the alltoall operation includes the tenth submatrix. The columns of the tenth submatrix are the Yth parts from left to right after the columns of the complete QKV matrix are divided into B parts on average. The rows of the tenth submatrix are the Zth parts from top to bottom after the rows of the complete QKV matrix are divided into C parts on average.

[0501] The processing module 3101 is specifically used to perform parallel operations based on the tenth submatrix to obtain the result of the parallel operation. The result of the parallel operation is the fifth output matrix of the self-attention module. The multi-head attention network includes a self-attention module and a linear module. The fifth output matrix has the same number of rows and columns as the tenth submatrix, but the content is different.

[0502] Processing module 3101 is specifically used to perform an alltoall operation based on the fifth output matrix to obtain the eleventh submatrix. The number of rows in the eleventh submatrix is ​​the same as the number of rows in the B*(Z-1)+Yth part from top to bottom after the rows of the complete QKV matrix are divided into B*C parts on average, but the content is different. The number of columns in the eleventh submatrix is ​​the same as the number of columns in the complete QKV matrix, but the content is different.

[0503] The processing module 3101 is specifically used to obtain the sixth output matrix based on the eleventh submatrix and the mapping matrix. The number of rows in the mapping matrix is ​​the same as the number of columns in the complete weight matrix, and the number of columns in the mapping matrix is ​​the same as the number of rows in the complete weight matrix.

[0504] In one possible implementation, the processing module 3101 is specifically used to perform an allgather operation based on the KV matrix in the tenth submatrix to obtain the twelfth submatrix. The twelfth submatrix is ​​the (X-1)*B+Yth part from left to right after the KV matrix in the complete QKV matrix is ​​divided into A*B parts by average columns.

[0505] The processing module 3101 is specifically used to perform attention calculations based on the Q matrix in the tenth sub-matrix and the twelfth sub-matrix to obtain the fifth output matrix.

[0506] In one possible implementation, the processing module 3101 is specifically used to perform attention calculation based on the tenth submatrix to obtain the seventh output matrix;

[0507] The processing module 3101 is specifically used to perform a ring operation based on the tenth submatrix and the second computing card to obtain the thirteenth submatrix. The second computing card is located in the third group of computing cards to which the first computing card belongs.

[0508] The processing module 3101 is specifically used to perform attention calculation based on the thirteenth submatrix to obtain the eighth output matrix;

[0509] The processing module 3101 is specifically used to obtain the fifth output matrix based on the seventh output matrix and the eighth output matrix.

[0510] In this embodiment, the operations performed by each module in the data processing device are similar to those described for the first computing card in the embodiments shown in Figures 5 to 18 above, and will not be repeated here.

[0511] In this embodiment, based on the above-described scheme, compared to the existing schemes that use a single parallel strategy for data processing, the scheme provided in this application can improve the data processing efficiency of multi-head attention networks for input data. Specifically, by introducing multiple parallel strategies and limiting the execution order between parallel strategies (starting with the second parallel strategy and ending with the third parallel strategy), the communication overhead introduced by each parallel strategy can be reduced, thereby reducing the total communication overhead and improving data processing efficiency. Furthermore, the scheme provided in this application can also achieve communication computation masking. For example, the second communication operation of the Q matrix can mask the calculations of the K and V matrices; the second and third communication operations of the K matrix can mask the calculation of the V matrix; and the second and third communication operations of the V matrix can mask the calculation of the attention score (i.e., the process of obtaining the second output matrix), thereby increasing throughput. In addition, compared to existing two-dimensional model parallel schemes (i.e., the input sequence cannot be segmented, and the length of the input sequence is limited), this application, through multi-dimensional parallelism, can not only achieve sequence segmentation but also model parallelism, thereby improving the data processing efficiency of multi-head attention networks.

[0512] The second scenario for the data processing device: The operations performed by each module in the data processing device are similar to those performed by the first computing card in the embodiments shown in Figures 19A to 23. In this scenario, the computing cluster deploys a multi-head attention network, and the computing cluster includes N computing cards. These N computing cards perform multiple communication operations, including parallel operations and all-reduce operations. The parallel operations are either all-gather operations or ring operations. The first computing card is any one of the N computing cards.

[0513] The functions of each module in the second scenario are as follows:

[0514] Processing module 3101 is used for QKV calculation;

[0515] The processing module 3101 is also used to perform parallel operations based on the results of QKV calculations;

[0516] The processing module 3101 is also used to perform an allreduce operation based on the results of the parallel operation. The allreduce operation is used to obtain the data processing results of the multi-head attention network.

[0517] In one possible implementation, N = A * C; A represents the degree of parallelism of the allreduce operation corresponding to the first parallel strategy, and C represents the degree of parallelism of the parallel operation corresponding to the third parallel strategy. The first parallel strategy has C first-group computing cards in parallel, and the third parallel strategy has A third-group computing cards in parallel. A and C are positive integers greater than 1. The first computing card is ranked as Xth in its first-group computing card and as Zth in its third-group computing card. X is greater than 0 and less than or equal to A; Z is greater than 0 and less than or equal to C.

[0518] Processing module 3101 is specifically used to perform QKV calculation based on the first submatrix and the second submatrix to obtain the QKV calculation result. The QKV calculation result includes: a third submatrix. The first submatrix is ​​the Zth submatrix from top to bottom after the feature matrix is ​​divided into C parts by average rows. The feature matrix is ​​the feature matrix corresponding to the input data of the multi-head attention network. The second submatrix is ​​the Xth submatrix from left to right after the complete weight matrix is ​​divided into A parts by average columns. The rows of the third submatrix are the Zth submatrix from top to bottom after the rows of the complete QKV matrix are divided into C parts by average rows. The columns of the third submatrix are the Xth submatrix from left to right after the columns of the complete QKV matrix are divided into A parts by average columns. The complete QKV matrix is ​​related to the feature matrix and the complete weight matrix.

[0519] The processing module 3101 is specifically used to perform parallel operations based on the third sub-matrix within the third group of computing cards to which the first computing card belongs, so as to obtain the result of the parallel operation. The result of the parallel operation is the first output matrix of the self-attention module. The multi-head attention network includes a self-attention module and a linear module. The number of rows and columns of the first output matrix is ​​the same as that of the third sub-matrix, but the content is different.

[0520] Processing module 3101 is specifically used to perform an allreduce operation based on the first output matrix and the mapping matrix that has been averaged and row-cut to obtain an eighth sub-matrix. The eighth sub-matrix has the same number of rows and columns as the first sub-matrix, but different content. The eighth sub-matrix also has the same number of rows and columns as the second output matrix, but different content. The number of rows in the mapping matrix is ​​the same as the number of columns in the complete weight matrix, and the number of columns in the mapping matrix is ​​the same as the number of rows in the complete weight matrix. The second output matrix has the same number of rows and columns as the first sub-matrix, but different content. The second output matrix is ​​the output matrix of the linear module in the multi-head attention network.

[0521] The processing module 3101 is specifically used to perform an allreduce operation based on the eighth submatrix to obtain the second output matrix within the first group of computing cards to which the first computing card belongs.

[0522] In one possible implementation, the processing module 3101 is specifically used to perform an allgather operation based on the KV matrix in the third submatrix to obtain the fifth submatrix, which is the Xth part from left to right after the KV matrix in the complete QKV matrix is ​​divided into A parts by average columns.

[0523] The processing module 3101 is specifically used to perform attention calculation based on the Q matrix in the third submatrix and the fifth submatrix to obtain the first output matrix.

[0524] In one possible implementation, the processing module 3101 is specifically used to perform attention calculation based on the third submatrix to obtain the third output matrix;

[0525] The processing module 3101 is specifically used to perform a third communication operation with the second computing card based on the third submatrix to obtain the sixth submatrix. The second computing card is located in the third group of computing cards to which the first computing card belongs.

[0526] The processing module 3101 is specifically used to perform attention calculation based on the sixth submatrix to obtain the fourth output matrix;

[0527] The processing module 3101 is specifically used to obtain the first output matrix based on the third output matrix and the fourth output matrix.

[0528] In this embodiment, the operations performed by each module in the data processing device are similar to those described for the first computing card in the embodiments shown in Figures 19A to 23 above, and will not be repeated here.

[0529] In this embodiment, compared to the existing approach of processing data using a single parallel strategy, the solution provided in this application can improve the data processing efficiency of multi-head attention networks on input data. Specifically, by introducing multiple parallel strategies and defining the execution order between them (first parallel strategy starts, third parallel strategy starts and ends, first parallel strategy ends), the communication overhead introduced by each parallel strategy can be reduced, thereby reducing the total communication overhead and improving data processing efficiency. Furthermore, the solution provided in this application can also achieve communication computation masking. For example, parallel operations on the V matrix can mask the calculation of attention scores, thereby increasing throughput.

[0530] The third scenario for the data processing device: The operations performed by each module in the data processing device are similar to those performed by the first computing card in the embodiments shown in Figures 24 to 27. In this scenario, the computing cluster deploys a multi-head attention network, and the computing cluster includes N computing cards. These N computing cards perform multiple communication operations, including all-to-all operations and all-reduce operations. The first computing card is any one of the N computing cards.

[0531] The functions of each module in the third scenario are as follows:

[0532] Processing module 3101 is used for QKV calculation;

[0533] Processing module 3101 is also used to perform alltoall operations based on the results calculated using QKV;

[0534] Processing module 3101 is also used to perform attention calculation based on the result of the alltoall operation;

[0535] The processing module 3101 is also used to perform an allreduce operation based on the result of attention calculation. The allreduce operation is used to obtain the data processing result of the multi-head attention network.

[0536] In one possible implementation, N = A * B; A represents the parallelism of the first parallel strategy corresponding to the allreduce operation, and B represents the parallelism of the second parallel strategy corresponding to the alltoall operation. The first parallel strategy has B first-group computing cards running in parallel, and the second parallel strategy has A second-group computing cards running in parallel. A and B are positive integers greater than 1. The first computing card is ranked Xth in its first-group computing card and Yth in its second-group computing card. X is greater than 0 and less than or equal to A; Y is greater than 0 and less than or equal to B.

[0537] The processing module 3101 is specifically used to perform QKV calculation based on the first submatrix and the second submatrix to obtain the QKV calculation result. The QKV calculation result includes: a third submatrix. The first submatrix is ​​the Yth submatrix from top to bottom after the feature matrix is ​​divided into B parts by average rows. The feature matrix is ​​the feature matrix corresponding to the input data of the multi-head attention network. The second submatrix is ​​the Xth submatrix from left to right after the complete weight matrix is ​​divided into A parts by average columns. The rows of the third submatrix are the Yth submatrix from top to bottom after the rows of the complete QKV matrix are divided into B parts by average rows. The columns of the third submatrix are the Xth submatrix from left to right after the columns of the complete QKV matrix are divided into A parts by average columns. The complete QKV matrix is ​​related to the feature matrix and the complete weight matrix.

[0538] The processing module 3101 is specifically used to perform an alltoall operation based on the third submatrix within the second group of computing cards to which the first computing card belongs, so as to obtain the result of the alltoall operation. The result of the alltoall operation includes a fourth submatrix. The columns of the fourth submatrix are the (X-1)*B+Yth parts from left to right after the columns of the complete QKV matrix are divided into A*B parts. The rows of the fourth submatrix are the rows of the complete QKV matrix.

[0539] The processing module 3101 is specifically used to perform attention calculation based on the fourth sub-matrix to obtain the result of the attention calculation. The result of the attention calculation is the first output matrix of the self-attention module. The multi-head attention network includes a self-attention module and a linear module. The number of rows and columns of the first output matrix is ​​the same as that of the fourth sub-matrix, but the content is different.

[0540] In one possible implementation, the processing module 3101 is specifically used to perform alltoall and allreduce operations based on the results of parallel operations.

[0541] In one possible implementation, the processing module 3101 is specifically used to perform an alltoall operation based on the first output matrix within the second group of computing cards to which the first computing card belongs, so as to obtain a seventh submatrix, wherein the number of rows of the seventh submatrix is ​​the same as the number of columns of the first output matrix, and the number of columns of the seventh submatrix is ​​the same as the number of rows of the first output matrix.

[0542] Processing module 3101 is specifically used to obtain the eighth submatrix based on the seventh submatrix and the mapping matrix that is averaged and row-cut. The eighth submatrix has the same number of rows and columns as the first submatrix, but different content. The eighth submatrix also has the same number of rows and columns as the second output matrix, but different content. The number of rows in the mapping matrix is ​​the same as the number of columns in the complete weight matrix, and the number of columns in the mapping matrix is ​​the same as the number of rows in the complete weight matrix. The second output matrix has the same number of rows and columns as the first submatrix, but different content. The second output matrix is ​​the output matrix of the linear module in the multi-head attention network.

[0543] The processing module 3101 is specifically used to perform an allreduce operation based on the eighth submatrix within the first group of computing cards to which the first computing card belongs, in order to obtain the second output matrix.

[0544] In this embodiment, the operations performed by each module in the data processing device are similar to those described for the first computing card in the embodiments shown in Figures 24 to 27 above, and will not be repeated here.

[0545] In this embodiment, compared to the existing approach of processing data using a single parallel strategy, the solution provided in this application can improve the data processing efficiency of multi-head attention networks on input data. Specifically, by introducing multiple parallel strategies and defining the execution order between them (first parallel strategy, second parallel strategy begins, second parallel strategy ends), the communication overhead introduced by each parallel strategy can be reduced, thereby lowering the total communication overhead and improving data processing efficiency. Furthermore, the solution provided in this application can also achieve communication computation masking. For example, the all-to-all operation of the Q matrix can mask the computation of the K and V matrices, the all-to-all operation of the K matrix can mask the computation of the V matrix, and the all-to-all operation of the V matrix can mask the computation of the attention score, thereby increasing throughput.

[0546] The functions of each module in the fourth scenario are as follows:

[0547] Processing module 3101 is used to perform an allgather operation on the first data to obtain the second data, wherein the first data is a submatrix of the feature matrix of the input data of the multi-head attention network;

[0548] The processing module 3101 is also used to perform QKV calculation on the second data and the first sub-model, wherein the first sub-model is a sub-matrix obtained by row-cutting and column-cutting the weight matrix of the multi-head attention network;

[0549] The processing module 3101 is also used to perform a reducescatter operation on the QKV calculation results to obtain third data;

[0550] Processing module 3101 is also used to perform the allgather operation on the KV matrix in the third data to obtain the fourth data;

[0551] The processing module 3101 is also used to perform self-attention calculation on the Q matrix in the third data and the KV matrix in the fourth data to obtain the fifth data;

[0552] Processing module 3101 is also used to perform an allgather operation on the fifth data to obtain the sixth data;

[0553] The processing module 3101 is also used to perform matrix multiplication on the sixth data and the second sub-model to obtain the seventh data. The second sub-model is a sub-matrix obtained by row-cutting and column-cutting the weight matrix of the multi-head attention network.

[0554] The processing module 3101 is also used to perform a reducescatter operation on the seventh data to obtain the processing result of the first data.

[0555] In one possible implementation, the first data is the average column sub-matrix of the feature matrix of the input data of the multi-head attention network.

[0556] In another possible implementation, the first data is the feature matrix of the input data of the multi-head attention network, which is then segmented along the sequence dimension, and the sub-sequence data obtained by the segmentation along the sequence dimension is then averaged and column-cut into a sub-matrix.

[0557] In one possible implementation, the first sub-model is a sub-matrix obtained based on the first partitioning scheme, and the second sub-model is a sub-matrix obtained based on the second partitioning scheme. The method further includes:

[0558] Based on the first constraint, the second constraint, and the third constraint, the number of segments and columns of the first segmentation scheme and the number of segments and columns of the second segmentation scheme are calculated.

[0559] Based on the number of rows and columns of the first segmentation scheme and the number of rows and columns of the second segmentation scheme, the parameter matrix of the multi-head attention network is segmented into rows and columns.

[0560] The first constraint is satisfied:

[0561] The second constraint satisfies: A = p1 * p2

[0562] The third constraint satisfies: A=p1′*p2′

[0563] Where A represents the parallelism of the first parallel strategy.

[0564] p1 is the number of rows in the first segmentation scheme.

[0565] p2 is the number of columns in the first segmentation scheme.

[0566] b, s, and h represent the batch size of the input data, the sequence length of the input data, and the hidden layer dimension, respectively.

[0567] p1' is the number of rows in the second segmentation scheme.

[0568] p2' is the number of columns in the second segmentation scheme.

[0569] Please refer to Figure 32, which is another schematic structural diagram of the data processing device provided in this application. The data processing device includes a logic circuit 3201 and an input / output interface 3202. The data processing device can be an integrated circuit, etc.

[0570] The acquisition module 3102 shown in Figure 31 can be a communication interface, which can be the input / output interface 3202 in Figure 32. This input / output interface 3202 can include both an input interface and an output interface. Alternatively, the communication interface can also be a transceiver circuit, which can include both an input interface circuit and an output interface circuit. The processing module 3101 shown in Figure 31 can be the logic circuit 3201 in Figure 32.

[0571] The logic circuit 3201 and the input / output interface 3202 can also perform other steps executed by the data processing device in any embodiment and achieve corresponding beneficial effects, which will not be elaborated here.

[0572] Optionally, the logic circuit 3201 can be a processing device, the functions of which can be partially or entirely implemented in software.

[0573] Optionally, the data processing device may include a memory and a processor, wherein the memory is used to store a computer program, and the processor reads and executes the computer program stored in the memory to perform the corresponding processing and / or steps in any of the method embodiments.

[0574] Optionally, the data processing device may consist only of a processor. A memory for storing computer programs is located outside the processing device, and the processor is connected to the memory via circuitry / wires to read and execute the computer programs stored in the memory. The memory and processor may be integrated together or physically independent of each other.

[0575] Optionally, the data processing device may be one or more chips, or one or more integrated circuits. For example, the processing device may be one or more field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), system-on-chips (SoCs), central processing units (CPUs), network processors (NPs), digital signal processors (DSPs), microcontroller units (MCUs), programmable logic devices (PLDs), or other integrated chips, or any group of the above chips or processors.

[0576] Please refer to Figure 33, which shows the data processing device 3300 involved in the above embodiments provided in the embodiments of this application. The data processing device 3300 may be the data processing device in Figure 3A above, or the terminal device in Figure 3B above, etc.

[0577] As shown in Figure 33, the data processing device 3300 includes a bus 3302, a processor 3304, a memory 3306, and a communication interface 3308. The processor 3304, memory 3306, and communication interface 3308 communicate with each other via the bus 3302. The data processing device 3300 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the data processing device 3300.

[0578] Bus 3302 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 33, but this does not imply that there is only one bus or one type of bus. Bus 3302 can include pathways for transmitting information between various components of the data processing device 3300 (e.g., memory 3306, processor 3304, communication interface 3308).

[0579] The processor 3304 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0580] The memory 3306 may include volatile memory, such as random access memory (RAM). The processor 3304 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0581] The memory 3306 stores executable program code, and the processor 3304 executes this executable program code to implement the functions of the aforementioned acquisition module and processing module, thereby realizing the aforementioned data processing method. That is, the memory 3306 stores instructions for executing the data processing method.

[0582] The communication interface 3308 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the data processing device 3300 and other devices or communication networks.

[0583] This application also provides a computing device cluster for implementing the functions of the data processing device cluster described above. The computing device cluster includes at least one computing device. This computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0584] Please refer to Figure 34, which is a schematic diagram of a computing device cluster provided in this application. As shown in Figure 34, the computing device cluster includes at least one data processing device 3300. The memory 3306 of one or more data processing devices 3300 in the computing device cluster may store the same instructions for executing data processing methods.

[0585] In some possible implementations, the memory 3306 of one or more data processing devices 3300 in the computing device cluster may also store partial instructions for executing data processing methods. In other words, a combination of one or more data processing devices 3300 can jointly execute instructions for executing data processing methods.

[0586] It should be noted that the memory 3306 in different data processing devices 3300 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data processing device. That is, the instructions stored in the memory 3306 of different data processing devices 3300 can implement the functions of one or more of the aforementioned acquisition and processing modules.

[0587] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 35 illustrates one possible implementation. Figure 35 is a schematic diagram of another computing device cluster structure provided in this application. As shown in Figure 35, in the computing device cluster 3500, two data processing devices 3300A and 3300B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this type of possible implementation, the memory 3306 in data processing device 3300A stores instructions for executing the functions of the acquisition module. Simultaneously, the memory 3306 in data processing device 3300B stores instructions for executing the functions of the processing module.

[0588] It should be understood that the functions of the data processing device 3300A shown in Figure 35 can also be performed by multiple data processing devices 3300. Similarly, the functions of the data processing device 3300B can also be performed by multiple data processing devices 3300.

[0589] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0590] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0591] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0592] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions may be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, characterized in that, The method is applied to a multi-head attention network in a computing cluster, which includes N computing cards and performs multiple communication operations, including: reducescatter operation, alltoall operation and allgather operation, and the first computing card is any one of the N computing cards; The method includes: The first computing card performs the allgather operation on the first data to obtain the second data, where the first data is a submatrix of the feature matrix of the input data of the multi-head attention network; The first computing card performs QKV calculation on the second data and the first sub-model, where the first sub-model is a sub-matrix obtained by row-slicing and column-slicing the weight matrix of the multi-head attention network; The first computing card performs the reducescatter operation on the result of the QKV calculation to obtain third data, which is used to obtain the processing result of the first data.

2. The method according to claim 1, characterized in that, The method further includes: The first computing card performs the allgather operation on the KV matrix in the third data to obtain the fourth data; The Q matrix in the third data and the KV matrix in the fourth data are used to perform self-attention calculation to obtain the fifth data; The first computing card performs the allgather operation on the fifth data to obtain the sixth data; The sixth data is multiplied by the second sub-model to obtain the seventh data. The second sub-model is a sub-matrix obtained by row-cutting and column-cutting the weight matrix of the multi-head attention network. The reducescatter operation is performed on the seventh data to obtain the processing result of the first data.

3. The method according to any one of claims 1-2, characterized in that, The first data is the average column sub-matrix of the feature matrix of the input data of the multi-head attention network.

4. The method according to any one of claims 1-3, characterized in that, The first sub-model is a sub-matrix obtained based on a first segmentation scheme, the second sub-model is a sub-matrix obtained based on a second segmentation scheme, and the method further includes: Based on the first constraint, the second constraint, and the third constraint, the first segmentation scheme and the second segmentation scheme are calculated. Based on the first segmentation scheme and the second segmentation scheme respectively, the parameter matrix of the multi-head attention network is segmented to obtain a first sub-model and a second sub-model; The first constraint is satisfied: The second constraint is satisfied: A = p1 * p2 The third constraint is satisfied: A = p1′ * p2′ Where A represents the parallelism of the first parallel strategy. p1 is the number of rows in the first segmentation scheme. p2 is the number of columns in the first segmentation scheme. b, s, and h represent the batch size of the input data, the sequence length of the input data, and the hidden layer dimension of the multi-head attention network, respectively. p1' is the number of rows in the second segmentation scheme. p2' is the number of columns in the second segmentation scheme.

5. The method according to any one of claims 1-4, characterized in that, N = A * B * C; A represents the parallelism of the first parallel strategy, B represents the parallelism of the second parallel strategy, and C represents the parallelism of the third parallel strategy. A, B, and C are positive integers greater than 1. The method further includes: performing row and column slicing on the weight matrix of the multi-head attention network to obtain the A sub-models of the multi-head attention network, wherein the first sub-model is any one of the A sub-models of the multi-head attention network.

6. The method according to any one of claims 1-5, characterized in that, The method also includes: The feature matrix of the input data of the multi-head attention network is segmented along the sequence dimension to obtain the C sub-sequence data; The first data is obtained by averaging any one of the C subsequence data.

7. A data processing device, characterized in that, Includes an acquisition module and a processing module; The acquisition module is used to acquire first data, which is a submatrix of the feature matrix of the input data of the multi-head attention network. The processing module is used to perform the following operations: Perform the allgather operation on the first data to obtain the second data; The second data and the first sub-model are subjected to QKV calculation, where the first sub-model is a sub-matrix obtained by row-slicing and column-slicing the weight matrix of the multi-head attention network; The QKV calculation result is subjected to a reducescatter operation to obtain third data, which is used to obtain the processing result of the first data.

8. The data processing device according to claim 7, characterized in that, The processing module is also used to perform the following operations: The first computing card performs the allgather operation on the KV matrix in the third data to obtain the fourth data; The Q matrix in the third data and the KV matrix in the fourth data are used to perform self-attention calculation to obtain the fifth data; The first computing card performs the allgather operation on the fifth data to obtain the sixth data; The sixth data is multiplied by the second sub-model to obtain the seventh data. The second sub-model is a sub-matrix obtained by row-cutting and column-cutting the weight matrix of the multi-head attention network. The reducescatter operation is performed on the seventh data to obtain the processing result of the first data.

9. The data processing apparatus according to any one of claims 7-8, characterized in that, The first data is the average column sub-matrix of the feature matrix of the input data of the multi-head attention network.

10. The data processing apparatus according to any one of claims 7-9, characterized in that, The first sub-model is a sub-matrix obtained based on the first segmentation scheme, and the second sub-model is a sub-matrix obtained based on the second segmentation scheme. The processing module is also used to perform the following operations: Based on the first constraint, the second constraint, and the third constraint, the first segmentation scheme and the second segmentation scheme are calculated. Based on the first segmentation scheme and the second segmentation scheme respectively, the parameter matrix of the multi-head attention network is segmented to obtain a first sub-model and a second sub-model; The first constraint is satisfied: The second constraint is satisfied: A = p1 * p2 The third constraint is satisfied: A = p1′ * p2′ Where A represents the parallelism of the first parallel strategy. p1 is the number of rows in the first segmentation scheme. p2 is the number of columns in the first segmentation scheme. b, s, and h represent the batch size of the input data, the sequence length of the input data, and the hidden layer dimension of the multi-head attention network, respectively. p1' is the number of rows in the second segmentation scheme. p2' is the number of columns in the second segmentation scheme.

11. The data processing apparatus according to any one of claims 7-10, characterized in that, N = A * B * C; A represents the parallelism of the first parallel strategy, B represents the parallelism of the second parallel strategy, and C represents the parallelism of the third parallel strategy. A, B, and C are positive integers greater than 1. The processing module is further configured to perform the following operations: perform row and column slicing on the weight matrix of the multi-head attention network to obtain the A sub-models of the multi-head attention network, wherein the first sub-model is any one of the A sub-models of the multi-head attention network.

12. The data processing apparatus according to any one of claims 7-11, characterized in that, The processing module is also used to perform the following operations: The feature matrix of the input data of the multi-head attention network is segmented along the sequence dimension to obtain the C sub-sequence data; The first data is obtained by averaging any one of the C subsequence data.

13. A data processing device, characterized in that, It includes at least one processor coupled to a memory; the at least one processor is used to perform the method as described in any one of claims 1 to 6.

14. A computing device cluster, characterized in that, Includes at least one device as described in claim 13.

15. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 6.

16. A chip, characterized in that, The chip is used to perform the method as described in any one of claims 1 to 6.

17. A readable storage medium, characterized in that, The storage medium stores a computer program or instructions, which, when executed by a data processing device, implement the method as described in any one of claims 1 to 6.

18. A computer program product, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 6.

Images