Sharded parallel model training method, device, equipment and storage medium

Abstract

The present disclosure provides a sharded parallel model training method, apparatus, device and storage medium. The present disclosure relates to the field of artificial intelligence technology, specifically natural language processing, deep learning, computer vision, image processing and other technical fields. The specific scheme is: deploy the model to be trained on multiple computing devices; segment the sequence length variables in the input variables of the model to be trained to obtain the segmented data; perform sharded parallel training on multiple computing devices according to the segmented data to obtain the target model. According to the scheme of the present disclosure, a sharded parallel model training method is adopted to train long sequence models, which can reduce the amount of communication between devices, reduce the cross-device communication time, and improve the efficiency of model training.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically natural language processing, deep learning, computer vision, image processing and other technical fields. Background Art

[0002] The Transformer model is a neural network model based on the attention mechanism. The Transformer model mainly consists of two parts: the encoder and the decoder. The encoder is used to convert the input sequence into a vector representation of a fixed length, and the decoder is used to convert the vector into an output sequence. Between the encoder and the decoder, there are one or more self-attention networks to capture the correlation between the input sequence and the output sequence. The long-sequence Transformer model faces huge challenges in training, and the existing strategies for training large models generally have the problem of low training efficiency. Summary of the invention

[0003] The present invention provides a sharded parallel model training method, apparatus, device and storage medium.

[0004] According to a first aspect of the present disclosure, a sharded parallel model training method is provided, comprising:

[0005] Deploy the model to be trained on multiple computing devices;

[0006] Split the sequence length variable in the input variable of the training model to obtain the split data;

[0007] According to the segmented data, sharded parallel training is performed on multiple computing devices to obtain the target model.

[0008] According to a second aspect of the present disclosure, there is provided an image processing method, comprising:

[0009] Inputting the image sequence data to be processed into a trained target model, wherein the trained target model is obtained by training according to the training method of the first aspect;

[0010] According to the trained target model, at least one image processing including image classification, image recognition and image segmentation is performed on the image sequence data to be processed to obtain an image processing result.

[0011] According to a third aspect of the present disclosure, a natural language processing method is provided, comprising:

[0012] Inputting the first type of sequence to be processed into a trained target model, wherein the trained target model is obtained by training according to the training method of the first aspect;

[0013] According to the trained target model, at least one natural language processing including information extraction, text classification, text recognition, speech recognition, and question answering is performed on the first type of sequences to be processed to obtain a natural language processing result.

[0014] According to a fourth aspect of the present disclosure, a computer vision processing method is provided, comprising:

[0015] Inputting the second type of sequence to be processed into a trained target model, wherein the trained target model is obtained by training according to the training method of the first aspect;

[0016] According to the trained target model, at least one computer vision processing including image recognition, target detection, semantic segmentation, video understanding and image generation is performed on the second type of sequences to be processed to obtain a computer vision processing result.

[0017] According to a fifth aspect of the present disclosure, a sharded parallel model training device is provided, comprising:

[0018] A deployment module, used to deploy the model to be trained on multiple computing devices;

[0019] A segmentation module is used to segment the sequence length variables in the input variables of the training model to obtain segmented data;

[0020] The first training module is used to perform sharded parallel training on multiple computing devices according to the segmented data to obtain a target model.

[0021] According to a sixth aspect of the present disclosure, there is provided an image processing device, comprising:

[0022] A first input module, used for inputting the image sequence data to be processed into a trained target model, the trained target model being obtained by training according to the training method of the first aspect;

[0023] The image processing module is used to perform at least one image processing including image classification, image recognition, and image segmentation on the image sequence data to be processed according to the trained target model to obtain an image processing result.

[0024] According to a seventh aspect of the present disclosure, there is provided a natural language processing apparatus, comprising:

[0025] A second input module, used for inputting the first type of sequence to be processed into a trained target model, where the trained target model is obtained by training according to the training method of the first aspect;

[0026] The natural language processing module is used to perform at least one natural language processing including information extraction, text classification, text recognition, speech recognition, and question answering on the first type of sequence to be processed according to the trained target model to obtain a natural language processing result.

[0027] According to an eighth aspect of the present disclosure, there is provided a computer vision processing device, comprising:

[0028] A third input module, used for inputting the second type of sequence to be processed into a trained target model, where the trained target model is obtained by training according to the training method of the first aspect;

[0029] The computer vision processing module is used to perform at least one computer vision processing including image recognition, target detection, semantic segmentation, video understanding and image generation on the second type of sequence to be processed according to the trained target model to obtain a computer vision processing result.

[0030] According to the ninth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; a memory communicatively connected to the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the sharded parallel model training method provided in the first aspect and / or the image processing method provided in the second aspect and / or the natural language processing method provided in the third aspect and / or the computer vision processing method provided in the fourth aspect.

[0031] According to the tenth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the sliced ​​parallel model training method provided by the first aspect and / or the image processing method provided by the second aspect and / or the natural language processing method provided by the third aspect and / or the computer vision processing method provided by the fourth aspect.

[0032] According to the eleventh aspect of the present disclosure, a computer program product is provided, including a computer program stored on a storage medium, which, when executed by a processor, implements the sharded parallel model training method provided in the first aspect and / or the image processing method provided in the second aspect and / or the natural language processing method provided in the third aspect and / or the computer vision processing method provided in the fourth aspect.

[0033] According to the solution disclosed in the present invention, a sharded parallel-based model training method is adopted to train a long-sequence Transformer model, which can reduce the amount of communication between devices, reduce the cross-device communication time, and improve the efficiency of model training.

[0034] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

[0036] Figure 1 is a simple schematic diagram of a Transformer layer according to an embodiment of the present disclosure;

[0037] Figure 2 is a schematic diagram of a tensor parallel process according to an embodiment of the present disclosure;

[0038] Figure 3 It is a flowchart of a sharded parallel model training method according to an embodiment of the present disclosure;

[0039] Figure 4 is a schematic diagram of a sharding parallel process according to an embodiment of the present disclosure;

[0040] Figure 5 is a schematic diagram of Reshard communication according to an embodiment of the present disclosure;

[0041] Figure 6 is a schematic diagram of a communication architecture according to an embodiment of the present disclosure;

[0042] Figure 7 is a schematic diagram of sending device slice data according to an embodiment of the present disclosure;

[0043] Figure 8 is a schematic diagram of receiving device slice data according to an embodiment of the present disclosure;

[0044] Fig. 9 is a schematic diagram of multi-point transmission and reception of slice data according to an embodiment of the present disclosure;

[0045] Fig.10 is a flow chart of hybrid parallelism based on sharding parallelism and tensor parallelism according to an embodiment of the present disclosure;

[0046] Fig.11 is a flowchart of an image processing method according to an embodiment of the present disclosure;

[0047] Fig.12 is a communication schematic diagram of a graphics processor according to an embodiment of the present disclosure;

[0048] Fig.13 is a flowchart of a natural language processing method according to an embodiment of the present disclosure;

[0049] Fig.14 is a flowchart of a computer vision processing method according to an embodiment of the present disclosure;

[0050] Fig.15 is a structural diagram of a sharded parallel model training device according to an embodiment of the present disclosure;

[0051] Fig.16 is a schematic structural diagram of an image processing device according to an embodiment of the present disclosure;

[0052] Fig.17 is a schematic diagram of the structure of a natural language processing device according to an embodiment of the present disclosure;

[0053] Fig.18 is a structural schematic diagram of a computer vision processing device according to an embodiment of the present disclosure;

[0054] Fig.19 It is a scenario diagram of a sharded parallel model training method according to an embodiment of the present disclosure;

[0055] Fig. 20 is a schematic diagram of a scene of an image processing method according to an embodiment of the present disclosure;

[0056] Fig.21 is a schematic diagram of a scenario of a natural language processing method according to an embodiment of the present disclosure;

[0057] Fig. 22 is a schematic diagram of a scene of a computer vision processing method according to an embodiment of the present disclosure;

[0058] Fig.23 It is a structural diagram of an electronic device used to implement the sharded parallel model training method and / or image processing method and / or natural language processing method and / or computer vision processing method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0059] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0060] The terms "first", "second", and "third", etc. in the specification embodiments and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product, or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0061] Before introducing the technical solutions of the embodiments of the present disclosure, the technical terms and basic models that may be used in the present disclosure are further explained:

[0062] Transformer model: The Transformer model is a neural network model based on the attention mechanism, which is mainly used for natural language processing (NLP), such as language translation, text classification, etc. The Transformer model is mainly composed of multiple Transformer layers, each of which contains a self-attention network and a feed-forward network (FFN).

[0063] Self-attention mechanism: Self-attention mechanism is a method that calculates the correlation between each word and other words and combines them. No matter how far apart the words are in the sequence, this method can capture the relationship between the elements in the input sequence. In the Transformer model, the self-attention mechanism is used to calculate the influence of each element in the input sequence or output sequence on other elements.

[0064] Feedforward neural network: A feedforward neural network is a neural network based on a fully connected layer (FC layer) that is used to perform nonlinear transformations on input sequences. In the Transformer model, the feedforward neural network includes at least two fully connected layers.

[0065] Figure 1 A simple diagram of the Transformer layer is shown, Figure 1 As shown, the specific calculation can be divided into the following steps:

[0066] 1. The input of the current Transformer layer is the output of the previous Transformer layer. Suppose the input of the current Transformer layer is matrix A, and the size of matrix A is [batch_size, seq_len, num_head*head_dim], where batch_size represents the size of the data batch (batch size), seq_len represents the sequence length, num_head represents the number of heads of Self-Attention, and head_dim represents the size of the hidden layer corresponding to each head.

[0067] 2. Matrix A passes through the normalization layer (Layer Norm) to obtain matrix B. The size of matrix B is [batch_size, seq_len, num_head*head_dim]; B = LayerNorm1(A); where LayerNorm1 represents the normalization layer in Self-Attention.

[0068] 3. Perform three linear transformations on the B matrix to obtain three transformed matrices Q, K, and V; the sizes of Q, K, and V are all [batch_size, seq_len, num_head, head_dim]; Q = Linear1(B); K = Linear2(B); V = Linear3(B); where the matrices Q, K, and V are obtained after the matrix B passes through the query-key-value (QKV) layer, Linear1 represents the layer in Self-Attention where the matrix B is transformed to obtain the matrix Q, Linear2 represents the layer in Self-Attention where the matrix B is transformed to obtain the matrix K, and Linear3 represents the layer in Self-Attention where the matrix B is transformed to obtain the matrix V.

[0069] 4. Through the self-attention calculation layer (FlashAttention layer), self-attention calculation is realized to obtain a matrix C whose size is [batch_size, seq_len, num_head*head_dim]; C = FlashAttention(Q, K, V); where FlashAttention represents the self-attention calculation layer in Self-Attention.

[0070] 5. Perform a linear transformation and reproject the matrix C to obtain the matrix D, the size of which is [batch_size, seq_len, num_head*head_dim]; D = OutLinear(C); where OutLinear represents the linear transformation layer in Self-Attention.

[0071] 6. Calculate the residual connection (Residual Add) to obtain the matrix E, which is used as the input of the FFN layer. The size of the matrix E is [batch_size, seq_len, num_head*head_dim]; E = D + A; where E represents the output matrix of Self-Attention, A represents the input matrix of Self-Attention, and D represents the output matrix of the linear transformation layer in Self-Attention.

[0072] 7. Matrix E passes through the LayerNorm layer in FFN to obtain matrix F, the size of matrix F is [batch_size, seq_len, num_head*head_dim]; F = LayerNorm2(E); where LayerNorm2 represents the normalization layer in FFN.

[0073] 8. Matrix F passes through the first fully connected layer in FFN to obtain matrix G, the size of which is [batch_size, seq_len, 4*num_head*head_dim]; G=FFNLinear1(F); where FFNLinear1 represents the first fully connected layer in FFN.

[0074] 9. After the second fully connected layer in FFN, the matrix H is obtained, and the size of the matrix H is [batch_size, seq_len, num_head*head_dim]; H = FFNLinear2(G); where FFNLinear2 represents the second fully connected layer in FFN.

[0075] 10. Calculate the residual connection and get the matrix Y. The size of matrix Y is [batch_size, seq_len, num_head*head_dim]. Matrix Y is also the input of the next Transformer layer; Y = H + E; Y represents the output matrix of the current Transformer layer, H represents the output matrix of the second fully connected layer in FFN, and E represents the output matrix of Self-Attention.

[0076] For the Transformer layer, tensor model parallelism can be introduced, which can effectively reduce the use of video memory of a single computing device without introducing additional redundant calculations. Specifically: In the Self-Attention structure, the parameters of the QKV layer are "column-split" and the parameters of the linear transformation layer (Out Linear) are "row-split", and mathematically equivalent output results are obtained through AllreduceSum communication. The tensor model parallelism of the FFN structure is similar to the Self-Attention structure. The parameters of FFNLinear1 are "column-split" and the parameters of FFNLinear2 are "row-split", and finally the mathematically equivalent output results are obtained through AllreduceSum communication. Among them, AllreduceSum is an operation mode in the full reduction (Allreduce).

[0077] Figure 2 A schematic diagram of the tensor parallel process is shown, as Figure 2 As shown in the figure, a computing device includes mp sub-computing devices (also called internal devices) inside. On the mp sub-computing devices, the tensor model parallelism of the Transformer layer is implemented, so the tensor model parallel dimension is mp. The specific calculation process is divided into the following steps:

[0078] 1. The input of the current Transformer layer is the output of the previous Transformer layer. Suppose the input of the current Transformer layer is matrix A, and the size of matrix A is [batch_size, seq_len, num_head*head_dim].

[0079] 2. Matrix A passes through the LayerNorm layer to obtain matrix B. The size of matrix B is [batch_size, seq_len, num_head*head_dim]; B = LayerNorm1(A); where LayerNorm1 represents the normalization layer in Self-Attention.

[0080] 3. Perform three linear transformations on the B matrix to obtain three transformed matrices Q, K, and V. Since the parameter matrix of the three linear transformations is "column-split" and divided into mp parts, the sizes of the transformed matrices Q, K, and V are all [batch_size, seq_len, num_head / mp, head_dim]; Q = Linear1(B); K = Linear2(B); V = Linear3(B); Among them, matrices Q, K, and V are obtained after matrix B passes through the QKV layer, Linear1 represents the layer in Self-Attention where matrix B is transformed to obtain matrix Q, Linear2 represents the layer in Self-Attention where matrix B is transformed to obtain matrix K, and Linear3 represents the layer in Self-Attention where matrix B is transformed to obtain matrix V.

[0081] 4. Through the FlashAttention layer, self-attention calculation is realized to obtain the matrix C. The size of the matrix C is [batch_size, seq_len, num_head / mp*head_dim]; C = FlashAttention(Q, K, V); where FlashAttention represents the self-attention calculation layer in Self-Attention.

[0082] 5. Perform a linear transformation layer and reproject the matrix C to obtain the matrix D'. Since the parameter matrix of Outlinear is "row-split" and divided into mp parts, the size of the matrix D' obtained after the transformation is [batch_size, seq_len, num_head*head_dim]; D' = OutLinear (C); where OutLinear represents the linear transformation layer in Self-Attention.

[0083] 6. AllreduceSum communication is performed between the mp sub-computing devices to restore the original D matrix; D = AllreduceSum (D′); AllreduceSum represents the Allreduce sum operation;

[0084] 7. Calculate the residual connection and get the matrix E, which will be used as the input of FFN. The size of E is [batch_size, seq_len, num_head*head_dim]; where E = D + A; E represents the output matrix of Self-Attention, A represents the input matrix of Self-Attention, and D represents the output matrix of the linear transformation layer in Self-Attention.

[0085] 8. Matrix E passes through the LayerNorm layer in the FFN structure to obtain matrix F. The size of matrix F is [batch_size, seq_len, num_head*head_dim]; F = LayerNorm2(E); where LayerNorm2 represents the normalization layer in FFN.

[0086] 9. Matrix F passes through the first fully connected layer in the FFN structure to obtain matrix G. Since the parameter matrix of FFNLinear1 is "column-split" and divided into mp parts, the size of G after transformation is [batch_size, seq_len, 4*num_head*head_dim / mp]; G = FFNLinear1(F); where FFNLinear1 represents the first fully connected layer in FFN.

[0087] 10. The matrix G passes through the second fully connected layer in the FFN structure to obtain the matrix H. Since the parameter matrix of FFNLinear2 is "row split" and divided into mp parts, the matrix H' is obtained after the transformation, and the size of the matrix H' is [batch_size, seq_len, num_head*head_dim]; H'=FFNLinear2(G) where FFNLinear2 represents the second fully connected layer in FFN.

[0088] 11. AllreduceSum communication is performed between mp sub-computing devices to restore the original matrix H; H = AllreduceSum (H′); AllreduceSum represents the Allreduce sum operation;

[0089] 12. Calculate the residual connection and get the matrix Y. The size of matrix Y is [batch_size, seq_len, num_head*head_dim]. Matrix Y is also the input of the next Transformer layer; Y = H + E; Y represents the output matrix of the current Transformer layer, H represents the output matrix of the second fully connected layer in FFN, and E represents the output matrix of Self-Attention.

[0090] In tensor model parallelism, the data type is floating point format (Brain Float16, referred to as BF16) / half-precision floating-point (Half-precision floating-point, referred to as FP16). A Transformer layer calculation requires 4 AllreduceSums, that is, 2 forward and 2 reverse. The total communication volume is: 8*batch_size*seq_len*num_head*head_dim bytes. The bus bandwidth is Bw, and the AllreduceSum communication time is:

[0091]

[0092] When the tensor parallel dimension mp is large, (mp-1) / mp≈1, then the communication time T is:

[0093]

[0094] It can be seen that the communication time T is proportional to the sequence length seq_len, but has nothing to do with the size of mp. In the Transformer model that processes very long sequences, the sequence length seq_len is very large, and the communication time T accounts for too high a proportion. At the same time, it is impossible to reduce the communication time by increasing the parallel dimension mp of the tensor model.

[0095] In order to at least partially solve one or more of the above problems and other potential problems, the present disclosure proposes a model training method based on sharding parallelism. By cutting the sequence length, the communication volume between devices is reduced, and the communication time is inversely proportional to the sharding parallel dimension, while no redundant calculation is increased. In this way, the cross-device communication time of long sequences can be reduced and the efficiency of model training can be improved.

[0096] The present disclosure provides a sharded parallel model training method. Figure 3 It is a flowchart of a sharded parallel model training method according to an embodiment of the present disclosure. The sharded parallel model training method can be applied to a sharded parallel model training device. The sharded parallel model training device is located on an electronic device. The electronic device includes but is not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, and the server can be a cloud server or an ordinary server. For example, mobile devices include but are not limited to: mobile phones, tablet computers, etc. In some possible implementations, the sharded parallel model training method can also be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 3 As shown, the sharded parallel model training method includes:

[0097] S301: Deploy the model to be trained on multiple computing devices;

[0098] S302: segmenting the sequence length variable in the input variable of the training model to obtain segmented data;

[0099] S303: Perform sharded parallel training on multiple computing devices according to the segmented data to obtain a target model.

[0100] Among them, sharding means: splitting the sequence length variable (seq_len) in the input variable.

[0101] In the embodiments of the present disclosure, the multiple computing devices may be multiple servers or multiple terminal devices. For example, the multiple computing devices may be multiple image processing devices; for another example, the multiple computing devices may be multiple natural language processing devices; for another example, the multiple computing devices may be multiple computer vision processing devices. The above is only an exemplary description and is not intended to limit all possible types of multiple computing devices, but it is not exhaustive here.

[0102] In the disclosed embodiment, the model to be trained may be a Transformer model, specifically a Transformer model based on a long sequence. Here, the long sequence is not limited to a long text sequence, a long speech sequence, a long image sequence, a long video sequence, etc. The above is only an exemplary description and is not intended to limit all possible types of long sequences, but it is not exhaustive here.

[0103] In the embodiments of the present disclosure, multiple computing devices can communicate with each other, and the communication method may include cross-device communication. Exemplarily, computing device 1 transmits at least part of the data in the matrix Q output by the QKV layer to computing device 2; transmits at least part of the data in the matrix K output by the QKV layer to computing device 2, and transmits at least part of the data in the matrix V output by its QKV layer to computing device 2. Similarly, computing device 2 may also transmit part of the data in the matrix Q output by the QKV layer to computing device 1, transmit at least part of the data in the matrix K output by the QKV layer to computing device 1, and transmit at least part of the data in the matrix V output by the QKV layer to computing device 1.

[0104] In the embodiment of the present disclosure, in the process of performing sharded parallel training between multiple devices, it is necessary for the computing devices to communicate with each other. Preferably, the communication method adopted between the computing devices can be Reshard communication. The Reshard communication is a basic communication method. At the same time, Reshard is also a basic communication operator. The Reshard communication can perform communication by calling some communication libraries.

[0105] In the disclosed embodiment, multiple computing devices (denoted as Device0, Device1, ..., Device n) are provided. The computing devices here can be understood as nodes, which can be servers, terminal computing devices, etc. Sliced ​​parallel training is performed between Device0, Device1, ..., Device n. In addition, each computing device may include multiple sub-computing devices (also referred to as internal devices). For example, Device0 may include three sub-computing devices, namely Device01, Device02, and Device03.

[0106] In the disclosed embodiments, the function of the target model depends on the content of the input sequence and the training requirements. For example, when the content of the input sequence is text, the target model is a text processing model. For another example, when the content of the input sequence is speech, the target model is a speech processing model. For another example, when the content of the input sequence is an image, the target model is an image processing model. For another example, when the content of the input sequence is a video, the target model is a video processing model. The above is only an exemplary description and is not intended to limit all possible functions of the target model, but it is not exhaustive here.

[0107] In the disclosed embodiment, the input variables include multiple variables, such as batch_size, seq_len and num_head*head_dim. That is, the input variables can be represented from three dimensions: batch_size, seq_len and num_head*head_dim; batch_size represents the size of the data batch, seq_len represents the sequence length, num_head*head_dim represents the hidden layer size; num_head represents the number of heads of Self-Attention, and head_dim represents the hidden layer size corresponding to each head.

[0108] In the disclosed embodiment, the segmented data may be understood as the total sequence length being segmented into multiple subsequence lengths.

[0109] In the embodiment of the present disclosure, if there are sep computing devices, the sequence length variable seq_len is divided into sep parts. Here, the division method can be equal division or unequal division. The division method can be set or adjusted according to user needs or device performance.

[0110] In the embodiment of the present disclosure, if the model to be trained includes N Transformer layers, they are respectively recorded as the 1st Transformer layer, the 2nd Transformer layer, the 3rd Transformer layer, ..., the Nth Transformer layer. Specifically, the input of the 2nd Transformer layer is the output of the 1st Transformer layer, and so on, the input of the Nth Transformer layer is the output of the N-1th Transformer layer.

[0111] If sep=3, seq_len=[x1x2x3x4x5x6x7x8x9x10x11x12x13], divide seq_len into 3 parts, specifically [x1x2x3x4], [x5x6x7x8] and [x9x10x11x12x13], use [x1x2x3x4] as the input of the first Transformer layer of the model to be trained on Device0; use [x5x6x7x8] as the input of the first Transformer layer of the model to be trained on Device1; use [x9x10x11x12x13] as the input of the first Transformer layer of the model to be trained on Device2.

[0112] If shard parallelism is implemented on sep computing devices, the dimension of shard parallelism is sep. Figure 4 A schematic diagram of the sharding parallel process is shown, such as Figure 4 As shown, the specific calculation process is divided into the following steps:

[0113] 1. The input of the current Transformer layer is the output of the previous Transformer layer. Suppose the input of the current Transformer layer is matrix A, and the size of matrix A is [batch_size, seq_len / sep, num_head*head_dim].

[0114] 2. After the LayerNorm layer, we get the matrix B. The size of the matrix B is [batch_size, seq_len / sep, num_head*head_dim]; B = LayerNorm1(A); where LayerNorm1 represents the normalization layer in Self-Attention.

[0115] 3. Perform three linear transformations on the B matrix to obtain three transformed matrices Q′, K′, and V′. The matrix sizes of matrices Q′, K′, and V′ are all [batch_size, seq_len / sep, num_head, head_dim]; Q′=Linear1(B); K′=Linear2(B); V′=Linear3(B); Among them, matrices Q′, K′, and V′ are obtained after matrix B passes through the QKV layer. Linear1 represents the layer in Self-Attention where matrix B is transformed to obtain matrix Q′, Linear2 represents the layer in Self-Attention where matrix B is transformed to obtain matrix K′, and Linear3 represents the layer in Self-Attention where matrix B is transformed to obtain matrix V′.

[0116] 4. Since the FlashAttention layer requires a complete seq_len, the matrices Q′, K′, and V′ need to restore the normal seq_len dimension through communication between computing devices to obtain matrices Q, K, and V respectively. The dimensions of matrices Q, K, and V are all [batch_size, seq_len, num_head / sep, head_dim]. Among them, Q = Reshard(Q′); K = Reshard(K′); V = Reshard(V′); Among them, Reshard() represents reslicing transformation.

[0117] 5. Through the FlashAttention layer, self-attention calculation is realized to obtain the matrix C′, the size of the matrix C′ is [batch_size, seq_len, num_head / sep, head_dim]; where C′=FlashAttention(Q, K, V); FlashAttention represents the self-attention calculation layer in Self-Attention.

[0118] 6. Perform Reshard transformation to split seq_len on sep computing devices to obtain matrix C. The size of matrix C is [batch_size, seq_len / sep, num_head, head_dim]; where C = Reshard(C′); where Reshard() represents reslicing transformation.

[0119] 7. Perform a linear transformation layer and reproject the matrix C to obtain the matrix D. The size of the matrix D is [batch_size, seq_len / sep, num_head*head_dim]; where D = OutLinear(C); where OutLinear represents the linear transformation layer in Self-Attention.

[0120] 8. Calculate the residual connection and get the matrix E. The matrix E is used as the input of the FFN layer. The size of the matrix E is [batch_size, seq_len / sep, num_head*head_dim]; E=D+A; where E represents the output matrix of Self-Attention, A represents the input matrix of Self-Attention, and D represents the output matrix of the linear transformation layer in Self-Attention.

[0121] 9. Matrix E passes through the LayerNorm layer in FFN to obtain matrix F. The size of matrix F is [batch_size, seq_len / sep, num_head*head_dim]; where F = LayerNorm2(E); where LayerNorm2 represents the normalization layer in FFN.

[0122] 10. Matrix F passes through the first fully connected layer in the FFN structure to obtain matrix G. The size of matrix G is [batch_size, seq_len / sep, 4*num_head*head_dim]; G = FFNLinear1(F); where FFNLinear1 represents the first fully connected layer in FFN.

[0123] 11. The matrix G passes through the second fully connected layer in the FFN structure to obtain the matrix H, the size of the matrix H is [batch_size, seq_len / sep, num_head*head_dim]; where H=FFNLinear2(G); FFNLinear2 represents the second fully connected layer in FFN.

[0124] 12. Calculate the residual connection and get the matrix Y. The size of matrix Y is [batch_size, seq_len / sep, num_head*head_dim]. Matrix Y is also the input of the next Transformer layer; Y = H + E; Y represents the output matrix of the current Transformer layer, H represents the output matrix of the second fully connected layer in FFN, and E represents the output matrix of Self-Attention.

[0125] 13. Since the model parameters are not divided in shard parallelism, after the reverse calculation is completed, the gradients of the model parameters need to be synchronized on sep computing devices.

[0126] The technical solution of the disclosed embodiment adopts a sharded parallel model training method to train long-sequence Transformer models, compared with the tensor parallel training method, which can reduce the communication volume between devices, reduce the cross-device communication time, and improve the efficiency of model training.

[0127] In the disclosed embodiment, segmenting the sequence length variable in the input variable of the model to be trained to obtain segmented data includes: segmenting the training sample data for the model to be trained in a manner of at least partially dividing the sequence length into equal parts to obtain the segmented data. The segmented data is a partial data block in the training sample data.

[0128] The training sample data may be represented as a long sequence, and the long sequence is segmented to obtain a plurality of segmented subsequences, wherein at least partially segmented means that at least two subsequences have the same length.

[0129] In the embodiment of the present disclosure, the training sample data may include image sample data, video sample data, audio sample data, text sample data, etc. The above is only an exemplary description and is not intended to be a complete limitation on the training sample data.

[0130] In the embodiment of the present disclosure, the segmentation method may include an equal segmentation method or a unequal segmentation method.

[0131] The technical solution of the disclosed embodiment divides the training sample data for the model to be trained into equal parts in the length of the sequence, and obtains the divided data. In this way, by dividing the long sequence into parts and using the divided data as the input of the Transformer layer, the cross-device communication volume of the long sequence can be reduced, thereby reducing the cross-device communication time and improving the efficiency of model training.

[0132] In some embodiments, the sharded parallel model training method further includes: performing cross-device communication between multiple computing devices based on the segmented data.

[0133] Here, the cross-device communication may include sending data and receiving data.

[0134] Figure 5 A schematic diagram of Reshard communication is shown, Figure 5As shown in the figure, computing device 0 (denoted as Device0) and computing device 1 (denoted as Device1) are used to perform Reshard communication. Specifically, Device0 obtains [8, 9, 12, 13] from Device1; Device0 generates a new matrix [0, 1, 4, 5, 8, 9, 12, 13] based on [8, 9, 12, 13] obtained from Device1. Device1 obtains [2, 3, 6, 7] from Device0; Device0 generates a new matrix [2, 3, 6, 7, 10, 11, 14, 15] based on [2, 3, 6, 7] obtained from Device1. From the above example, it can be seen that the shape of the newly generated matrix has changed compared to the original matrix, but the number of elements has not changed. Therefore, for each device, different data can be obtained from different devices, combined with its own data, to regenerate a new matrix. That is, Q is regenerated based on Q′; K is regenerated based on K′; and V is regenerated based on V′.

[0135] In the embodiment of the present disclosure, the cross-device communication between multiple computing devices based on the split data may include: distributing sep copies of data to sep computing devices; the sep computing devices perform a first Reshard communication to restore seq_len / sep to seq_len; the sep computing devices perform a second Reshard communication to re-split seq_len into seq_len / sep.

[0136] Figure 6 A schematic diagram of the communication architecture is shown, such as Figure 6 As shown, computing device 0 (denoted as Device 0), computing device 1 (denoted as Device 1), computing device 2 (denoted as Device 2), and device 3 (denoted as Device 3). Computing device 0 communicates with computing device 1, computing device 2, and computing device 3, respectively. Among them, each computing device can be a server, a terminal, or between a terminal and a server, or an internally arranged processor, and the processor can be a graphics processing unit (Graphics Processing Unit, GPU), a central processing unit (Central Processing Unit, CPU), or a microprocessor. GPU can be a processor in a computer graphics card, which is composed of a motherboard connection device, a monitor connection device, a processor, and a memory, and is used to process the image signal sent by the CPU and then transmit it to the display. Specifically, it can be applied to the generation and image processing of 2D or 3D graphics. It should be noted that the number of computing devices can be set or adjusted according to demand.

[0137] Figure 7A schematic diagram of sending device slice data is shown, such as Figure 7 As shown, computing device 0 (denoted as Device 0), computing device 1 (denoted as Device 1), computing device 2 (denoted as Device 2), and computing device 3 (denoted as Device 3). Computing device 0 divides the data into 4 parts, namely, horizontal squares, diagonal squares, vertical squares, and blank squares. Computing device 0 sends the data in the diagonal squares to computing device 1; computing device 0 sends the data in the vertical squares to computing device 2; computing device 0 sends the data in the blank squares to computing device 3.

[0138] Figure 8 A schematic diagram of receiving device slice data is shown, such as Figure 8 As shown, computing device 0 (denoted as Device 0), computing device 1 (denoted as Device 1), computing device 2 (denoted as Device 2), and computing device 3 (denoted as Device 3). Computing device 0 receives the data of the diagonal square sent by computing device 1; computing device 0 receives the data of the vertical square sent by computing device 2; computing device 0 receives the data of the blank square sent by computing device 3.

[0139] Fig. 9 A schematic diagram of multi-point transmission and reception of slice data is shown, as shown in Fig. 9 As shown, computing device 0 (denoted as Device 0), computing device 1 (denoted as Device 1), computing device 2 (denoted as Device 2), and computing device 3 (denoted as Device 3). Computing device 0 sends data of horizontal squares to computing device 1, computing device 2, and computing device 3; computing device 1 sends data of diagonal squares to computing device 0, computing device 2, and computing device 3; computing device 2 sends data of horizontal squares to computing device 0, computing device 1, and computing device 3; computing device 3 sends data of blank squares to computing device 0, computing device 1, and computing device 2. Computing device 0, computing device 1, computing device 2, and computing device 3 receive data sent by other computing devices respectively.

[0140] based on Figures 6 to 9 The schematic diagram shown provides the following exemplary explanations of the various data communication forms involved:

[0141] Broadcast: When a server has calculated its part of the parameter data, it will send this part of the data to all other servers at the same time without splitting it in distributed training. This operation is called broadcast.

[0142] Scattering: When a server has calculated its own part of the parameter data, but the total parameter data on the server is too large, the data on this server can be divided into several data blocks (buffers) of equal size, and then one of the data blocks is sent to other servers according to the sequence (rank index). This is called scattering. Figure 7 , Figure 8 and Fig. 9 This communication method can be applied to both.

[0143] Aggregation: The data blocks on other servers are directly spliced ​​together, and the root server (Root Rank) obtains this data.

[0144] AllReduce: Perform a reduction operation (such as maximum value, sum) on the data on all servers, and then write the data to the root server or the memory of the specific server. The above-mentioned shard parallelism mainly involves AllReduce communication.

[0145] The technical solution of the disclosed embodiment performs cross-device communication between multiple computing devices based on the segmented data, which can reduce the time of cross-device communication and help improve the efficiency of parallel training.

[0146] In some embodiments, the model to be trained includes: at least one Transformer layer, each Transformer layer includes a self-attention mechanism network and a feedforward neural network cascaded together through a residual connection.

[0147] In the disclosed embodiment, the residual connection refers to: using the output of Self-Attention as the input of FFN.

[0148] In the disclosed embodiments, the self-attention mechanism network is also called a Self-Attention module or a Self-Attention structure; the feed-forward neural network is also called a FFN module or a FFN structure.

[0149] In the disclosed embodiment, the Self-Attention may include: a normalization layer (LayerNorm1), a QKV layer, a self-attention calculation layer (FlashAttention) and a linear transformation layer (OutLinear). The FFN includes: a normalization layer (LayerNorm2) and two fully connected layers (FFNLinear1+FFNLinear2).

[0150] The technical solution of the embodiment of the present disclosure can obtain a high-performance target model based on the structure of the model to be trained, which helps to improve the accuracy of the target model in the fields of image processing, natural language processing, and computer vision processing.

[0151] In some embodiments, the sharded parallel model training method further includes: restoring the dimension of the segmented sequence length variable to the original dimension of the sequence length variable by performing a first resharding transformation in Self-Attention.

[0152] In the embodiment of the present disclosure, there are three dimensional variables in the input variables, the first dimension is the batch size (expressed by batch_size), the second dimension is the sequence length (expressed by seq_len), and the third dimension is the model parameter (expressed by num_head*head_dim).

[0153] In the embodiment of the present disclosure, the original dimension is the dimension of the sequence length variable before segmentation. seq_len / sep represents the dimension of the sequence length variable after segmentation, and seq_len represents the original dimension of the sequence length variable.

[0154] In the embodiment of the present disclosure, the first Reshard transformation is to restore the split seq_len, that is, seq_len / sep to seq_len; the second Reshard transformation is to split the restored seq_len into seq_len / sep again.

[0155] by Figure 4Taking the sharded parallel flow chart shown in the figure as an example, first, the input matrix A of Self-Attention is expressed as [batch_size, seq_len / sep, num_head*head_dim]. Among them, batch_size represents the batch size, seq_len represents the sequence length, num_head represents the number of heads of the self-attention layer in the Transformer layer, head_dim represents the size of the hidden layer corresponding to each head, and sep represents the number of nodes. Secondly, the first Reshard transformation is performed, and the input is Q′=[batch_size, seq_len / sep, num_head, head_dim]; K′=[batch_size, seq_len / sep, num_head, head_dim]; V′=[batch_size, seq_len / sep, num_head, head_dim]. Thirdly, Q′, K′, and V′ are restored to the normal seq_len dimension through inter-device communication, and the matrices Q, K, and V are obtained respectively. Specifically, Q = [batch_size, seq_len, num_head / sep, head_dim]; K = [batch_size, seq_len, num_head / sep, head_dim]; V = [batch_size, seq_len, num_head / sep, head_dim]. Again, through the FlashAttention layer, the self-attention calculation is implemented to obtain the matrix C′, and the size of C′ is [batch_size, seq_len, num_head / sep, head_dim]. Finally, the second Reshard transformation is performed, and the input is C′ = [batch_size, seq_len, num_head / sep, head_dim]; according to C = Reshard(C′), the second Reshard transformation is performed to implement the segmentation of seq_len on sep computing devices, and the matrix C is obtained. The size of the matrix C is [batch_size, seq_len / sep, num_head, head_dim].

[0156] The technical solution of the embodiment of the present disclosure, in order to meet the input requirements of the FlashAttention layer, restores the dimension of the sequence length variable after segmentation to the original dimension of the sequence length variable, which can avoid calculation errors and improve the accuracy of training.

[0157] In some embodiments, the sharded parallel model training method also includes: reslicing the sequence length variables in the matrix output by the self-attention calculation layer by performing a second Reshard transformation in the self-attention mechanism network, and the self-attention mechanism network includes a self-attention calculation layer.

[0158] The technical solution of the embodiment of the present disclosure performs a second Reshard transformation in the self-attention mechanism network to re-segment the sequence length variables in the matrix output by the FlashAttention layer. In this way, by segmenting long sequences, the communication time of long sequences across devices can be reduced, thereby improving the efficiency of model training.

[0159] In some embodiments, by performing a first Reshard transformation in a self-attention mechanism network, the dimension of the split sequence length variable is restored to the original dimension of the sequence length variable, including: obtaining a fourth matrix based on the third matrix through the first Reshard transformation; wherein the dimension of the sequence length variable in the fourth matrix is ​​the original dimension of the sequence length variable, the third matrix is ​​obtained after the second matrix passes through the QKV layer, and the second matrix is ​​obtained by splitting the sequence length variable in the first matrix of the normalization layer input into the self-attention mechanism network.

[0160] In some embodiments, the sequence length variables in the first matrix (referred to as matrix A) of the normalization layer of the input self-attention mechanism network are segmented to obtain a second matrix (referred to as matrix B). After the second matrix passes through the QKV layer, a third matrix (matrix Q′K′V′) is obtained. The third matrix (matrix Q′K′V′) of each device is obtained through communication between computing devices. The first Reshard transformation is performed on the third matrix (matrix Q′K′V′) of each computing device to obtain a fourth matrix (matrix QKV) corresponding to each computing device, wherein the dimension of the sequence length variable in the fourth matrix is ​​the original dimension of the sequence length variable.

[0161] The technical solution of the embodiment of the present disclosure restores the dimension of the sequence length variable to the original dimension through the first reshard to meet the input conditions of the next layer, which can avoid errors in cross-device communication and help reduce the time cost of parallel communication.

[0162] In some embodiments, by performing a second Reshard transformation in the self-attention mechanism network, the sequence length variables in the matrix output by the self-attention calculation layer are re-divided, including: through the second Reshard transformation, a sixth matrix is ​​obtained based on the fifth matrix output by the self-attention calculation layer, wherein the dimension of the sequence length variable in the fifth matrix is ​​the original dimension of the sequence length variable.

[0163] In the embodiment of the present disclosure, the fifth matrix (matrix C') of each computing device is obtained through communication between computing devices; wherein the dimension of the sequence length variable in the fifth matrix is ​​the original dimension of the sequence length variable. A second Reshard transformation is performed on the fifth matrix of each computing device respectively to obtain a sixth matrix (matrix C) corresponding to each computing device, wherein the dimension of the sequence length variable in the sixth matrix is ​​the dimension after the sequence length variable is segmented.

[0164] The technical solution of the embodiment of the present disclosure obtains the sixth matrix based on the fifth matrix output by the self-attention calculation layer through the second Reshard transformation. In this way, the sequence length variable restored to the original dimension is re-segmented, which helps to reduce the time of cross-device communication and improve the efficiency of model training.

[0165] In some embodiments, the sharded parallel model training method further includes: synchronizing the parameter gradients of the model to be trained on each device during or after the reverse calculation.

[0166] In the embodiment of the present disclosure, if the model to be trained includes N Transformer layers, the reverse calculation is performed after the Nth Transformer layer is completed. The reverse calculation includes but is not limited to calculating the loss of classification and label, calculating the loss of prediction words and label words, etc.

[0167] In the embodiments of the present disclosure, model parameters will not be split during sharded parallel training.

[0168] Exemplarily, taking FFNlinear1 as an example, if sep devices synchronize the gradients of their respective FFNlinear1 parameters, then the gradients of the FFNlinear1 parameters corresponding to the sep devices are added and divided by sep to obtain the FFNlinear1 parameters of the target model.

[0169] The technical solution of the embodiment of the present disclosure synchronizes the parameter gradients of the model to be trained on each device during or after the reverse calculation, which can avoid redundant calculations, help reduce the time cost of parallel training, and thus help improve the efficiency of parallel training. The effect will be better if the parameter gradients of the model to be trained are synchronized on each device during the reverse calculation.

[0170] In some embodiments, training a model to be trained deployed on multiple computing devices includes: performing sharded parallel training between the multiple computing devices; and performing tensor parallel training within each of the multiple computing devices.

[0171] In the embodiments of the present disclosure, sharded parallel training is performed between computing devices, and tensor parallel training is performed within each computing device.

[0172] For example, within the device of Device0, tensor parallel training can be performed in sub-computing devices such as Device01, Device02, and Device03, and between sub-computing devices such as Device01, Device02, and Device03.

[0173] In the disclosed embodiment, the model parameter variables in the tensor parallel are the model's own parameters in the model to be trained, specifically including parameters of each layer such as the normalization layer, the QKV linear layer, and the FlashAttention layer.

[0174] The technical solution of the disclosed embodiment performs hybrid training through sharding parallelism and tensor parallelism, which can reduce the amount of calculation and the number of samples without increasing the amount of calculation, thereby helping to reduce redundant calculations, thereby helping to reduce the communication time across devices in long sequences and improving the efficiency of parallel training.

[0175] In some embodiments, training a to-be-trained model deployed on a plurality of computing devices according to a pre-training method includes: constructing a first communication group and a second communication group, wherein the first communication group is responsible for shard parallel communication, the second communication group is responsible for tensor parallel communication, and the first communication group and the second communication group can be used simultaneously;

[0176] The sharded parallel training is performed between multiple computing devices, including:

[0177] Sharded parallel training is performed between multiple computing devices based on the first communication group; during the reverse calculation process or after the reverse calculation is completed, the parameter gradient of the target model is synchronized on each computing device using the first communication group.

[0178] Among them, tensor parallel training is performed within each of the multiple computing devices, including:

[0179] Tensor parallel training is performed within each of the plurality of computing devices based on the second communication group.

[0180] In some embodiments, a first communication group (also called a SeP communication group) may be formed between the multiple computing devices, and Reshard communication may be adopted between the multiple computing devices.

[0181] In some embodiments, a second communication group (also called an MP communication group) may be formed between multiple internal devices of each computing device, and AllreduceSum communication may be adopted between the multiple internal devices.

[0182] In some embodiments, the first communication group and the second communication group do not affect each other, and the two communication groups can be used simultaneously. Assume that sep computing devices are used to train the model to be trained, and each computing device includes mp internal devices, then the parallelism of the first communication group is sep; the parallelism of the second communication group is mp.

[0183] In some embodiments, tensor parallelism is the parameter gradient that does not require synchronization of the model. Sharding parallelism is the parameter gradient that requires synchronization of the model. In practical applications, tensor parallelism and sharding parallelism do not affect each other.

[0184] Fig.10 A flow chart of hybrid parallelism based on sharding parallelism and tensor parallelism is shown, as Fig.10 As shown, the specific calculation process is divided into the following steps:

[0185] 1. The input of the current Transformer layer is the output of the previous Transformer layer. Suppose the input of the current Transformer layer is matrix A, and the size of matrix A is [batch_size, seq_len / sep, num_head*head_dim].

[0186] 2. After the LayerNorm layer, we get the matrix B. The size of the matrix B is [batch_size, seq_len / sep, num_head*head_dim]; B = LayerNorm1(A); where LayerNorm1 represents the normalization layer in Self-Attention.

[0187] 3. Perform three linear transformations on the B matrix respectively. The parameter matrices of the three Linears are "column split" in the MP communication group and divided into mp parts. Therefore, the three transformed matrices Q′, K′, and V′ are obtained. The sizes of the matrices Q′, K′, and V′ are all [batch_size, seq_len / sep, num_head / mp, head_dim]; Q′=Linear1(B); K′=Linear2(B); V′=Linear3(B); Among them, the matrices Q′, K′, and V′ are obtained after the matrix B passes through the QKV layer. Linear1 represents the layer in Self-Attention that transforms the matrix B to obtain the matrix Q′, Linear2 represents the layer in Self-Attention that transforms the matrix B to obtain the matrix K′, and Linear3 represents the layer in Self-Attention that transforms the matrix B to obtain the matrix V′.

[0188] 4. Since the FlashAttention layer requires a complete seq_len, Q′, K′, and V′ need to communicate between SeP communication groups to restore the normal seq_len dimension and obtain matrices Q, K, and V respectively. The dimensions of matrices Q, K, and V are [batch_size, seq_len, num_head / (sep*mp), head_dim]; Q = Reshard(Q′); K = Reshard(K′); V = Reshard(V′); where Reshard() represents reslicing transformation.

[0189] 5. Through the FlashAttention layer, self-attention calculation is realized to obtain the matrix C′. The size of the matrix C′ is [batch_size, seq_len, num_head / (sep*mp), head_dim]; C′=FlashAttention(Q,K,V); FlashAttention represents the self-attention calculation layer in Self-Attention.

[0190] 6. Perform Reshard transformation in the SeP communication group to implement segmentation of seq_len on sep devices to obtain a matrix C whose size is [batch_size, seq_len / sep, num_head / mp, head_dim]; C = Reshard(C′); wherein Reshard() represents the reslicing transformation.

[0191] 7. Perform a linear transformation layer and reproject the matrix C to obtain the matrix D′. Since the parameter matrix of Outlinear is "row-split" in the MP communication group and divided into mp parts, the size of D′ after transformation is [batch_size, seq_len / sep, num_head*head_dim]; D′=OutLinear(C); where OutLinear represents the linear transformation layer in Self-Attention.

[0192] 8. AllreduceSum communication is performed between MP communication groups to restore the original matrix D, whose size is [batch_size, seq_len / sep, num_head*head_dim]; D=AllreduceSum(D′); AllreduceSum represents the Allreduce summation operation.

[0193] 9. Calculate the residual connection and get the matrix E. The matrix E is used as the input of the FFN layer. The size of the matrix E is [batch_size, seq_len / sep, num_head*head_dim]; E=D+A; where E represents the output matrix of Self-Attention, A represents the input matrix of Self-Attention, and D represents the output matrix of the linear transformation layer in Self-Attention.

[0194] 10. Matrix E passes through the LayerNorm layer in FFN to obtain matrix F. The size of matrix F is [batch_size, seq_len / sep, num_head*head_dim]; where F = LayerNorm2(E); where LayerNorm2 represents the normalization layer in FFN.

[0195] 11. Matrix F passes through the first fully connected layer in the FFN structure to obtain matrix G. Since the parameter matrix of FFNLinear1 is "column-split" and divided into mp parts, the size of G after transformation is [batch_size, seq_len / sep, 4*num_head*head_dim / mp]; where G = FFNLinear1(F); where FFNLinear1 represents the first fully connected layer in FFN.

[0196] 12. The matrix G passes through the second fully connected layer in the FFN structure to obtain the matrix H. Since the parameter matrix of FFNLinear2 is "row-split" and divided into mp parts, the matrix H' is obtained after transformation, and the size of the matrix H' is [batch_size, seq_len / sep, num_head*head_dim]; H'=FFNLinear2(G); FFNLinear2 represents the second fully connected layer in FFN.

[0197] 13. AllreduceSum communication is performed between the MP communication groups to restore the original matrix H, where the size of H is [batch_size, seq_len / sep, num_head*head_dim]; H=AllreduceSum(H′); AllreduceSum represents the Allreduce summation operation.

[0198] 14. Calculate the residual connection and get the matrix Y. The size of matrix Y is [batch_size, seq_len / sep, num_head*head_dim]. Matrix Y is also the input of the next Transformer layer; Y = H + E; Y represents the output matrix of the current Transformer layer, H represents the output matrix of the second fully connected layer in FFN, and E represents the output matrix of Self-Attention.

[0199] 15. Since the model parameters are not divided in shard parallelism, the model parameter gradients need to be synchronized on the SeP communication group after the reverse is completed.

[0200] The technical solution of the disclosed embodiment performs shard parallel training between multiple computing devices; and performs tensor parallel training inside each of the multiple computing devices. In this way, by taking advantage of the high bandwidth within the computing device and the low bandwidth between computing devices, tensor parallelism is performed within the computing device and shard parallelism is performed between computing devices, thereby realizing mixed training of tensor parallelism and shard parallelism, which can effectively reduce communication time and improve the efficiency of parallel training.

[0201] In some embodiments, shard parallel training is performed between multiple computing devices; and tensor parallel training is performed within each of the multiple computing devices, including: in a self-attention mechanism network, by performing a first Reshard transformation in the self-attention mechanism network, the dimension of the split sequence length variable is restored to the original dimension of the sequence length variable; by performing a second Reshard transformation in the self-attention mechanism network, the sequence length variable in the matrix output by the self-attention calculation layer is re-split; and

[0202] The model parameter variables in the matrix passing through the QKV layer are split into columns; the model parameter variables in the linear transformation layer passing through the self-attention mechanism network are split into rows; in the feedforward neural network, the model parameter variables in the matrix passing through the first fully connected layer of the feedforward neural network are split into columns; the model parameter variables in the matrix passing through the second fully connected layer of the feedforward neural network are split into rows.

[0203] In some embodiments, sliced ​​parallel training is performed between multiple computing devices; and tensor parallel training is performed within each of the multiple computing devices, including: segmenting the sequence length variables in the first matrix (denoted as matrix A) of the normalization layer of the input self-attention mechanism network to obtain a second matrix (denoted as matrix B); segmenting the model parameter variables in the second matrix passing through the QKV layer by columns through the second communication group to obtain a third matrix (denoted as matrix Q′, K′, V′); wherein the sizes of Q′, K′, and V′ are all [batch_size, seq_len / sep, num_head / mp, head_dim]; and transforming the third matrix of each computing device into a fourth matrix corresponding to each computing device (denoted as matrix Q, K, V) through the first communication group, wherein the dimensions of the sequence length variables in the fourth matrix are the original dimensions of the sequence length variables (Q, K, V = [batch_size, seq_len, num_head / (sep*mp), head_dim m]); use the second communication group to obtain the fifth matrix (denoted as matrix C′) of each computing device; through the first communication group, transform the fifth matrix (denoted as matrix C) of each computing device into the sixth matrix corresponding to each computing device; wherein the dimension of the sequence length variable in the fifth matrix is ​​the original dimension of the sequence length variable, and the dimension of the sequence length variable in the sixth matrix is ​​the dimension after segmentation; through the second communication group, the model parameter variables in the linear transformation layer (OutLinear) are segmented into rows to obtain the seventh matrix (denoted as matrix D′, the size of matrix D′ is [batch_size, seq_len / sep, num_head*head_dim]); combined with the second communication group to communicate within each computing device, an eighth matrix (denoted as matrix D, the size of matrix D is [batch_size, seq_len / sep, num_head*head_dim]; wherein, D=AllreduceSum(D')) is obtained based on the seventh matrix.

[0204] In some embodiments, sliced ​​parallel training is performed between multiple computing devices; and tensor parallel training is performed within each of the multiple computing devices, including: in a feedforward neural network, the ninth matrix (denoted as matrix E) of the normalization layer of the input feedforward neural network is processed to obtain a tenth matrix (denoted as matrix F), and the ninth matrix is ​​obtained based on the eighth matrix (denoted as matrix D) and the first matrix (denoted as matrix A); through the second communication group, the model parameter variables in the tenth matrix passing through the first fully connected layer of the feedforward neural network are column-sliced ​​to obtain an eleventh matrix (denoted as matrix G); the model parameter variables in the eleventh matrix passing through the second fully connected layer of the feedforward neural network are row-sliced ​​to obtain a twelfth matrix (denoted as matrix H′); in combination with the second communication group, the communication between the devices within each computing device is combined to obtain a thirteenth matrix (denoted as matrix H) based on the twelfth matrix; based on the thirteenth matrix (denoted as matrix H) and the ninth matrix (denoted as matrix E), a fourteenth matrix (denoted as matrix Y) is obtained.

[0205] In sharded parallel training, the data type is BF16 / FP16, and one Transformer layer calculation requires 8 reshards (4 forward and 4 reverse). The total communication volume is: 16*batch_size*seq_len*num_head*head_dim / sep bytes. The bus bandwidth is Bw. The reshard communication time is:

[0206]

[0207] Since additional gradient AllreduceSum synchronization is required, the gradient type is FP32 and the total common gradient is 48*num_head 2 *head_dim 2 bytes, the communication time is:

[0208]

[0209] However, since gradient synchronization can overlap with reverse calculation and T2 is independent of the sequence length seq_len, we only consider T1 and increase the shard parallel dimension sep to effectively reduce the communication time.

[0210] The technical solution of the disclosed embodiment can perform hybrid training through tensor parallelism and sharding parallelism, which helps to avoid redundant calculations, thereby reducing the cross-device communication time of long or very long sequences and improving the efficiency of parallel training.

[0211] The present disclosure provides an image processing method. Fig.11: is a flowchart of an image processing method according to an embodiment of the present disclosure. The image processing method can be applied to an image processing device, and the image processing device can be applied to an electronic device. The electronic device includes but is not limited to a fixed device and / or a mobile device. For example, a fixed device includes but is not limited to a server, and the server can be a cloud server or an ordinary server. For example, a mobile device includes but is not limited to a mobile phone, a tablet computer, etc. In some possible implementations, the image processing method can also be implemented by a processor calling a computer-readable instruction stored in a memory. Fig.11 As shown, the image processing method includes:

[0212] S1101: inputting the image sequence data to be processed into a trained target model, wherein the trained target model is obtained by training according to the training method described above;

[0213] S1102: According to the trained target model, perform at least one image processing including image classification, image recognition, and image segmentation on the image sequence data to be processed to obtain an image processing result.

[0214] In some embodiments, the image sequence data to be processed can be obtained from an image data source; it can also be captured from a web page to obtain the image sequence data to be processed; it can also be obtained by intercepting multiple frames of images in a video to obtain the image sequence data to be processed.

[0215] Fig.12 A communication diagram of a graphics processor is shown, Fig.12 As shown, graphics processor 0 (referred to as GPU0), graphics processor 1 (referred to as GPU1), graphics processor 2 (referred to as GPU2), and graphics processor 3 (referred to as GPU3). Graphics processor 0 distributes data to graphics processor 1, graphics processor 2, and graphics processor 3.

[0216] In the embodiments of the present disclosure, when the target model is used for image recognition, the target model is an image recognition model. Specifically, a plurality of images to be recognized are obtained to obtain image sequence data to be recognized, and the image sequence data to be recognized is input into the image recognition model to obtain the recognition result of the image to be recognized output by the image recognition model. Exemplarily, a plurality of images to be recognized are obtained to obtain image sequence data to be recognized; the image sequence data to be recognized is an image of "rose", "lily", and "jasmine", the image sequence data to be recognized is input into the image recognition model to obtain the recognition results of rose, lily, and jasmine.

[0217] The technical solution of the disclosed embodiment uses a target model obtained through sliced ​​parallel or hybrid parallel training to perform image processing, which can provide a high-performance target model for the image processing field, thereby improving the speed and accuracy of image processing.

[0218] The present disclosure provides a natural language processing method. Fig.13 : is a flowchart of a natural language processing method according to an embodiment of the present disclosure, and the natural language processing method can be applied to a natural language processing device. The natural language processing device can be applied to electronic devices. The electronic devices include but are not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, and the servers can be cloud servers or ordinary servers. For example, mobile devices include but are not limited to mobile phones, tablet computers, etc. In some possible implementations, the natural language processing method can also be implemented by a processor calling computer-readable instructions stored in a memory. Fig.13 As shown, the natural language processing method includes:

[0219] S1301: inputting the first type of sequence to be processed into a trained target model, where the trained target model is obtained by training according to the training method described above;

[0220] S1302: According to the trained target model, perform at least one natural language processing including information extraction, text classification, text recognition, speech recognition, and question answering on the first type of sequences to be processed to obtain a natural language processing result.

[0221] In some embodiments, the first type of sequence to be processed can be obtained from a data source; or it can be captured from a web page to obtain the first type of sequence to be processed.

[0222] In some embodiments, the first type of sequence to be processed may be a plurality of text data or long text data, or may be a plurality of audio data or long audio data.

[0223] In the embodiments of the present disclosure, when the target model is used for text recognition, the target model is a text recognition model. Specifically, a plurality of data to be recognized is obtained to obtain sequence data to be recognized, and the sequence data to be recognized is input into the text recognition model to obtain the recognition result of the data to be recognized output by the text recognition model. Exemplarily, a plurality of data to be recognized is obtained to obtain a sequence of data to be recognized; the sequence of data to be recognized is represented as data of "roses in a vase", "lilies in a vase", and "jasmine in a vase", the sequence of data to be recognized is input into the text recognition model to obtain the text recognition result of "roses in a vase; lilies in a vase; jasmine in a vase".

[0224] The technical solution of the disclosed embodiment uses a target model obtained through sharded parallel or hybrid parallel training for natural language processing, which can provide a high-performance target model for the field of natural language processing and improve the speed and accuracy of natural language processing results.

[0225] The present disclosure provides a computer vision processing method. Fig.14 : is a flowchart of a computer vision processing method according to an embodiment of the present disclosure, and the computer vision processing method can be applied to a computer vision processing device. The computer vision processing device can be applied to electronic devices. The electronic devices include but are not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, and the servers can be cloud servers or ordinary servers. For example, mobile devices include but are not limited to mobile phones, tablet computers, etc. In some possible implementations, the computer vision processing method can also be implemented by a processor calling computer-readable instructions stored in a memory. Fig.14 As shown, the computer vision processing method includes:

[0226] S1401: inputting the second type of sequence to be processed into a trained target model, wherein the trained target model is obtained by training according to the training method described above;

[0227] S1402: According to the trained target model, at least one computer vision processing including image recognition, target detection, semantic segmentation, video understanding and image generation is performed on the second type of sequence to be processed to obtain a computer vision processing result.

[0228] In some embodiments, the second type of sequence to be processed may be a plurality of text data or long text data, a plurality of audio data or long audio data, or a plurality of image data.

[0229] In the embodiments of the present disclosure, when the target model is used for target detection, the target model is a target detection model. Specifically, a plurality of images to be detected are obtained to obtain sequence data of the images to be detected, and the sequence data of the images to be detected are input into the target detection model to obtain the detection results of the images to be detected output by the image detection model. Exemplarily, the images to be detected are images of cats and rabbits, and the images to be detected are input into the target detection model to obtain the output target detection results of cats and rabbits.

[0230] The technical solution of the disclosed embodiment uses a target model obtained through sliced ​​parallel or hybrid parallel training for image processing, which can provide a high-performance target model for the field of computer vision processing and improve the speed and accuracy of computer vision processing.

[0231] It should be understood that Figure 1 , Figure 2 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Fig. 9 , Fig.10 and Fig.12The schematic diagram shown is only exemplary and not restrictive, and it is scalable, and those skilled in the art can Figure 1 , Figure 2 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Fig. 9 , Fig.10 and Fig.12 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0232] The present disclosure provides a sharded parallel model training device, such as Fig.15 As shown, the sharded parallel model training device may include: a deployment module 1501, used to deploy the model to be trained on multiple computing devices; a segmentation module 1502, used to segment the sequence length variables in the input variables of the model to be trained to obtain segmented data; a first training module 1503, used to perform sharded parallel training on multiple computing devices according to the segmented data to obtain a target model.

[0233] In some embodiments, the segmentation module 1502 is used to: segment the training sample data for the model to be trained in a manner that the sequence length is at least partially equally divided to obtain segmented data, wherein the segmented data is a partial data block in the training sample data.

[0234] In some embodiments, the first training module 1503 includes: a cross-device communication submodule, which is used to perform cross-device communication between multiple computing devices based on the segmented data.

[0235] In some embodiments, the model to be trained includes: at least one Transformer layer, each Transformer layer includes a self-attention mechanism network and a feedforward neural network cascaded together through a residual connection.

[0236] In some embodiments, the sharded parallel model training device further includes: a recovery module 1504 ( Fig.15 ), which is used to restore the dimension of the segmented sequence length variable to the original dimension of the sequence length variable by performing the first Reshard transformation in the self-attention mechanism network.

[0237] In some embodiments, the sharded parallel model training device further includes: a resharding module 1505 ( Fig.15), which is used to reshard the sequence length variables in the matrix output by the self-attention calculation layer by performing a second Reshard transformation in the self-attention mechanism network, and the self-attention mechanism network includes the self-attention calculation layer.

[0238] In some embodiments, the recovery module 1504 ( Fig.15 (not shown in the figure), which is used to obtain a fourth matrix based on the third matrix through the first Reshard transformation; wherein the dimension of the sequence length variable in the fourth matrix is ​​the original dimension of the sequence length variable, the third matrix is ​​obtained after the second matrix passes through the QKV layer, and the second matrix is ​​obtained by dividing the sequence length variable in the first matrix of the normalization layer input into the self-attention mechanism network.

[0239] In some embodiments, the resegmentation module 1505 ( Fig.15 (not shown) is used to obtain a sixth matrix based on the fifth matrix output by the self-attention calculation layer through a second Reshard transformation, wherein the dimension of the sequence length variable in the fifth matrix is ​​the original dimension of the sequence length variable.

[0240] In some embodiments, the sharded parallel model training device further includes: a synchronization module 1506 ( Fig.15 ), which is used to synchronize the parameter gradients of the model to be trained on each computing device during or after the reverse calculation.

[0241] In some embodiments, the sharded parallel model training device further includes: a second training module 1507 ( Fig.15 ), for performing tensor parallel training within each of the multiple computing devices.

[0242] In some embodiments, the sharded parallel model training device further includes: a construction module 1508 ( Fig.15 ), for constructing a first communication group and a second communication group, wherein the first communication group is responsible for shard parallel communication, and the second communication group is responsible for tensor parallel communication, and the first communication group and the second communication group can be used simultaneously. The first training module 1503 is also used to perform shard parallel training between multiple computing devices based on the first communication group; the second training module 1507 ( Fig.15 ), and is further used to perform tensor parallel training within each of the plurality of computing devices based on the second communication group.

[0243] In some embodiments, the first training module 1503 is also used to restore the dimension of the segmented sequence length variable to the original dimension of the sequence length variable by performing the first Reshard transformation in the self-attention mechanism network; and re-segment the sequence length variable in the matrix output by the self-attention calculation layer by performing the second Reshard transformation in the self-attention mechanism network. In some embodiments, the second training module 1507 ( Fig.15 (not shown in the figure), and is also used to split the model parameter variables in the matrix that passes through the QKV layer into columns; to split the model parameter variables in the linear transformation layer that passes through the self-attention mechanism network into rows; in the feedforward neural network, to split the model parameter variables in the matrix that passes through the first fully connected layer of the feedforward neural network into columns; and to split the model parameter variables in the matrix that passes through the second fully connected layer of the feedforward neural network into rows.

[0244] Those skilled in the art should understand that the functions of each processing module in the sharded parallel model training device of the embodiment of the present disclosure can be understood with reference to the relevant description of the aforementioned sharded parallel model training method, and each processing module in the sharded parallel model training device of the embodiment of the present disclosure can be realized by an analog circuit that realizes the function of the embodiment of the present disclosure, or by running software that executes the function of the embodiment of the present disclosure on an electronic device.

[0245] The sharded parallel model training device of the disclosed embodiment adopts a sharded parallel model training method to train a long sequence Transformer model, which can reduce the cross-device communication time and improve the efficiency of model training.

[0246] The present disclosure provides an image processing device, such as Fig.16 As shown, the image processing device includes: a first input module 1601, used to input the image sequence data to be processed into a trained target model, and the trained target model is obtained by training according to the training method described above; an image processing module 1602, used to perform at least one image processing including image classification, image recognition, and image segmentation on the image sequence data to be processed according to the trained target model.

[0247] Those skilled in the art should understand that the functions of each processing module in the image processing device of the embodiment of the present disclosure can be understood with reference to the relevant description of the aforementioned image processing method, and each processing module in the image processing device of the embodiment of the present disclosure can be realized by an analog circuit that realizes the functions described in the embodiment of the present disclosure, or can be realized by running software that executes the functions described in the embodiment of the present disclosure on an electronic device.

[0248] The image processing device of the embodiment of the present disclosure can improve the speed and accuracy of image processing.

[0249] The present disclosure provides a natural language processing device, such as Fig.17 As shown, the natural language processing device includes: a second input module 1701, used to input the first type of sequence to be processed into a trained target model, and the trained target model is obtained by training according to the training method described above; a natural language processing module 1702, used to perform at least one natural language processing including information extraction, text classification, text recognition, speech recognition, and question answering on the first type of sequence to be processed according to the trained target model to obtain a natural language processing result.

[0250] Those skilled in the art should understand that the functions of each processing module in the natural language processing device of the embodiment of the present disclosure can be understood with reference to the relevant description of the aforementioned natural language processing method. Each processing module in the natural language processing device of the embodiment of the present disclosure can be realized by an analog circuit that implements the functions described in the embodiment of the present disclosure, or can be realized by running software that executes the functions described in the embodiment of the present disclosure on an electronic device.

[0251] The natural language processing device of the embodiment of the present disclosure can improve the speed and accuracy of natural language processing.

[0252] The present disclosure provides a computer vision processing device, such as Fig.18 As shown, the computer vision processing device includes: a third input module 1801, used to input the second type of sequence to be processed into a trained target model, and the trained target model is obtained by training according to the above-mentioned training method; a computer vision processing module 1802, used to perform at least one computer vision processing including image recognition, target detection, semantic segmentation, video understanding and image generation on the second type of sequence to be processed according to the trained target model to obtain a computer vision processing result.

[0253] Those skilled in the art should understand that the functions of each processing module in the computer vision processing device of the embodiments of the present disclosure can be understood with reference to the relevant description of the aforementioned computer vision processing method. Each processing module in the computer vision processing device of the embodiments of the present disclosure can be realized by an analog circuit that implements the functions described in the embodiments of the present disclosure, or can be realized by running software that executes the functions described in the embodiments of the present disclosure on an electronic device.

[0254] The computer vision processing device of the embodiment of the present disclosure can improve the speed and accuracy of computer vision processing.

[0255] The embodiment of the present disclosure provides a schematic diagram of a scenario of sharded parallel model training, such as Fig.19 shown.

[0256] As mentioned above, the sharded parallel model training method provided by the embodiment of the present disclosure is applied to electronic devices. Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices.

[0257] Specifically, the electronic device can perform the following operations:

[0258] Deploy the model to be trained on multiple computing devices;

[0259] Split the sequence length variable in the input variable of the training model to obtain the split data;

[0260] According to the segmented data, sharded parallel training is performed on multiple computing devices to obtain the target model.

[0261] The model to be trained can be obtained from a data source. The data source can be various forms of data storage devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The data source can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. In addition, the data source and the user terminal can be the same device.

[0262] It should be understood that Fig.19 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Fig.19 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0263] The present disclosure provides a schematic diagram of an image processing scenario, such as Fig. 20 shown.

[0264] As mentioned above, the image processing method provided by the embodiment of the present disclosure is applied to electronic devices. Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices.

[0265] Specifically, the electronic device can perform the following operations:

[0266] Inputting the image sequence data to be processed into the trained target model, the trained target model is obtained by training according to the training method described above;

[0267] According to the trained target model, at least one image processing including image classification, image recognition and image segmentation is performed on the image sequence data to be processed to obtain an image processing result.

[0268] The image sequence data to be processed may be obtained from an image data source. The image data source may be a data storage device in various forms, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The image data source may also represent various forms of mobile devices, such as a personal digital assistant, a cellular phone, a smart phone, a wearable device, and other similar computing devices. In addition, the image data source and the user terminal may be the same device.

[0269] It should be understood that Fig. 20 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Fig. 20 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0270] The present disclosure provides a natural language processing scenario diagram, such as Fig.21 shown.

[0271] As mentioned above, the natural language processing method provided by the embodiment of the present disclosure is applied to electronic devices. Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices.

[0272] Specifically, the electronic device can perform the following operations:

[0273] Inputting the first type of sequence to be processed into the trained target model, the trained target model is obtained by training according to the training method described above;

[0274] According to the trained target model, at least one natural language processing including information extraction, text classification, text recognition, speech recognition, and question answering is performed on the first type of sequences to be processed to obtain a natural language processing result.

[0275] The first type of sequence to be processed can be obtained from a data source. The data source can be various forms of data storage devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The data source can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. In addition, the data source and the user terminal can be the same device.

[0276] It should be understood that Fig.21 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Fig.21 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0277] The present disclosure provides a schematic diagram of a scene for computer vision processing, such as Fig. 22 shown.

[0278] As mentioned above, the computer vision processing method provided by the embodiments of the present disclosure is applied to electronic devices. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices.

[0279] Specifically, the electronic device can perform the following operations:

[0280] Inputting the second type of sequence to be processed into the trained target model, the trained target model is obtained by training according to the training method described above;

[0281] According to the trained target model, at least one computer vision processing including image recognition, target detection, semantic segmentation, video understanding and image generation is performed on the second type of sequences to be processed to obtain a computer vision processing result.

[0282] The second type of sequence to be processed can be obtained from a data source. The data source can be various forms of data storage devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The data source can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. In addition, the data source and the user terminal can be the same device.

[0283] It should be understood that Fig. 22The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Fig. 22 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0284] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0285] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0286] Fig.23 A schematic block diagram of an example electronic device 2300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0287] like Fig.23 As shown, the device 2300 includes a computing unit 2301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 2302 or a computer program loaded from a storage unit 2308 to a random access memory (RAM) 2303. In RAM 2303, various programs and data required for the operation of the device 2300 can also be stored. The computing unit 2301, ROM 2302 and RAM 2303 are connected to each other via a bus 2304. An input / output (I / O) interface 2305 is also connected to the bus 2304.

[0288] A number of components in the device 2300 are connected to the I / O interface 2305, including: an input unit 2306, such as a keyboard, a mouse, etc.; an output unit 2307, such as various types of displays, speakers, etc.; a storage unit 2308, such as a disk, an optical disk, etc.; and a communication unit 2309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 2309 allows the device 2300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0289] The computing unit 2301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 2301 include, but are not limited to, a central processing unit CPU, a graphics processing unit (Graphics Processing Unit, GPU), various dedicated artificial intelligence (Artificial Intelligence, AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (Digital Signal Processor, DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 2301 performs the various methods and processes described above, such as a sharded parallel model training method and / or an image processing method and / or a natural language processing method and / or a computer vision processing method. For example, in some embodiments, a sharded parallel model training method and / or an image processing method and / or a natural language processing method and / or a computer vision processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 2308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 2300 via ROM 2302 and / or a communication unit 2309. When the computer program is loaded into RAM 2303 and executed by the computing unit 2301, one or more steps of the sharded parallel model training method and / or image processing method and / or natural language processing method and / or computer vision processing method described above may be performed. Alternatively, in other embodiments, the computing unit 2301 may be configured to execute the sharded parallel model training method and / or image processing method and / or natural language processing method and / or computer vision processing method by any other appropriate means (e.g., by means of firmware).

[0290] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0291] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0292] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories, read-only memories, erasable programmable read-only memories (EPROM), flash memories, optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0293] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0294] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0295] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on the respective computers and having a client and server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0296] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0297] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A sharded parallel model training method, comprising: Deploy the model to be trained on multiple computing devices; Segmenting the sequence length variable in the input variable of the model to be trained to obtain segmented data; Performing sharded parallel training on the multiple computing devices according to the segmented data to obtain a target model; The sequence length variables in the input variables of the model to be trained are segmented, including: performing two reshard transformations before and after the self-attention calculation layer in the self-attention mechanism network of the model to be trained; Among them, two Reshard transformations are performed before and after the self-attention calculation layer in the self-attention mechanism network of the model to be trained, including: by performing the first Reshard transformation in the self-attention mechanism network, the dimension of the split sequence length variable in the input matrix of the self-attention calculation layer is restored to the original dimension of the sequence length variable; by performing the second Reshard transformation in the self-attention mechanism network, the sequence length variable in the matrix output by the self-attention calculation layer is re-split.

2. The method according to claim 1, wherein: The segmenting of the sequence length variable in the input variable of the model to be trained to obtain segmented data includes: The training sample data for the model to be trained is segmented in a manner of at least partially dividing the sequence length into equal parts to obtain the segmented data, wherein the segmented data is a partial data block in the training sample data.

3. The method according to claim 2, further comprising: Cross-device communication is performed between the multiple computing devices based on the segmented data.

4. The method according to any one of claims 1 to 3, wherein: The model to be trained includes: at least one Transformer layer, each Transformer layer includes a self-attention mechanism network and a feedforward neural network cascaded together through a residual connection.

5. The method according to claim 1, wherein: By performing the first Reshard transformation in the self-attention mechanism network, the dimension of the segmented sequence length variable is restored to the original dimension of the sequence length variable, including: Through the first Reshard transformation, a fourth matrix is ​​obtained based on the third matrix; wherein, the dimension of the sequence length variable in the fourth matrix is ​​the original dimension of the sequence length variable, the third matrix is ​​obtained after the second matrix passes through the query-key-value pair attention QKV layer, and the second matrix is ​​obtained by splitting the sequence length variable in the first matrix of the normalization layer input into the self-attention mechanism network.

6. The method according to claim 1, wherein: By performing a second Reshard transformation in the self-attention mechanism network, the sequence length variables in the matrix output by the self-attention calculation layer are re-sliced, including: Through the second Reshard transformation, a sixth matrix is ​​obtained based on the fifth matrix output by the self-attention calculation layer, wherein the dimension of the sequence length variable in the fifth matrix is ​​the original dimension of the sequence length variable.

7. The method according to claim 4, further comprising: During the reverse calculation process or after the reverse calculation is completed, the parameter gradients of the model to be trained are synchronized on each computing device.

8. The method according to claim 1, further comprising: Tensor parallel training is performed within each of the plurality of computing devices.

9. The method according to claim 8, further comprising: Constructing a first communication group and a second communication group, wherein the first communication group is responsible for shard parallel communication, the second communication group is responsible for tensor parallel communication, and the first communication group and the second communication group can be used simultaneously; The performing of sharded parallel training on the plurality of computing devices includes: performing the sharded parallel training among the plurality of computing devices based on the first communication group; The method further comprises: performing tensor parallel training within each of the plurality of computing devices, comprising: The tensor parallel training is performed within each of the plurality of computing devices based on the second communication group.

10. The method according to claim 8, wherein: Performing tensor parallel training within each of the plurality of computing devices includes: The model parameter variables in the matrix passing through the QKV layer are split into columns; the model parameter variables in the linear transformation layer passing through the self-attention mechanism network are split into rows; In a feedforward neural network, the model parameter variables in the matrix passing through the first fully connected layer of the feedforward neural network are split into columns; and the model parameter variables in the matrix passing through the second fully connected layer of the feedforward neural network are split into rows.

11. An image processing method, comprising: Inputting the image sequence data to be processed into a trained target model, wherein the trained target model is obtained by training according to any one of the training methods of claims 1 to 10; According to the trained target model, at least one image processing including image classification, image recognition and image segmentation is performed on the image sequence data to be processed to obtain an image processing result.

12. A natural language processing method, comprising: Inputting the first type of sequence to be processed into a trained target model, wherein the trained target model is obtained by training according to any one of the training methods of claims 1 to 10; According to the trained target model, at least one natural language processing including information extraction, text classification, text recognition, speech recognition, and question answering is performed on the first type of sequences to be processed to obtain a natural language processing result.

13. A computer vision processing method, comprising: Inputting the second type of sequence to be processed into a trained target model, wherein the trained target model is obtained by training according to any one of the training methods of claims 1 to 10; According to the trained target model, at least one computer vision processing including image recognition, target detection, semantic segmentation, video understanding and image generation is performed on the second type of sequence to be processed to obtain a computer vision processing result.

14. A sharded parallel model training device, comprising: A deployment module, used to deploy the model to be trained on multiple computing devices; A segmentation module, used for segmenting the sequence length variable in the input variable of the model to be trained to obtain segmented data; A first training module, configured to perform sharded parallel training on the plurality of computing devices according to the segmented data to obtain a target model; The segmentation module is further used to segment the sequence length variables in the input variables of the model to be trained, including: performing two reshard transformations before and after the self-attention calculation layer in the self-attention mechanism network of the model to be trained; Among them, the segmentation module includes: a recovery module, which is used to restore the segmented sequence length variable dimension in the input matrix of the self-attention calculation layer to the original dimension of the sequence length variable by performing the first Reshard transformation in the self-attention mechanism network; a re-segmentation module, which is used to re-segment the sequence length variable in the matrix output by the self-attention calculation layer by performing the second Reshard transformation in the self-attention mechanism network.

15. The device according to claim 14, wherein: The segmentation module is used for: The training sample data for the model to be trained is segmented in a manner of at least partially dividing the sequence length into equal parts to obtain the segmented data, wherein the segmented data is a partial data block in the training sample data.

16. The device according to claim 15, wherein: The first training module includes: The cross-device communication submodule is used to perform cross-device communication between the multiple computing devices based on the segmented data.

17. The device according to any one of claims 14 to 16, wherein: The model to be trained includes: at least one Transformer layer, each Transformer layer includes a self-attention mechanism network and a feedforward neural network cascaded together through a residual connection.

18. The device according to claim 14, wherein: The recovery module is used to obtain a fourth matrix based on the third matrix through the first Reshard transformation; wherein the dimension of the sequence length variable in the fourth matrix is ​​the original dimension of the sequence length variable, the third matrix is ​​obtained after the second matrix passes through the QKV layer, and the second matrix is ​​obtained by dividing the sequence length variable in the first matrix of the normalization layer input into the self-attention mechanism network.

19. The device according to claim 14, wherein: The resharding module is used to obtain a sixth matrix based on the fifth matrix output by the self-attention calculation layer through a second Reshard transformation, wherein the dimension of the sequence length variable in the fifth matrix is ​​the original dimension of the sequence length variable.

20. The apparatus of claim 17, further comprising: The synchronization module is used to synchronize the parameter gradients of the model to be trained on each computing device during or after the reverse calculation.

21. The apparatus of claim 14, further comprising: A second training module is used to perform tensor parallel training within each of the multiple computing devices.

22. The apparatus according to claim 21, further comprising: A construction module, used to construct a first communication group and a second communication group, wherein the first communication group is responsible for shard parallel communication, the second communication group is responsible for tensor parallel communication, and the first communication group and the second communication group can be used simultaneously; The first training module is further used to perform the sharded parallel training between the multiple computing devices based on the first communication group; the second training module is further used to perform the tensor parallel training within each of the multiple computing devices based on the second communication group.

23. The device according to claim 21, wherein The second training module is used to: The model parameter variables in the matrix passing through the QKV layer are split into columns; the model parameter variables in the linear transformation layer passing through the self-attention mechanism network are split into rows; in the feedforward neural network, the model parameter variables in the matrix passing through the first fully connected layer of the feedforward neural network are split into columns; the model parameter variables in the matrix passing through the second fully connected layer of the feedforward neural network are split into rows.

24. An image processing device, comprising: A first input module, used for inputting the image sequence data to be processed into a trained target model, wherein the trained target model is obtained by training according to any one of the training methods of claims 1-10; The image processing module is used to perform at least one image processing including image classification, image recognition, and image segmentation on the image sequence data to be processed according to the trained target model to obtain an image processing result.

25. A natural language processing device, comprising: A second input module, used for inputting the first type of sequence to be processed into a trained target model, wherein the trained target model is obtained by training according to any one of the training methods of claims 1-10; A natural language processing module is used to perform at least one natural language processing including information extraction, text classification, text recognition, speech recognition, and question answering on the first type of sequence to be processed according to the trained target model to obtain a natural language processing result.

26. A computer vision processing device, comprising: A third input module, used for inputting the second type of sequence to be processed into a trained target model, wherein the trained target model is obtained by training according to any one of the training methods of claims 1-10; A computer vision processing module is used to perform at least one computer vision processing including image recognition, target detection, semantic segmentation, video understanding and image generation on the second type of sequence to be processed according to the trained target model to obtain a computer vision processing result.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

29. A computer program product, comprising a computer program stored on a storage medium, the computer program implementing the method according to any one of claims 1 to 13 when executed by a processor.

Images