Large model training method, device and equipment based on causal attention lossless acceleration algorithm

Abstract

The embodiment of the invention discloses a large model training method, device and equipment based on a causal attention lossless acceleration algorithm. The large model training method comprises the steps of obtaining training data to be trained; based on the training data to be trained, generating a Query matrix and a Key matrix; on the basis of a preset number of blocks, the Query matrix and the Key matrix are blocked, and a sub-matrix network is obtained; calculating the sub-blocks in the sub-matrix network according to a preset linear combination rule according to a preset number of structured matrix multiplication to obtain a multiplication result; performing mask multiplication operation on the multiplication result to obtain a mask result; performing addition operation on the multiplication result and the mask result to obtain a mask attention matrix; and based on the mask attention matrix, training to obtain a target large model.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of large model, and particularly relates to a large model training method, device and equipment based on causal attention lossless acceleration algorithm. BACKGROUND

[0002] In recent years, the Transformer architecture has become the core technology framework in multiple fields such as natural language processing (NLP), computer vision (CV) and generative artificial intelligence (such as large language model LLM). Compared with traditional recurrent neural network (RNN) and convolutional neural network (CNN), Transformer can more efficiently process sequence data in parallel and has a significant advantage in modeling long-distance dependencies. Among them, causal attention is a key mechanism in autoregressive Transformer models, especially in GPT series models, which plays a crucial role in building understanding and generation capabilities of context.

[0003] In the prior art, the causal attention mechanism needs to calculate the following matrix: mask(QK^T), where Q and K are feature matrices with size L*d, the elements of the lth row of the Q matrix (i.e. Query matrix) and the K matrix (i.e. Key matrix) represent the "query hidden features and "key" hidden features of the lth word in the current text, which will be used to model the context correlation. QK^T is a real matrix with size L*L, L is the sequence length, and d is the feature dimension. mask() is a lower triangular mask that sets the upper triangular part (excluding the diagonal element) of the matrix to 0. In the process of training a large model using causal attention with mask(QK^T) as the core, the complexity of calculating mask(QK^T) is O(L^2). In the long sequence (such as L>1024) scenario, the computational complexity O(L^2) becomes a performance bottleneck, especially affecting the training and inference efficiency. Traditional algorithms (such as torch.tril or masked_fill of PyTorch) use element-wise multiplication to multiply QK^T with the lower triangular mask, resulting in a large number of invalid calculations, which therefore seriously affects the efficiency of large model training.

[0004] Therefore, there is a need for a new large model training method based on causal attention lossless acceleration algorithm. SUMMARY

[0005] The embodiment of the present specification provides a large model training method, device and equipment based on a causal attention lossless acceleration algorithm, which is used to solve the following technical problem: in the prior art, in the process of training a large model by using mask (QK^T) as the core, the complexity of calculating mask (QK^T) is O (L^2). In a long sequence (such as L>1024) scene, the calculation complexity O (L^2) becomes a performance bottleneck, which especially affects the training and inference efficiency. The traditional algorithm (such as torch.tril or masked_fill of PyTorch) adopts element-by-element multiplication to multiply QK^T and the lower triangular mask, which leads to a large amount of invalid calculation, and therefore, the efficiency of large model training is seriously affected.

[0006] To solve the above technical problem, the embodiment of the present specification is implemented as follows:

[0007] The embodiment of the present specification provides a large model training method based on a causal attention lossless acceleration algorithm, which comprises the following steps:

[0008] Obtaining training data to be trained;

[0009] Generating a Query matrix and a Key matrix based on the training data to be trained;

[0010] Based on a preset block number, the Query matrix and the Key matrix are blocked to obtain a submatrix network;

[0011] According to a preset number of structured matrix multiplication, the subblocks in the submatrix network are calculated according to a preset linear combination rule to obtain a multiplication result;

[0012] Performing a mask multiplication operation on the multiplication result to obtain a mask result;

[0013] Performing an addition operation on the multiplication result and the mask result to obtain a mask attention matrix;

[0014] Based on the mask attention matrix, a target large model is trained and obtained.

[0015] The embodiment of the present specification also provides a large model training device based on a causal attention lossless acceleration algorithm, which comprises the following steps:

[0016] An acquisition module is configured to acquire training data to be trained;

[0017] A Query matrix and a Key matrix generation module is configured to generate a Query matrix and a Key matrix based on the training data to be trained;

[0018] A matrix blocking module is configured to block the Query matrix and the Key matrix based on a preset block number to obtain a submatrix network;

[0019] a complete multiplication calculation module configured to perform calculation on sub-blocks in the sub-matrix network according to a preset linear combination rule according to a preset number of structured matrix multiplications to obtain a multiplication result;

[0020] a mask multiplication calculation module configured to perform mask multiplication operation on the multiplication result to obtain a mask result;

[0021] an addition module configured to perform addition operation on the multiplication result and the mask result to obtain a mask attention matrix;

[0022] a large model generation module configured to train a target large model based on the mask attention matrix.

[0023] Embodiments of the present specification also provide an electronic device, comprising:

[0024] at least one processor; and

[0025] a memory in communication connection with the at least one processor; wherein

[0026] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0027] obtain training data to be trained;

[0028] generate a Query matrix and a Key matrix based on the training data to be trained;

[0029] perform block division on the Query matrix and the Key matrix based on a preset number of blocks to obtain a sub-matrix network;

[0030] perform calculation on sub-blocks in the sub-matrix network according to a preset linear combination rule according to a preset number of structured matrix multiplications to obtain a multiplication result;

[0031] perform mask multiplication operation on the multiplication result to obtain a mask result;

[0032] perform addition operation on the multiplication result and the mask result to obtain a mask attention matrix;

[0033] train a target large model based on the mask attention matrix.

[0034] The large model training method based on the lossless acceleration algorithm of causal attention provided in the embodiments of this specification involves: acquiring training data to be trained; generating a query matrix and a key matrix based on the training data; dividing the query matrix and the key matrix into sub-matrix networks based on a preset number of blocks; performing structured matrix multiplication on the sub-blocks in the sub-matrix networks according to a preset linear combination rule to obtain multiplication results; performing a mask multiplication operation on the multiplication results to obtain a mask result; performing an addition operation on the multiplication results and the mask result to obtain a masked attention matrix; and training the target large model based on the masked attention matrix. This method can minimize unnecessary computational overhead in causal attention while maintaining accurate and error-free calculation results, thereby achieving efficient lossless acceleration and enabling efficient and rapid training of large models. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A schematic diagram of the system architecture of a large model training method based on a causal attention lossless acceleration algorithm provided in the embodiments of this specification;

[0037] Figure 2 A flowchart illustrating a large model training method based on a lossless acceleration algorithm for causal attention, provided in an embodiment of this specification;

[0038] Figure 3 This is a schematic diagram of the structure of the Q matrix sub-block and K matrix sub-block provided in the embodiments of this specification;

[0039] Figure 4 A framework diagram of a large model training method based on a lossless acceleration algorithm of causal attention provided in the embodiments of this specification;

[0040] Figure 5 This is a schematic diagram of a large model training device based on a lossless acceleration algorithm for causal attention, provided as an embodiment of this specification. Detailed Implementation

[0041] In order for those skilled in the art to better understand the technical solutions in the specification, the technical solutions in the specification will be clearly and completely described below in conjunction with the drawings in the specification. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the specification embodiments, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of the present application.

[0042] The core idea of the causal attention mechanism is that, in the process of generating a sequence, the model can only access the current and previous position information at each time step, and cannot see the future content, thereby strictly guaranteeing the causality of the generation process. Implementing this idea requires relying on two steps. First, calculate the "attention score" matrix. Each element in this matrix represents the semantic relevance between the current word and some other position, and is the key to "information flow" in language modeling. Second, through the "mask" operator, ensure that the model obtains information from historical word elements and cannot peek at future word elements - that is, the model can only obtain information from historical positions when generating the current word. This mechanism not only conforms to the time sequence logic in language generation, but also effectively prevents information "leakage" and ensures that the prediction process does not rely on future context. In addition, causal attention enables the model to dynamically focus on key information in the historical context at each step, thereby improving the modeling ability of long-distance dependencies, which is an indispensable ability in language understanding and generation tasks.

[0043] In terms of implementation, in order to guarantee the time causality, a lower triangular mask must be added to the attention matrix QK^T, so as to shield the information of future positions. Specifically, we perform a mask operation on the attention score matrix, retaining the main diagonal and the elements below it, that is, only allowing each position to focus on itself and the sequence positions before it. This process can be represented as applying a mask function mask(QK^T) to QK^T to ensure that the attention mechanism strictly follows the time sequence. This step is an indispensable technical detail in the causal attention mechanism, and directly affects the reliability and expressiveness of the model in the generation task.

[0044] In the prior art, in the process of training a large model using mask(QK^T) as the core causal attention, the complexity of calculating mask(QK^T) is O(L^2). In the long sequence (such as L>1024) scenario, the calculation complexity O(L^2) becomes a performance bottleneck, especially affecting the training and inference efficiency. The traditional algorithm (such as torch.tril or masked_fill of PyTorch) uses element-wise multiplication to multiply QK^T with the lower triangular mask, resulting in a large number of invalid calculations, thus seriously affecting the efficiency of large model training. Therefore, the embodiment of the present specification provides a large model training method based on causal attention lossless acceleration algorithm, which can perform matrix blocking on the Query matrix and the Key matrix, complete multiplication calculation, mask multiplication calculation, and addition integration, and finally generate a mask matrix of the attention matrix, realizing the same effect as the attention matrix, but using a different method, which can reduce invalid calculations and realize lossless acceleration.

[0045] Figure 1 A system architecture schematic diagram of a large model training method based on causal attention lossless acceleration algorithm is provided in the embodiment of the present specification. As shown in Figure 1 The system architecture 100 can include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 is used to provide a communication link medium between the terminal devices 101, 102, 103, and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0046] The terminal devices 101, 102, and 103 interact with the server 105 through the network 104 to receive or send messages, etc. Various client applications can be installed on the terminal devices 101, 102, and 103. For example, a special program based on causal attention lossless acceleration algorithm large model training method, etc.

[0047] The terminal devices 101, 102, and 103 can be hardware or software. When the terminal devices 101, 102, and 103 are hardware, they can be various special-purpose or general-purpose electronic devices, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc. When the terminal devices 101, 102, and 103 are software, they can be installed in the above-mentioned electronic devices. They can be implemented as multiple software or software modules (such as multiple software or software modules used to provide distributed services), or as a single software or software module.

[0048] The server 105 can be a server providing various services, such as a backend server providing services for a client application installed on the terminal device 101, 102, 103. For example, the server can perform a large model training method based on a causal attention lossless acceleration algorithm, so as to display a large model training result based on the causal attention lossless acceleration algorithm on the terminal device server 101, 102, 103.

[0049] The server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (such as multiple software or software modules for providing distributed services), or as a single software or software module.

[0050] Figure 2 A flowchart of a large model training method based on a causal attention lossless acceleration algorithm provided by an embodiment of the present specification. From a program perspective, the execution subject of the flow can be a program loaded on an application server or an application terminal. It can be understood that the method can be executed by any device, equipment, platform, device cluster with computing and processing capability. As shown in the figure, the large model training method includes: Figure 2

[0051] Step S201: Obtain training data to be trained.

[0052] The training data to be trained is training data used for target large model training, therefore, the type of the target large model to be trained directly determines the type of the training data to be trained.

[0053] In the embodiment of the present specification, the training data to be trained is text data, and / or data converted into text data;

[0054] The target large model is a large model processing text data, and / or a large model processing data converted into text data.

[0055] ​In the embodiments of this specification, the target large model is obtained by training based on the training data to be trained. Since the training data to be trained is text data, and / or data that can be converted into text data can be video data or audio data, it can also be considered that the target model is obtained by training using text data, video data, and audio data as training data to be trained. However, in specific implementations, if it is not direct text data, it needs to be converted into text data first. For example, when video data or audio data is used as training data to be trained, it is necessary to convert the audio data in the video data into text data, or extract the text from the video data, or convert the audio data into text data.

[0056] The acquisition of training data is carried out using existing technologies, which will not be elaborated here.

[0057] Step S203: Based on the training data to be trained, generate the Query matrix and the Key matrix.

[0058] The causal attention matrix is ​​the core mechanism for autoregressive generation in the Transformer model. Essentially, it ensures that the model can only access information from the current position and previous information during prediction by using a lower triangular mask matrix, thus avoiding the leakage of future information.

[0059] The core formula of the causal attention matrix can be summarized as follows:

[0060] Attention(Q, K, V) = softmax(mask(QKᵀ) / d k )·V;

[0061] Where Q (Query), K (Key), and V (Value) are the input matrices. The dimension of the key vector. d k The square root of the key vector dimension, through d k This prevents the gradient from vanishing due to an excessively large dot product. `mask(·)` represents the lower triangular masking operator, which sets the upper triangular portion of the matrix (excluding diagonal elements) to 0. This mask ensures that the i-th position can only see the current and previous information, thus ensuring the causality of the generated data. The `Softmax(·)` operator only operates on the lower triangular portion including diagonal elements.

[0062] In the embodiments described in this specification, the dimension of the Query matrix is ​​L*d, and the dimension of the Key matrix is ​​L*d, where L represents the token sequence length and d represents the feature dimension.

[0063] In the embodiments of the present application, the method for generating the Query matrix and the Key matrix based on the training data to be trained is known in the art and will not be described herein. In the embodiments of the present application, the Query matrix is referred to as Q matrix and the Key matrix is referred to as K matrix.

[0064] The present application is directed to the attention matrix during calculation, the traditional algorithm (such as torch.tril or masked_fill of PyTorch) uses element-wise multiplication to multiply QK^T with the lower triangular mask, resulting in a large number of invalid calculations, that is, the upper triangular part is zeroed but still participates in the calculation. In the embodiments of the present application, only the lower triangular part is calculated to solve the problem of waste of calculation, and at the same time, an innovative algorithm is further proposed for calculation of the lower triangular part to further save calculation.

[0065] Step S205: based on the preset block number, the Query matrix and the Key matrix are blocked to obtain a submatrix network.

[0066] In the embodiments of the present application, the Query matrix and the Key matrix are blocked based on the preset block number to obtain a submatrix network, which specifically includes:

[0067] Based on the preset block number, the Query matrix is blocked to obtain a Q matrix subblock, and the Key matrix is blocked to obtain a K matrix subblock, and the Q matrix subblock and the K matrix subblock form a submatrix network;

[0068] The preset block number is 4*4.

[0069] In the embodiments of the present application, the dimensions of the Query matrix and the Key matrix are L*d, then the Q matrix subblock and the K matrix subblock form a submatrix network including 16 subblocks, and the dimensions of each subblock in the submatrix network are (m / 4)*(n / 4), the Q matrix subblock includes: Q1, Q2, Q3, Q4, Q5, Q6, Q7, Q8, Q9, Q 10 , Q 11 , Q 12 , Q 13 , Q 14 , Q 15 , Q 16 ; and the K matrix subblock includes: K1, K2, K3, K4, K5, K6, K7, K8, K9, K 10 , K 11 , K 12 , K 13 , K 14 , K 15 , K 16 .

[0070] In order to facilitate the understanding of the method provided by the embodiments of the present specification, the following will be described by taking specific embodiments as examples.

[0071] The Query matrix Q is a matrix with a dimension of L*d, that is, each sub-block of the Q matrix sub-block has a size of (L / 4)*(d / 4); the Key matrix K is a matrix with a dimension of L*d, that is, each sub-block of the K matrix sub-block has a size of (L / 4)*(d / 4).

[0072] The Query matrix and the Key matrix are divided into 16 sub-blocks, and the Q matrix sub-blocks include Q1, Q2, Q3, Q4, Q5, Q6, Q7, Q8, Q9, Q 10 , Q 11 , Q 12 , Q 13 , Q 14 , Q 15 , Q 16， , that is, .

[0073] The K matrix sub-blocks include K1, K2, K3, K4, K5, K6, K7, K8, K9, K 10 , K 11 , K 12 , K 13 , K 14 , K 15 , K 16 , that is, .

[0074] Figure 3 The structure diagram of the Q matrix sub-block and the K matrix sub-block provided by the embodiments of the present specification is shown.

[0075] The Query matrix and the Key matrix are divided into Q matrix sub-blocks and K matrix sub-blocks, which can facilitate subsequent multiplication operations and can be operated in blocks, thereby providing the possibility of parallelization and pruning.

[0076] Step S207: According to the structured matrix multiplication of the preset number of times, the sub-blocks in the sub-matrix network are calculated according to the preset linear combination rule to obtain the multiplication result.

[0077] In the embodiments of the present specification, the preset number of times is 24 times;

[0078] According to the structured matrix multiplication of the preset number of times, the sub-blocks in the sub-matrix network are calculated according to the preset linear combination rule to obtain the multiplication result, specifically including:

[0079] The sub-matrix network is composed of 24 sub-matrixes, and the sub-matrixes in the sub-matrix network are combined according to a preset linear combination rule to obtain the multiplication result, the multiplication result including m1, m2, m3, m4, m5, m6, m7, m8, m9, m 10 , m 11 , m 12 , m 13 , m 14 , m 15 , m 16 , m 17 , m 18 , m 19 , m 20 , m 21 , m 22 , m 23 , m 24 ;

[0080] The preset linear combination rule includes:

[0081] m1= (Q8+ Q 11 ) · (-K2+ K3- K4+ K8) T ;

[0082] m2= (Q 15 + Q5) · (K1- K5- K6+ K7) T ;

[0083] m3= (-Q 10 + Q 16 + Q 12 ) · (-K2+ K 12 ) T ;

[0084] m4= (Q 13 + Q9- Q 14 ) · (K9- K6) T ;

[0085] m5= (-Q6+ Q 15 - Q7) · (K2+ K 11 ) T ;

[0086] m6= (Q6+ Q7- Q 11 ) · (K6+ K 11 ) T ;

[0087] m7= (Q6+ Q7) · K 11 T ;

[0088] m8= (-Q 14- Q 10 + Q6-Q 15 + Q7+ Q 16 + Q 12 )·K2 T ;

[0089] m9= (Q 13 + Q9- Q 14 -Q 10 + Q6+ Q7- Q 11 )·K6 T ;

[0090] m 10 = Q 11 · ( K2- K3+ K7+ K 11 + K4- K8) T ;

[0091] m 11 = Q5· ( K5+ K6- K7) T ;

[0092] m 12 = Q8· ( K2- K3+ K4) T ;

[0093] m 13 = Q 15 · ( - K1+K5 + K6+ K3- K7+ K 11 ) T ;

[0094] m 14 = (Q 13 + Q9+ Q 15 )· ( - K1+ K5+ K6) T ;

[0095] m 15 = (Q 11 + Q 16 + Q 12 )· ( K2+ K4- K8) T ;

[0096] m 16 = (Q9- Q 16 )· ( K1- K8) T ;

[0097] m 17 = (Q 10 -Q 12 )·K 12 T ;

[0098] m 18 = (Q 13 -Q 14 ) · K9 T ;

[0099] m 19 = (-Q 15 + Q7+ Q8) · (- K2+ K3) T ;

[0100] m 20 = Q9 · (K5+ K9- K8) T ;

[0101] m 21 = (Q9-Q8+ Q 12 ) · K8 T ;

[0102] m 22 = (Q 13 -Q5+ Q 16 ) · K1 T ;

[0103] m 23 = Q 16 · (- K1+K4+ K 12 ) T ;

[0104] m 24 = Q 14 · (K9+K2+ K 10 ) T ;

[0105] wherein,

[0106] T denotes matrix transposition.

[0107] Continuing the previous example, since the Query matrix and the Key matrix are divided into Q matrix sub-blocks and K matrix sub-blocks, 24 structured matrix multiplications are performed on the Q matrix sub-blocks and the K matrix sub-blocks, thereby obtaining the effective calculation region under the mask in the attention matrix, i.e., obtaining the calculation result of the lower triangular region.

[0108] In each multiplication operation based on each sub-block of the Q matrix sub-block and the K matrix sub-block, the product of Q and K is matrix transposed to obtain the multiplication result. After the multiplication operation, the local structure can be maximally utilized and redundant calculation can be avoided, the related items in the lower triangular mask are accurately calculated, the upper triangular region finally masked by the mask is skipped, and the operation merging can be performed by utilizing the vector redundancy and the local stationarity.

[0109] The lower triangular matrix in the attention matrix refers to the area below the diagonal line of the attention matrix and including the diagonal line, and all elements in the area above the diagonal line of the matrix are 0. The upper triangular area in the attention matrix refers to the area above the diagonal line of the attention matrix, and all elements in the area are 0.

[0110] In the embodiment, m1, m2, m3, m4, m5, m6, m7, m8, m9, m 10 , m 11 , m 12 , m 13 , m 14 , m 15 , m 16 , m 17 , m 18 , m 19 , m 20 , m 21 , m 22 , m 23 , m 24 all represent product results. m1= (Q8+ Q 11 )·(-K2+ K3-K4+ K8) T , wherein Q8, Q 11 are row vectors, representing query vectors, (Q8+ Q 11 ) represents a combination of query vectors, K2, K3, K4, K8 are column vectors, representing key vectors, (-K2+ K3-K4+ K8) represents a combination of key vectors, + represents similarity, and - represents dissimilarity. m1 is a weighted sum of positive and negative similarities. It should be particularly noted that + and - in the embodiments of the present specification represent similar meanings, generally + represents similarity, - represents dissimilarity, or + represents positive contribution, and - represents negative contribution. The equivalent expansion form of m1= (Q8+ Q 11 )·(-K2+ K3-K4+ K8) T is: m1=-Q8·K2 T + Q8·K3 T - Q8·K4 T + Q8·K8 T -Q 11 ·K2 T +Q 11 ·K3 T -Q 11 ·K4 T +Q 11 ·K8 T It can be seen that m1 is composed of 8 independent dot product operations. Similarly, the equivalent expansion form of other results of the product results and the meaning of the results can be derived.

[0111] As mentioned above, all elements in the upper triangle of the attention matrix are 0, and through the operation of this step, the invalid calculation in the upper triangle of the attention matrix can be eliminated, and the computing power consumption is reduced.

[0112] Step S209: performing a mask multiplication operation on the multiplication result to obtain a mask result.

[0113] In a sequence generation task (such as text generation), the model can only access the information of the previous position when predicting the current position, and cannot see the future information. Therefore, in order to realize this constraint, the constraint of the lower triangular matrix is needed. In the embodiment of the present specification, based on the multiplication result, in order to further return a result containing only the lower triangular region, a mask multiplication operation needs to be further performed.

[0114] In the embodiment of the present specification, the mask multiplication operation on the multiplication result to obtain the mask result specifically includes:

[0115] The mask multiplication operation is performed four times on the multiplication result to obtain the mask result, and the mask result includes r1, r2, r3 and r4.

[0116] Wherein,

[0117] r1 = mask([Q1 Q2 Q3 Q4] ⋅ [K1 K2 K3 K4]ᵀ);

[0118] r2 = mask(Q5 + Q7 − Q 11 , −K6 − K7);

[0119] r3 = mask(Q 10 , K6 + K 10 + K 12 );

[0120] r4 = mask([Q 13 Q 14 Q 15 Q 16 ] ⋅ [K 13 K 14 K 15 K 16 ]ᵀ);

[0121] T represents matrix transposition;

[0122] Mask(·) represents a lower triangular mask operator, that is, the upper triangular part of the matrix (not including the diagonal element) is set to 0.

[0123] Continuing the previous example, r1, r2, r3, r4 represent four mask multiplication operations on the multiplication result, and the corresponding mask results, which represent attention scores, so r1, r2, r3, r4 can be considered as four attention scores, i.e., the first attention score, the second attention score, the third attention score, and the fourth attention score. Among them, r1 = mask([Q1 Q2 Q3 Q4] ⋅ [K1 K2 K3 K4]ᵀ) represents the matrix multiplication of the query vector [Q1 Q2 Q3 Q4] and the transposed key vector [K1 K2 K3 K4], and the lower triangular mask (mask) is applied to the result, and finally the first attention score r1 is obtained; r2 = mask(Q5 + Q7 − Q 11 , −K6 − K7) represents the combination of the query vector (Q5 + Q7 − Q 11 ) and the key vector (−K6 − K7), and the lower triangular mask is applied to finally obtain the second attention score r2; r3 = mask(Q 10 , K6 + K 10 + K 12 ) represents the combination of the query selection (Q 10 ) and the key (K6 + K 10 + K 12 ), and the lower triangular mask is applied to finally obtain the third attention score r3; r4 = mask([Q 13 Q 14 Q 15 Q 16 ] ⋅ [K 13 K 14 K 15 K 16 ]ᵀ) represents the matrix composition of the query vector combination [Q 13 Q 14 Q 15 Q 16 ] and the transposed key vector combination [K 13 K 14 K 15 K 16 ], and the lower triangular mask (mask) is applied to the result, and finally the fourth attention score r4 is obtained. When the multiplication result is subjected to four mask multiplication operations to obtain r1, r2, r3, r4, the results of r1, r2, r3, r4 only focus on the query vector and key vector of the corresponding block in the formula, and do not focus on other areas.

[0124] Through the operation of this step, the lower triangular constraint can be guaranteed, thereby supporting scale expansion.

[0125] Step S211: performing an addition operation on the multiplication result and the mask result to obtain a mask matrix.

[0126] In the embodiments of the present specification, the addition operation on the multiplication result and the mask result to obtain the mask matrix specifically includes:

[0127] The addition operation on the multiplication result and the mask result to obtain the mask matrix is as follows:

[0128] The addition operation specifically is:

[0129] S 11 =r1;

[0130] S 21 =m2-m5-m7+m 11 +m 12 +m 13 +m 19 ;

[0131] S 22 =mask(m1+m6-m7+m 10 +m 11 +m 129 )+ r2;

[0132] S 31 =m1+m3+m 12 +m 15 +m 16 +m 17 +m 21 -m 23 ;

[0133] S 32 =m1-m4+m6-m7-m9+m 10 +m 12 +m 18 +m 20 +m 21 ;

[0134] S 33 =mask(m4-m6+m7+m9-m 17 -m 18 )+ r3;

[0135] S 41 =m2-m3-m5-m7-m8+m 11 +m 13 -m 17 +m 22 +m 23 ;

[0136] S 42 =m2+m4 +m11 +m 14 +m 16 -m 18 -m 20 +m 22 ;

[0137] S 43 =m3+m5 +m7+m8+m 17 +m 18 +m 24 ;

[0138] S 44 =r4;

[0139] S 11 、S 21 、S 22 、S 31 、S 32 、S 33 、S 41 、S 42 、S 43 、S 44 representing the region of the lower triangular structure of the mask matrix.

[0140] Continuing the previous example, based on the multiplication results m1-m 24 and the mask results r1-r4, linear combination is performed to obtain the mask matrix.

[0141] The mask matrix obtained in this step is strictly consistent with the original mask attention result, which is equivalent to the standard mask (QKᵀ).

[0142] In order to further understand the large model training method based on the causal attention lossless acceleration algorithm provided by the embodiments of the present specification, Figure 4 the framework schematic diagram of the large model training method based on the causal attention lossless acceleration algorithm provided by the embodiments of the present specification is shown in Figure 4 , and the large model training method comprises:

[0143] Based on the training data to be trained, QKᵀ (Query matrix and Key matrix) is generated; QKᵀ is divided into blocks to obtain a plurality of Q sub-matrices and a plurality of K sub-matrices; matrix multiplication is performed on the Q sub-matrices and the K sub-matrices to generate multiplication results m1-m 24 ; mask multiplication operation is performed on the multiplication results m1-m 24 to obtain mask results r1-r4; based on the multiplication results m1-m 24 and the mask results r1-r4, addition operation is performed to obtain a mask attention matrix S, the mask attention matrix S is completely consistent with mask (QKᵀ); based on the mask attention matrix S, a target large model is trained and obtained.

[0144] Therefore, by using the method provided by the embodiments of the present specification, the process of generating mask(QKᵀ) from QKᵀ is changed.

[0145] The method for training a large model based on a causal attention lossless acceleration algorithm provided by the embodiments of the present specification comprises the following steps: obtaining training data to be trained; generating a Query matrix and a Key matrix based on the training data to be trained; performing block processing on the Query matrix and the Key matrix based on a preset number of blocks to obtain a sub-matrix network; performing calculation on sub-blocks in the sub-matrix network according to a preset number of structured matrix multiplications and a preset linear combination rule to obtain a multiplication result; performing a mask multiplication operation on the multiplication result to obtain a mask result; performing an addition operation on the multiplication result and the mask result to obtain a mask attention matrix; and training a target large model based on the mask attention matrix. The method can minimize unnecessary calculation overhead in causal attention while maintaining the accuracy of the calculation result, thereby achieving efficient lossless acceleration and achieving the purpose of efficient and rapid large model training.

[0146] The above describes a method for training a large model based on a causal attention lossless acceleration algorithm. Correspondingly, the present specification also provides a device for training a large model based on a causal attention lossless acceleration algorithm, as shown in Figure 5 . Figure 5 A schematic diagram of a device for training a large model based on a causal attention lossless acceleration algorithm provided by the embodiments of the present specification is shown in the figure. The device comprises:

[0147] The acquisition module 501 acquires training data to be trained.

[0148] The Query matrix and Key matrix generation module 503 generates a Query matrix and a Key matrix based on the training data to be trained.

[0149] The matrix block processing module 505 performs block processing on the Query matrix and the Key matrix based on a preset number of blocks to obtain a sub-matrix network.

[0150] The complete multiplication calculation module 507 performs calculation on sub-blocks in the sub-matrix network according to a preset number of structured matrix multiplications and a preset linear combination rule to obtain a multiplication result.

[0151] The mask multiplication calculation module 509 performs a mask multiplication operation on the multiplication result to obtain a mask result.

[0152] The addition module 511 performs an addition operation on the multiplication result and the mask result to obtain a mask attention matrix.

[0153] The large model generation module 513 trains a target large model based on the mask attention matrix.

[0154] Embodiments of the present specification also provide an electronic device, comprising:

[0155] at least one processor; and

[0156] a memory in communication with the at least one processor; wherein

[0157] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0158] obtain training data to be trained;

[0159] generate a Query matrix and a Key matrix based on the training data to be trained;

[0160] based on a preset block number, block the Query matrix and the Key matrix to obtain a submatrix network;

[0161] According to the preset number of structured matrix multiplication, the subblocks in the submatrix network are calculated according to the preset linear combination rule to obtain a multiplication result;

[0162] performing a mask multiplication operation on the multiplication result to obtain a mask result;

[0163] performing an addition operation on the multiplication result and the mask result to obtain a mask attention matrix;

[0164] training a target large model based on the mask attention matrix.

[0165] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or possible.

[0166] The various embodiments described in this specification are presented by way of example, and each embodiment is not inherently more important than any other embodiment. The embodiments described herein can be combined, other embodiments can be implemented, and structural or logical changes can be made without departing from the scope of the present disclosure. The preceding description, as such, is not to be understood in absolutely restrictive senses. The same, similar or identical components are denoted by the same reference numerals in the drawings and repeated explanations are omitted.

[0167] The apparatus, electronic device, and non-transitory computer storage medium provided by the embodiments of the present disclosure correspond to the method, and therefore have similar beneficial technical effects. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device, and non-transitory computer storage medium will not be described here.

[0168] In the 1990s, it was possible to distinguish whether an improvement in a technology was a hardware improvement (e.g., an improvement in the circuit structure of a diode, transistor, switch, etc.) or a software improvement (an improvement in a method flow). However, as technology has advanced, many improvements in method flows today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into a hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A designer programs a digital system "integrated" on a PLD by himself / herself, without having to ask a chip manufacturer to design and manufacture a special integrated circuit chip. Moreover, instead of manually manufacturing an integrated circuit chip, this programming is now mostly implemented using "logic compiler" software, which is similar to a software compiler used when developing a program, and the original code before compilation must also be written in a specific programming language, which is called a hardware description language (HDL), and there are many types of HDL, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that it is only necessary to logically program a method flow using the above-mentioned hardware description languages and program it into an integrated circuit to easily obtain a hardware circuit that implements the logical method flow.

[0169] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to implementing the controller in pure computer readable program code, it is also possible to implement the controller in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. to perform the same functions by logically programming the method steps. Such a controller can therefore be considered as a hardware component, and the means included therein for performing various functions can also be considered as structures within the hardware component. Alternatively, the means for performing various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0170] The systems, apparatuses, modules or units illustrated by the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0171] For the sake of description, the above apparatuses are described in functional division and are described respectively. Of course, the functions of each unit can be implemented in the same or more software and / or hardware when implementing one or more embodiments of the present specification.

[0172] Those skilled in the art will understand that the embodiments of the present specification can be provided as a method, a system or a computer program product. Therefore, the embodiments of the present specification can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0173] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0174] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0176] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0177] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0178] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0179] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to encompass non-exclusive inclusion, such that processes, methods, articles or devices that comprise a list of elements not only include those elements, but also include other elements not expressly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0180] The specification can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0181] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0182] The above merely provides a description of the embodiments of the present application, but should not be taken as a limitation. Various changes and modifications can be made to the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall fall within the scope of claims of the present application.

Claims

1. A large model training method based on a causal attention lossless acceleration algorithm, characterized in that, The large model training method comprises: obtaining training data to be trained; based on the training data to be trained, generating a Query matrix and a Key matrix; based on a preset block number, block the Query matrix and the Key matrix to obtain a submatrix network; according to a preset number of structured matrix multiplication, calculate the subblocks in the submatrix network according to a preset linear combination rule to obtain a multiplication result; performing a mask multiplication operation on the multiplication result to obtain a mask result; performing an addition operation on the multiplication result and the mask result to obtain a mask attention matrix; based on the mask attention matrix, training to obtain a target large model.

2. The large model training method of claim 1, wherein, The training data to be trained is text data and / or data converted into text data; The target large model is a large model for processing text data and / or a large model for processing data converted into text data.

3. The large model training method of claim 1, wherein, The training data to be trained is text data, audio data and / or video data; The target large model is a large model for processing text data, a large model for processing audio data and / or a large model for processing video data.

4. The large model training method of claim 1, wherein, Based on the preset block number, the Query matrix and the Key matrix are blocked to obtain a submatrix network, specifically comprising: based on the preset block number, block the Query matrix to obtain a Q matrix subblock, and block the Key matrix to obtain a K matrix subblock, the Q matrix subblock and the K matrix subblock form a submatrix network; The preset block number is 4*4.

5. The large model training method of claim 4, wherein, The dimension of the Query matrix is L*d, and the dimension of the Key matrix is L*d. The sub-matrix network composed of the Q matrix sub-block and the K matrix sub-block includes 16 sub-blocks, and the dimension of each sub-block in the sub-matrix network is (L / 4)*(L / 4). The Q matrix sub-block includes: Q1, Q2, Q3, Q4, Q5, Q6, Q7, Q8, Q9, Q 10 , Q 11 , Q 12 , Q 13 , Q 14 , Q 15 , Q 16 ; and the K matrix sub-block includes: K1, K2, K3, K4, K5, K6, K7, K8, K9, K 10 , K 11 , K 12 , K 13 , K 14 , K 15 , K 16 , wherein L represents the token sequence length, and d represents the feature dimension.

6. The large model training method of claim 5, wherein, The preset number is 24 times; According to a preset number of structured matrix multiplication, calculate the subblocks in the submatrix network according to a preset linear combination rule to obtain a multiplication result, specifically comprising: The sub-matrix network is composed of 24 sub-matrixes, and the sub-matrixes are composed of 16 sub-blocks. The sub-blocks in the sub-matrix network are combined according to a preset linear combination rule to obtain a multiplication result, the multiplication result including m1, m2, m3, m4, m5, m6, m7, m8, m9, m 10 , m 11 , m 12 , m 13 , m 14 , m 15 , m 16 , m 17 , m 18 , m 19 , m 20 , m 21 , m 22 , m 23 , m 24 ; The preset linear combination rule comprises: m1= (Q8+ Q 11 ) · ( - K2+ K3- K4+ K8) T ; m2= (Q 15 + Q5) · (K1- K5- K6+ K7) T ; m3= (-Q 10 + Q 16 + Q 12 ) · (- K2+ K 12 ) T ; m4= (Q 13 + Q9- Q 14 ) · (K9- K6) T ; m5= (-Q6+ Q 15 - Q7) · (K2+ K 11 ) T ; m6 = (Q6+ Q7- Q 11 ) · (K6+ K 11 ) T ; m7 = (Q6+ Q7) K 11 T ; m8= (-Q 14 - Q 10 + Q6-Q 15 + Q7+ Q 16 + Q 12 ) · K2 T ; m9= (Q 13 + Q9- Q 14 -Q 10 + Q6+ Q7- Q 11 ) · K6 T ; m 10 = Q 11 ·( K2- K3+ K7+ K 11 + K4- K8) T ; m 11 = Q5·(K5+ K6- K7) T ; m 12 = Q8·(K2- K3+ K4) T ; m 13 = Q 15 ·( - K1+K5 + K6+ K3- K7+ K 11 ) T ; m 14 = (Q 13 + Q9+ Q 15 ) · ( - K1+ K5+ K6) T ; m 15 = (Q 11 + Q 16 + Q 12 ) · (K2+ K4- K8) T ; m 16 = (Q9- Q 16 ) · (K1- K8) T ; m 17 = (Q 10 - Q 12 ) · K 12 T ; m 18 = (Q 13 -Q 14 ) · K9 T ; m 19 = (-Q 15 + Q7+ Q8) · (- K2+ K3) T ; m 20 = Q9·(K5+ K9- K8) T ; m 21 = (Q9 - Q8 + Q 12 ) · K8 T ; m 22 = (Q 13 -Q5+ Q 16 ) · K1 T ; m 23 = Q 16 ·( - K1+K4+ K 12 ) T ; m 24 = Q 14 ·( K9+K2+ K 10 ) T ; wherein, T represents matrix transposition.

7. The large model training method of claim 6, wherein, The mask multiplication operation on the multiplication result to obtain the mask result, specifically comprising: performing 4 times mask multiplication operation on the multiplication result to obtain the mask result, the mask result comprising: r1, r2, r3, r4; wherein, r1 = mask([Q1 Q2 Q3 Q4] ⋅ [K1 K2 K3 K4]ᵀ); r2 = mask(Q5 + Q7 - Q 11 , -K6 - K7); r3 = mask(Q 10 , K6 + K 10 + K 12 ); r4 = mask([Q 13 Q 14 Q 15 Q 16 ] ⋅ [K 13 K 14 K 15 K 16 ]ᵀ); T represents matrix transposition; Mask(·) represents a lower triangular mask operator.

8. The large model training method of claim 7, wherein, The addition operation on the multiplication result and the mask result to obtain the mask attention matrix, specifically comprising: performing the following addition operation on the multiplication result and the mask result to obtain the mask attention matrix; The addition operation is specifically: S 11 =r1； S 21 = m2- m5- m7+ m 11 + m 12 + m 13 + m 19 ; S 22 = mask (m1 + m6 - m7 + m 10 + 11 + 129 ) + r2; S 31 = m1+ m3+ m 12 + 15 + 16 + 17 + 21 - 23 ; S 32 = m1 - m4 + m6 - m7 - m9 + m 10 + 12 + 18 + 20 + 21 ; S 33 = mask (m4 - m6 + m7 + m9 - m 17 -m 18 )+ r3; S 41 = m2-m3-m5-m7-m8+m 11 + 13 - 17 + 22 + 23 ; S 42 = m2+ m4+ m 11 + m 14 + m 16 - m 18 - m 20 + m 22 ; S 43 = m3+ m5+ m7+ m8+ m 17 + 18 + m 24 ; S 44 =r4； S 11 , S 21 , S 22 , S 31 , S 32 , S 33 , S 41 , S 42 , S 43 , S 44 denotes a region of a lower triangular structure of the mask matrix.

9. A large model training device based on a causal attention lossless acceleration algorithm, characterized in that, The large model training device comprises: an acquisition module for acquiring training data to be trained; a Query matrix and a Key matrix generation module for generating a Query matrix and a Key matrix based on the training data to be trained; a matrix block module for blocking the Query matrix and the Key matrix based on a preset block number to obtain a submatrix network; A complete multiplication calculation module calculates the sub-blocks in the sub-matrix network according to a preset linear combination rule according to a preset number of structured matrix multiplications, to obtain a multiplication result; A mask multiplication calculation module performs a mask multiplication operation on the multiplication result to obtain a mask result; An addition module performs an addition operation on the multiplication result and the mask result to obtain a mask attention matrix; A large model generation module trains a target large model based on the mask attention matrix.

10. An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: obtain training data to be trained; generate a Query matrix and a Key matrix based on the training data to be trained; block the Query matrix and the Key matrix based on a preset number of blocks to obtain a sub-matrix network; calculate the sub-blocks in the sub-matrix network according to a preset linear combination rule according to a preset number of structured matrix multiplications, to obtain a multiplication result; perform a mask multiplication operation on the multiplication result to obtain a mask result; perform an addition operation on the multiplication result and the mask result to obtain a mask attention matrix; train a target large model based on the mask attention matrix.

Images