Large model training method, device and equipment based on fast causal attention algorithm
By partitioning the S and V matrices and performing structured matrix multiplication, the problem of redundant computation in the causal attention algorithm is solved, achieving high efficiency and speed in training large models.
Patent Information
- Application Number
- CN202511678375.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-17
- Publication Date
- 2026-02-10
AI Technical Summary
In existing technologies, the fast causal attention algorithm suffers from a large amount of invalid computation due to the upper triangular part of the S matrix being 0, which affects the training efficiency of large models.
By dividing the S and V matrices into blocks and employing structured matrix multiplication and mask multiplication operations, redundant computations are reduced, and a fast attention matrix is generated.
While maintaining the accuracy of the calculation results, we aim to minimize unnecessary computational overhead and achieve efficient and rapid training of large models.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This specification relates to the field of large model technology, and in particular to a method, apparatus and device for training large models based on a fast causal attention algorithm. Background Technology
[0002] In recent years, the Transformer architecture has become a core technical framework in many fields, including Natural Language Processing (NLP), Computer Vision (CV), and Generative Artificial Intelligence (such as Large Language Models (LLM)). Compared to traditional Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs), Transformers can process sequential data in parallel more efficiently and show significant advantages in modeling long-range dependencies. Causal attention is a key mechanism in autoregressive Transformer models, especially the GPT series models, playing a crucial role in building the ability to understand and generate context.
[0003] In existing technologies, the fast causal attention algorithm SV (where S is a lower triangular matrix and V is a full matrix) calculates SV by default according to the definition of matrix multiplication, i.e., S = softmax(mask(QK^T), and then calculates S*V. Since the upper triangular part of S is all 0, this leads to a large number of "0 multiplied by a certain number" operations. Of course, "0 multiplied by a certain number" still results in 0, so these multiplication operations are redundant, leading to a large amount of invalid computation. Therefore, it has been verified that this affects the efficiency of training large models.
[0004] Therefore, a new method for training large models based on a fast causal attention algorithm is needed. Summary of the Invention
[0005] This specification provides a method, apparatus, and device for training large models based on a fast causal attention algorithm, to address the following technical problem: In the prior art, the fast causal attention algorithm SV (where S is a lower triangular matrix and V is a full matrix) calculates SV by default according to the definition of matrix multiplication, i.e., S = softmax(mask(QK^T), and then calculates S*V. Since the upper triangular part of S is all 0, this leads to a large number of "0 multiplied by a certain number" operations. Of course, "0 multiplied by a certain number" still results in 0, so these multiplication operations are redundant, leading to a large amount of invalid computation, thus affecting the efficiency of large model training.
[0006] To solve the above-mentioned technical problems, the embodiments in this specification are implemented as follows:
[0007] This specification provides an embodiment of a large model training method based on a fast causal attention algorithm, including:
[0008] Obtain the training data to be trained;
[0009] Based on the training data to be trained, generate the S matrix and the V matrix;
[0010] Based on a preset number of blocks, the S matrix and the V matrix are divided into blocks to obtain a sub-matrix network;
[0011] The sub-blocks in the sub-matrix network are calculated according to a preset linear combination rule based on a first preset number of structured matrix multiplications to obtain the first multiplication result;
[0012] Perform a second preset number of masked multiplication operations on the first multiplication result to obtain the second multiplication result;
[0013] The first multiplication result is added to the second result to obtain the fast attention matrix;
[0014] Based on the fast attention matrix, the target large model is trained and obtained.
[0015] This specification also provides an embodiment of a large model training device based on a fast causal attention algorithm, comprising:
[0016] The module retrieves the training data to be trained.
[0017] The S-matrix and V-matrix generation module generates S-matrix and V-matrix based on the training data to be trained;
[0018] The matrix partitioning module divides the S matrix and the V matrix into blocks based on a preset number of blocks to obtain a sub-matrix network;
[0019] The first multiplication calculation module performs calculations on the sub-blocks in the sub-matrix network according to a preset linear combination rule, based on a first preset number of structured matrix multiplications, to obtain the first multiplication result;
[0020] The second multiplication calculation module performs a second preset number of mask multiplication operations on the first multiplication result to obtain the second multiplication result;
[0021] The addition module adds the first multiplication result to the second result to obtain a fast attention matrix;
[0022] The large model generation module trains and obtains the target large model based on the fast attention matrix.
[0023] This specification also provides an electronic device, including:
[0024] At least one processor; and,
[0025] A memory communicatively connected to the at least one processor; wherein,
[0026] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:
[0027] Obtain the training data to be trained;
[0028] Based on the training data to be trained, generate the S matrix and the V matrix;
[0029] Based on a preset number of blocks, the S matrix and the V matrix are divided into blocks to obtain a sub-matrix network;
[0030] The sub-blocks in the sub-matrix network are calculated according to a preset linear combination rule based on a first preset number of structured matrix multiplications to obtain the first multiplication result;
[0031] Perform a second preset number of masked multiplication operations on the first multiplication result to obtain the second multiplication result;
[0032] The first multiplication result is added to the second result to obtain the fast attention matrix;
[0033] Based on the fast attention matrix, the target large model is trained and obtained.
[0034] The large model training method based on the fast causal attention algorithm provided in this specification involves: acquiring training data to be trained; generating an S matrix and a V matrix based on the training data; dividing the S matrix and the V matrix into sub-matrix networks based on a preset number of blocks to obtain a sub-matrix network; performing a first preset number of structured matrix multiplications on the sub-blocks of the sub-matrix network according to a preset linear combination rule to obtain a first multiplication result; performing a second preset number of mask multiplication operations on the first multiplication result to obtain a second multiplication result; adding the first multiplication result and the second result to obtain a fast attention matrix; and training a target large model based on the fast attention matrix. This method can minimize unnecessary computational overhead in fast attention while maintaining accurate and error-free calculation results, thereby achieving efficient and lossless acceleration and enabling efficient and rapid training of large models. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0036] Figure 1 A schematic diagram of a system architecture for a large model training method based on a fast causal attention algorithm, provided in the embodiments of this specification;
[0037] Figure 2 A flowchart illustrating a large model training method based on a fast causal attention algorithm provided in the embodiments of this specification;
[0038] Figure 3 This is a schematic diagram of the structure of the S-matrix sub-block and V-matrix sub-block provided in the embodiments of this specification;
[0039] Figure 4 A framework diagram of a large model training method based on a fast causal attention algorithm provided in the embodiments of this specification;
[0040] Figure 5 This is a schematic diagram of a large model training device based on a fast causal attention algorithm, provided as an embodiment of this specification. Detailed Implementation
[0041] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.
[0042] Figure 1 This diagram illustrates the system architecture of a large model training method based on a fast causal attention algorithm, as provided in the embodiments of this specification. Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.
[0043] Terminal devices 101, 102, and 103 interact with server 105 via network 104 to receive or send messages, etc. Various client applications can be installed on terminal devices 101, 102, and 103, such as dedicated programs for large model training methods based on fast causal attention algorithms.
[0044] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various dedicated or general-purpose electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services) or as a single software program or software module.
[0045] Server 105 can be a server that provides various services, such as a backend server that provides services to client applications installed on terminal devices 101, 102, and 103. For example, the server can perform large model training based on the fast causal attention algorithm so that the training results of the large model based on the fast causal attention algorithm can be displayed on terminal device servers 101, 102, and 103.
[0046] Server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services), or as a single software program or software module.
[0047] Figure 2 This is a flowchart illustrating a large model training method based on a fast causal attention algorithm, provided in an embodiment of this specification. From a programming perspective, the execution entity of the process can be a program hosted on an application server or application terminal. It is understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities. Figure 2 As shown, the training method for this large model includes:
[0048] Step S201: Obtain the training data to be trained.
[0049] The training data to be trained is the training data used for training the target large model. Therefore, the type of the target large model to be trained directly determines the type of training data to be trained.
[0050] In the embodiments of this specification, the training data to be trained is text data, and / or data converted into text data;
[0051] The target large model is a large model for processing text data, and / or a large model for data converted into text data.
[0052] In the embodiments of this specification, the target large model is obtained by training based on the training data to be trained. Since the training data to be trained is text data, and / or data that can be converted into text data can be video data or audio data, it can also be considered that the target model is obtained by training using text data, video data, and audio data as training data to be trained. However, in specific implementations, if it is not direct text data, it needs to be converted into text data first. For example, when video data or audio data is used as training data to be trained, it is necessary to convert the audio data in the video data into text data, or extract the text from the video data, or convert the audio data into text data.
[0053] The acquisition of training data is carried out using existing technologies, which will not be elaborated here.
[0054] Step S203: Based on the training data to be trained, generate the S matrix and the V matrix.
[0055] In the Transformer model, S=softmax(mask(QKᵀ)) is the core computational step of the self-attention mechanism. Its core is to convert the unnormalized attention scores into a probability distribution, thereby obtaining the attention weight matrix S, and then weighting and aggregating the value matrix S.
[0056] In the embodiments of this specification, the S matrix is the output matrix of causal attention, i.e., S = softmax(mask(QK^T)), where S is a lower triangular matrix of dimension L * L, L is the length of the token sequence, and Q and K are the Query and Key matrices, respectively, both of dimension L * d. `mask()` is the lower triangular masking operator, and `softmax()` is the row-normalized exponential function. In this embodiment, the method of forming the S matrix does not constitute a limitation of this application.
[0057] In the embodiments described in this specification, the S matrix has a dimension of L*L, and the V matrix has a dimension of L*d, where L represents the token sequence length and d represents the feature dimension.
[0058] In the embodiments of this specification, the method for generating the S matrix and V matrix based on the training data to be trained is prior art and will not be described in detail here. In the embodiments of this specification, the S matrix is the attention weight matrix, and the V matrix is the aggregation value matrix.
[0059] Step S205: Based on a preset number of blocks, divide the S matrix and the V matrix into blocks to obtain a sub-matrix network.
[0060] In the embodiments of this specification, the step of dividing the S matrix and the V matrix into blocks based on a preset number of blocks to obtain a sub-matrix network specifically includes:
[0061] Based on the preset number of blocks, the S matrix is divided into blocks to obtain S matrix sub-blocks, and the V matrix is divided into blocks to obtain V matrix sub-blocks. The V matrix sub-blocks and the V matrix sub-blocks form a sub-matrix network.
[0062] The preset number of blocks is 4*4.
[0063] In the embodiments described in this specification, the dimension of the S matrix is L*L, and the dimension of the V matrix is L*d. The sub-matrix network composed of the S matrix sub-blocks and the V matrix sub-blocks comprises 16 sub-blocks. The dimension of each sub-block in the S matrix is (L / 4)*(L / 4). The S matrix sub-blocks include: S1, S2, S3, S4, S5, S6, S7, S8, S9, S... 10 S 11 S 12 S 13 S 14 S 15 S 16 The dimension of each sub-block in the V matrix is (L / 4)*(d / 4), and the V matrix sub-blocks include: V1, V2, V3, V4, V5, V6, V7, V8, V9, V 10 V 11 V 12 V 13 V 14 V 15 V 16 , where L represents the token sequence length and d represents the feature dimension.
[0064] To facilitate understanding of the methods provided in the embodiments of this specification, specific embodiments will be used as examples for illustration below.
[0065] If S is a matrix with dimension L*L, then the size of each sub-block of S is (L / 4)*(L / 4); If V is a matrix with dimension L*d, then the size of each sub-block of V is (L / 4)*(d / 4).
[0066] The S matrix and V matrix are each divided into 16 sub-blocks. The S matrix sub-blocks include: S1, S2, S3, S4, S5, S6, S7, S8, S9, S... 10 S 11 S 12 S 13 S 14 S 15 S 16 ;,Right now ;
[0067] The V matrix sub-blocks include: V1, V2, V3, V4, V5, V6, V7, V8, V9, V 10 V 11 V 12 V 13 V 14 V 15 V 16 ,Right now .
[0068] Figure 3 This is a schematic diagram of the structure of the S-matrix sub-block and V-matrix sub-block provided in the embodiments of this specification.
[0069] Dividing the S and V matrices into S matrix sub-blocks and V matrix sub-blocks facilitates subsequent multiplication operations in blocks, providing possibilities for parallel processing and pruning.
[0070] Step S207: According to the first preset number of structured matrix multiplications, the sub-blocks in the sub-matrix network are calculated according to the preset linear combination rules to obtain the first multiplication result.
[0071] In this embodiment of the specification, the first preset number of times is 24 times;
[0072] The first multiplication, performed according to a first preset number of structured matrix multiplications, calculates the sub-blocks in the submatrix network according to a preset linear combination rule to obtain the first multiplication result, specifically including:
[0073] The sub-blocks in the submatrix network are multiplied 24 times according to a preset linear combination rule to obtain the first multiplication result, which includes m1, m2, m3, m4, m5, m6, m7, m8, m9, m 10 m11 m 12 m 13 m 14 m 15 m 16 m 17 m 18 m 19 m 20 m 21 m 22 m 23 m 24 ;
[0074] The preset linear combination rules include:
[0075] m1=(S3+ S4+ S5)·(- V2+ V3- V4+ V8);
[0076] m2=(S2+ S7+ S8)·(V1- V5- V6+ V7);
[0077] m3=(S4- S7+ S9)·(- V2+ V 12 );
[0078] m4=(-S5+ S6+ S8)·(V9- V6);
[0079] m5=(- S2- S7+ S9)·(V2+ V 11 );
[0080] m6=(S3+ S5- S6)·(V6+ V 11 );
[0081] m7=(-S2- S3- S5+S6- S7+ S9)·V 11 ;
[0082] m8 = (-S7 + S9)·V2;
[0083] m9 = (-S5 + S6)·V6;
[0084] m 10 =(S3+ S5)·(V2- V3+ V7+ V 11 (+ V4- V8)
[0085] m 11 = (S2+ S3+ S7+ S8)·(V5+ V6- V7);
[0086] m 12= (S2+ S3+ S4+ S5)·(V2- V3+ V4);
[0087] m 13 = (S2+ S7)·(- V1+V5 + V6+ V3- V7+ V 11 );
[0088] m 14 = S8·(- V1+ V5+ V6)
[0089] m 15 = S4·( V2+ V4- V8;
[0090] m 16 =(S4+ S8)·(V1- V8);
[0091] m 17 =(S4- S6- S7+ S9)·V 12 ;
[0092] m 18 = (S5 - S6 - S8 + S9)·V9;
[0093] m 19 = S2·(- V2+ V3;
[0094] m 20 = (S5- S8)·(V5+V9- V8);
[0095] m 21 = (S4 + S5)·V8;
[0096] m 22 = (S7 + S8)·V1;
[0097] m 23 =(-S4+ S7)·(- V1+V4+ V 12 );
[0098] m 24 = S9·( V9+V2+ V 10 ).
[0099] Continuing from the previous example, since the S matrix and V matrix are divided into S matrix sub-blocks and V matrix sub-blocks, 24 matrix multiplications are performed on the S matrix sub-blocks and V matrix sub-blocks to obtain the effective computation region under the masking effect in the attention matrix, that is, to obtain the computation result of the upper triangular region.
[0100] Each multiplication operation, based on sub-blocks of the S and V matrices, involves multiplying the combination of S and V to obtain the first multiplication result. Through multiplication, the local structure can be utilized to the maximum extent while avoiding redundant calculations, enabling accurate calculation of relevant terms in the upper triangular matrix. Furthermore, the operation can be merged by leveraging vector redundancy and local stationarity.
[0101] An upper triangular matrix is a matrix in which all elements in the lower left region of the diagonal are 0.
[0102] In this embodiment, m1, m2, m3, m4, m5, m6, m7, m8, m9, m 10 m 11 m 12 m 13 m 14 m 15 m 16 m 17 m 18 m 19 m 20 m 21 m 22 m 23 m 24 All represent the result of the first multiplication. In m1 = (S3 + S4 + S5) · (- V2 + V3 - V4 + V8), S3, S4, and S5 are all L / 4 * L / 4 matrices representing attention weight vectors, (S3 + S4 + S5) represents the combination of attention weight vectors, V2, V3, V4, and V8 are all matrices representing aggregate value vectors, (- V2 + V3 - V4 + V8) represents the combination of aggregate value vectors, + indicates similarity, - indicates dissimilarity, and m1 is the weighted sum of positive and negative similarities. It should be noted that in the embodiments of this specification, + and - represent similar meanings, generally + represents similarity and - represents dissimilarity, or + represents positive contribution and - represents negative contribution. m1 = (S3 + S4 + S5) · (- V2 + V3 - V4 + V8) means calculating the matrix (S3 + S4 + S5) multiplied by the matrix (- V2 + V3 - V4 + V8). Using the method of this application, only one multiplication needs to be calculated.
[0103] As mentioned above, this step can eliminate invalid calculations in the lower triangle and reduce computing power consumption.
[0104] Step S209: Perform a second preset number of mask multiplication operations on the first multiplication result to obtain the second multiplication result.
[0105] In this embodiment of the specification, the second preset number of times is 10. The step of performing a second preset number of mask multiplication operations on the first multiplication result to obtain the second multiplication result specifically includes:
[0106] Perform 10 masked multiplication operations on the first multiplication result to obtain the second multiplication result, wherein the masked result includes: h1, h2, h3, h4, h5, h6, h7, h8, h9, h 10 ;
[0107] in,
[0108] h1 = S3·(- V6+ V7)
[0109] h2 = S3·(V6+ V 10 + V 12 );
[0110] h3 = S1·V1;
[0111] h4 = S1·V2;
[0112] h5 = S1·V3;
[0113] h6 = S1·V4;
[0114] h7 = S 10 ·V 13 ;
[0115] h8 = S 10 ·V 14 ;
[0116] h9 = S 10 ·V 15 ;
[0117] h 10 = S 10 ·V 16 .
[0118] Continuing from the previous example, h1, h2, h3, h4, h5, h6, h7, h8, h9, h 10 This represents the value obtained by performing a 10-mask multiplication operation on the result of the first multiplication. Here, h1 = S3·(-V6+V7) means multiplying matrix S3 by matrix (-V6+V7) to obtain h1, and so on.
[0119] This step ensures the lower triangular constraint, thus supporting scaling.
[0120] Step S211: Add the first multiplication result to the second result to obtain the fast attention matrix.
[0121] In this embodiment of the specification, the step of adding the first multiplication result and the second result to obtain the fast attention matrix specifically includes:
[0122] The first multiplication result and the second result are added together as follows to obtain the fast attention matrix;
[0123] The addition operation is specifically as follows:
[0124] O1=h3;
[0125] O2=h4;
[0126] O3=h5;
[0127] O4=h6;
[0128] O5=m2+m 11 -m 22 +h2;
[0129] O6 = -m5 + m6 + m7 + m8 + m9;
[0130] O7=-m5+m6+m7+m8+m9+m 19 +h2;
[0131] O8=m1+m 12 +m 19 -m 21 ;
[0132] O9=m4+m9+m 14 +m 16 +m 20 +m 21 ;
[0133] O 10 =-m3-m8-m9+m 17 +h2;
[0134] O 11 =m1-m6-m9+m 10 +m 15 -h2;
[0135] O 12 =m3+m8+m 15 -m 17 +m 21 ;
[0136] O 13 =m4+m9+m 14 +m 18 +m 22 +h7;
[0137] O 14 =-m4-m8-m9-m 18 +m 24 +h8;
[0138] O 15 =m2-m5-m8+m 13 +m 14 -m 19 +h9;
[0139] O 16 =m3+m8+m 15 -m 16 +m 22 +m 23 +h 10 .
[0140] Continuing with the previous example, based on the aforementioned multiplication result m1-m 24 and the mask result h1-h 10 Linear combinations are performed to obtain a fast attention matrix.
[0141] It should be noted that the method provided in the embodiments of this specification can be applied not only to fast causal attention algorithms, but also to the rapid calculation of the gradient of the causal attention mechanism. Specifically, according to the backpropagation rule, the gradient of the causal attention mechanism can also be written as matrix multiplication AB=C, where any A, B, or C is a lower triangular matrix. The algorithm described above can reduce the computational cost of the gradient of the causal attention mechanism without loss, thereby further accelerating the training of large models.
[0142] To further understand the large model training method based on the fast causal attention algorithm provided in the embodiments of this specification Figure 4 This specification provides a schematic diagram of the framework for a large model training method based on a fast causal attention algorithm, as illustrated in the embodiments of this specification. Figure 4 As shown, the training method for this large model includes:
[0143] Based on the training data to be trained, generate S matrix and V matrix; divide the S matrix and V matrix into blocks to obtain several S sub-matrices and several V sub-matrices; perform structured matrix composition on the S sub-matrices and V sub-matrices to generate the first multiplication result m1-m 24 For the first multiplication result m1-m 24 Perform mask multiplication to obtain the second multiplication result h1-h 10 Based on the first multiplication result m1-m 24 The result of the second multiplication, h1-h 10 Addition operations are performed to obtain a fast attention matrix; based on the fast attention matrix, the target large model is trained.
[0144] Therefore, using the method provided in the embodiments of this specification is equivalent to changing the process of generating S*V.
[0145] The large model training method based on the fast causal attention algorithm provided in this specification involves: acquiring training data to be trained; generating an S matrix and a V matrix based on the training data; dividing the S matrix and the V matrix into sub-matrix networks based on a preset number of blocks; performing structured matrix multiplication on the sub-blocks of the sub-matrix networks according to a preset linear combination rule to obtain a first multiplication result; performing a second preset number of masked multiplication operations on the first multiplication result to obtain a second multiplication result; adding the first multiplication result and the second result to obtain a fast attention matrix; and training the target large model based on the fast attention matrix. This method can minimize unnecessary computational overhead in fast attention while maintaining accurate and error-free calculation results, thereby achieving efficient and lossless acceleration and enabling efficient and rapid training of large models.
[0146] The above content details a large model training method based on a fast causal attention algorithm. Correspondingly, this specification also provides a large model training device based on the fast causal attention algorithm, such as... Figure 5 As shown. Figure 5 This is a schematic diagram of a large model training device based on a fast causal attention algorithm, provided in an embodiment of this specification. The large model training device includes:
[0147] Get module 501 to obtain the training data to be trained;
[0148] S-matrix and V-matrix generation module 503 generates S-matrix and V-matrix based on the training data to be trained;
[0149] The matrix segmentation module 505 divides the S matrix and the V matrix into blocks based on a preset number of blocks to obtain a sub-matrix network;
[0150] The first multiplication calculation module 507 calculates the sub-blocks in the sub-matrix network according to a preset linear combination rule based on a first preset number of structured matrix multiplications to obtain the first multiplication result;
[0151] The second multiplication calculation module 509 performs a second preset number of mask multiplication operations on the first multiplication result to obtain the second multiplication result;
[0152] Addition module 511 adds the first multiplication result to the second result to obtain a fast attention matrix;
[0153] The large model generation module 513 trains and obtains the target large model based on the fast attention matrix.
[0154] This specification also provides an electronic device, including:
[0155] At least one processor; and,
[0156] A memory communicatively connected to the at least one processor; wherein,
[0157] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:
[0158] Obtain the training data to be trained;
[0159] Based on the training data to be trained, generate the S matrix and the V matrix;
[0160] Based on a preset number of blocks, the S matrix and the V matrix are divided into blocks to obtain a sub-matrix network;
[0161] The sub-blocks in the sub-matrix network are calculated according to a preset linear combination rule based on a first preset number of structured matrix multiplications to obtain the first multiplication result;
[0162] Perform a second preset number of masked multiplication operations on the first multiplication result to obtain the second multiplication result;
[0163] The first multiplication result is added to the second result to obtain the fast attention matrix;
[0164] Based on the fast attention matrix, the target large model is trained and obtained.
[0165] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0166] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for apparatus, electronic devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0167] The apparatus, electronic device, and non-volatile computer storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, electronic device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device, and non-volatile computer storage medium will not be repeated here.
[0168] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.
[0169] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0170] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.
[0171] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.
[0172] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0173] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0174] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0175] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0176] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0177] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0178] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0179] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0180] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside on local and remote computer storage media, including storage devices.
[0181] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0182] The above description is merely an embodiment of this specification and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.
Claims
1. A method for training large models based on a fast causal attention algorithm, characterized in that, The large model training method includes: Obtain the training data to be trained; Based on the training data to be trained, generate the S matrix and the V matrix; Based on a preset number of blocks, the S matrix and the V matrix are divided into blocks to obtain a sub-matrix network; The sub-blocks in the sub-matrix network are calculated according to a preset linear combination rule based on a first preset number of structured matrix multiplications to obtain the first multiplication result; Perform a second preset number of masked multiplication operations on the first multiplication result to obtain the second multiplication result; The first multiplication result is added to the second result to obtain the fast attention matrix; Based on the fast attention matrix, the target large model is trained and obtained.
2. The large model training method as described in claim 1, characterized in that, The training data to be trained is text data, and / or data converted into text data; The target large model is a large model for processing text data, and / or a large model for data converted into text data.
3. The large model training method as described in claim 1, characterized in that, The training data to be trained is text data, and / or audio data, and / or video data; The target large model is a large model for processing text data, and / or a large model for processing audio data, and / or a large model for processing video data.
4. The large model training method as described in claim 1, characterized in that, The step of dividing the S matrix and the V matrix into blocks based on a preset number of blocks to obtain a sub-matrix network specifically includes: Based on the preset number of blocks, the S matrix is divided into blocks to obtain S matrix sub-blocks, and the V matrix is divided into blocks to obtain V matrix sub-blocks. The V matrix sub-blocks and the V matrix sub-blocks form a sub-matrix network. The preset number of blocks is 4*4.
5. The large model training method as described in claim 4, characterized in that, The S matrix has a dimension of L*L, and the V matrix has a dimension of L*d. The S matrix sub-blocks and the V matrix sub-blocks form a submatrix network comprising 16 sub-blocks. Each sub-block in the S matrix has a dimension of (L / 4)*(L / 4). The S matrix sub-blocks include: S1, S2, S3, S4, S5, S6, S7, S8, S9, S... 10 S 11 S 12 S 13 S 14 S 15 S 16 The dimension of each sub-block in the V matrix is (L / 4)*(d / 4), and the V matrix sub-blocks include: V1, V2, V3, V4, V5, V6, V7, V8, V9, V 10 V 11 V 12 V 13 V 14 V 15 V 16 , where L represents the token sequence length and d represents the feature dimension.
6. The large model training method as described in claim 5, characterized in that, The first preset number of times is 24 times; The first multiplication, performed according to a first preset number of structured matrix multiplications, calculates the sub-blocks in the sub-matrix network according to a preset linear combination rule to obtain the first multiplication result, specifically including: The sub-blocks in the submatrix network are multiplied 24 times according to a preset linear combination rule to obtain the first multiplication result, which includes m1, m2, m3, m4, m5, m6, m7, m8, m9, m 10 m 11 m 12 m 13 m 14 m 15 m 16 m 17 m 18 m 19 m 20 m 21 m 22 m 23 m 24 ; The preset linear combination rules include: m1=(S3+ S4+ S5)·(- V2+ V3- V4+ V8); m2=(S2+ S7+ S8)·(V1- V5- V6+ V7); m3 = (S4 - S7 + S9) · (- V2 + V 12 (); m4=(-S5+ S6+ S8)·(V9- V6); m5 = (- S2 - S7 + S9) · (V2 + V 11 (); m6=(S3+ S5- S6)·( V6+ V 11 ); m7 = (-S2- S3- S5+S6- S7+ S9) · V 11 ; m8 = (-S7 + S9)·V2; m9 = (-S5 + S6)·V6; m 10 =(S3+ S5)·( V2- V3+ V7+ V 11 + V4-V8); m 11 = (S2+ S3+ S7+ S8) · (V5+ V6- V7) ; m 12 = (S2 + S3 + S4 + S5) · (V2 - V3 + V4) ; m 13 = (S2+ S7)·( - V1+V5 + V6+ V3- V7+ V 11 ); m 14 = S8·(- V1+ V5+ V6); m 15 = S4·( V2+ V4- V8); m 16 =(S4+ S8)·( V1- V8); m 17 =(S4- S6- S7+ S9)·V 12 ; m 18 =(S5- S6- S8+ S9)·V9; m 19 = S2·(- V2+ V3) ; m 20 =(S5- S8)·(V5+V9- V8); m 21 =(S4+ S5)·V8; m 22 =(S7+ S8)·V1; m 23 =(-S4+ S7)·( - V1+V4+ V 12 ); m 24 = S9·( V9+V2+ V 10 ().
7. The large model training method as described in claim 6, characterized in that, The second preset number of operations is 10. The process of performing a second preset number of mask multiplication operations on the first multiplication result to obtain the second multiplication result specifically includes: Perform 10 masked multiplication operations on the first multiplication result to obtain the second multiplication result, wherein the masked result includes: h1, h2, h3, h4, h5, h6, h7, h8, h9, h 10 ; in, h1 = S3·(- V6+ V7) h2 = S3 (V6+ V 10 + In 12 ); h3 = S1·V1; h4 = S1·V2; h5 = S1·V3; h6 = S1·V4; h7 = S 10 ·V 13 ; h8 = S 10 ·V 14 ; h9 = S 10 ·V 15 ; h 10 = S 10 ·V 16 。 8. The large model training method as described in claim 7, characterized in that, The step of adding the first multiplication result to the second result to obtain the fast attention matrix specifically includes: The first multiplication result and the second result are added together as follows to obtain the fast attention matrix; The addition operation is specifically as follows: O1=h3; O2=h4; O3=h5; O4=h6; O5=m2+m 11 -m 22 +h2; O6 = -m5 + m6 + m7 + m8 + m9; O7 = -m5 + m6 + m7 + m8 + m9 + m 19 +h2; O8=m1+m 12 +m 19 -m 21 ; O9=m4+m9+m 14 +m 16 +m 20 +m 21 ; O 10 =-m3-m8-m9+m 17 +h2; O 11 =m1-m6-m9+m 10 +m 15 -h2; O 12 =m3+m8+m 15 -m 17 +m 21 ; O 13 =m4+m9+m 14 +m 18 +m 22 +h7; O 14 =-m4-m8-m9-m 18 +m 24 +h8; O 15 =m2-m5-m8+m 13 +m 14 -m 19 +h9; O 16 =m3+m8+m 15 -m 16 +m 22 +m 23 +h 10 。 9. A large model training device based on a fast causal attention algorithm, characterized in that, The large model training device includes: The module retrieves the training data to be trained. The S-matrix and V-matrix generation module generates S-matrix and V-matrix based on the training data to be trained; The matrix partitioning module divides the S matrix and the V matrix into blocks based on a preset number of blocks to obtain a sub-matrix network; The first multiplication calculation module performs calculations on the sub-blocks in the sub-matrix network according to a preset linear combination rule, based on a first preset number of structured matrix multiplications, to obtain the first multiplication result; The second multiplication calculation module performs a second preset number of mask multiplication operations on the first multiplication result to obtain the second multiplication result; The addition module adds the first multiplication result to the second result to obtain a fast attention matrix; The large model generation module trains and obtains the target large model based on the fast attention matrix.
10. An electronic device, comprising: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: Obtain the training data to be trained; Based on the training data to be trained, generate the S matrix and the V matrix; Based on a preset number of blocks, the S matrix and the V matrix are divided into blocks to obtain a sub-matrix network; The sub-blocks in the sub-matrix network are calculated according to a preset linear combination rule based on a first preset number of structured matrix multiplications to obtain the first multiplication result; Perform a second preset number of masked multiplication operations on the first multiplication result to obtain the second multiplication result; The first multiplication result is added to the second result to obtain the fast attention matrix; Based on the fast attention matrix, the target large model is trained and obtained.