Sparse processing method and apparatus for sparse attention network, and electronic device

Abstract

A sparse processing method and apparatus for a sparse attention network, and an electronic device. The method specifically comprises: in a memory cell, using constraint conditions to construct a masked sparse representation (S301), wherein the constraint conditions are: the dimension of a mask matrix is [B,a,S], B represents the batch size, a represents the number of heads, S represents the sequence length, and each element in the S dimension represents a masked starting row of each column in the mask matrix; and a calculation unit acquiring the masked sparse representation from the memory cell, using the masked sparse representation to perform sparse processing on input data, and storing the sparse processing result into the memory cell (S302). According to the method, the constraint conditions are used to construct the masked sparse representation, so that memory consumption can be reduced from a quadratic order of the sequence length to a linear order of the sequence length, thereby remarkably reducing memory requirements during large model training, and improving the training efficiency.

Description

Sparse processing method, device and electronic device for sparse attention network Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as large models, deep learning, and natural language processing. Background Art

[0002] In the training of large models based on transformers, the attention module is an important component of the transformer module. The computational complexity and memory overhead of attention are in a quadratic relationship with the sequence length. Flash Attention, proposed by the industry, aims to reduce the memory complexity of causal attention so that it is close to a linear relationship with the sequence length. However, when a custom mask needs to be passed in, the memory overhead of the mask itself is still related to the square of the sequence length, which limits the effectiveness of Flash Attention in reducing memory consumption. Therefore, how to further reduce memory consumption while maintaining the performance of the attention mechanism, especially when using custom masks, remains an important issue in current research and practice.

[0003] Summary of the Invention

[0004] The present disclosure provides a sparse processing method, device and electronic device for a sparse attention network.

[0005] According to a first aspect of the present disclosure, a sparse processing method for a sparse attention network is provided, which is applied to a computing device, the computing device including a computing unit and a memory unit, and the method includes:

[0006] A masked sparse representation is constructed in the memory unit using constraints. The constraints are as follows: the dimension of the mask matrix is ​​[B, a, S], where B represents the batch size, a represents the number of heads, and S represents the sequence length. Each element in the S dimension represents the starting row of each column in the mask matrix to be masked.

[0007] The computing unit obtains the masked sparse representation from the memory unit, performs sparse processing on the input data using the masked sparse representation, and stores the sparse processing result in the memory unit.

[0008] According to a second aspect of the present disclosure, a method for fine-tuning a large language model is provided, comprising:

[0009] Obtain a dataset for the target task;

[0010] Fine-tune the large language model based on the dataset to adapt it to the needs of the target task;

[0011] Among them, fine-tuning training of the large language model based on the dataset includes:

[0012] Combined with the sparse attention mechanism, fine-tuning and training of large language models based on datasets;

[0013] Among them, the sparse attention mechanism adopts the sparse processing method of the sparse attention network provided in the first aspect.

[0014] According to a third aspect of the present disclosure, a method for processing data of a large language model is provided, comprising:

[0015] Get input data;

[0016] Input data into the large language model so that the large language model processes the input data in combination with the sparse attention mechanism to generate output data;

[0017] Among them, the sparse attention mechanism adopts the sparse processing method of the sparse attention network provided in the first aspect.

[0018] According to a fourth aspect of the present disclosure, a sparse processing apparatus for a sparse attention network is provided, which is applied to a computing device, the computing device including a computing unit and a memory unit, and the apparatus including:

[0019] A construction module for constructing a masked sparse representation in memory cells using constraints: the mask matrix has dimensions [B, a, S], where B represents the batch size, a represents the number of heads, and S represents the sequence length. Each element in the S dimension represents the starting row of each column in the mask matrix to be masked.

[0020] The sparse processing module is used to control the computing unit to obtain the mask sparse representation from the memory unit, perform sparse processing on the input data using the mask sparse representation, and store the sparse processing results in the memory unit.

[0021] According to a fifth aspect of the present disclosure, a device for fine-tuning a large language model is provided, comprising:

[0022] The first acquisition module is used to obtain a data set for a target task;

[0023] The fine-tuning module is used to fine-tune the large language model based on the dataset to adapt it to the requirements of the target task;

[0024] Among them, fine-tuning training of the large language model based on the dataset includes:

[0025] Combined with the sparse attention mechanism, fine-tuning and training of large language models based on datasets;

[0026] Among them, the sparse attention mechanism adopts the sparse processing method of the sparse attention network provided in the first aspect.

[0027] According to a sixth aspect of the present disclosure, there is provided a data processing device for a large language model, comprising:

[0028] A second acquisition module is used to acquire input data;

[0029] A data processing module is used to input input data into the large language model so that the large language model can process the input data in combination with the sparse attention mechanism to generate output data;

[0030] Among them, the sparse attention mechanism adopts the sparse processing method of the sparse attention network provided in the first aspect.

[0031] According to a seventh aspect of the present disclosure, there is provided an electronic device, including:

[0032] at least one processor;

[0033] a memory communicatively coupled to the at least one processor;

[0034] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to execute the sparse processing method of the sparse attention network provided by the first aspect and / or the fine-tuning method of the large language model provided by the second aspect and / or the data processing method of the large language model provided by the third aspect.

[0035] According to an eighth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the sparse processing method of the sparse attention network provided by the first aspect and / or the fine-tuning method of the large language model provided by the second aspect and / or the data processing method of the large language model provided by the third aspect.

[0036] According to a ninth aspect of the present disclosure, a computer program product is provided, comprising a computer program stored on a storage medium, wherein when the computer program is executed by a processor, the computer program implements the sparse processing method of the sparse attention network provided in the first aspect and / or the fine-tuning method of the large language model provided in the second aspect and / or the data processing method of the large language model provided in the third aspect.

[0037] According to the solution disclosed in the present invention, by adopting constraints to construct a masked sparse representation, the memory consumption can be reduced from the square of the sequence length to the linear order of the sequence length, thereby significantly reducing the memory requirements during large model training and improving training efficiency.

[0038] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0040] Figure 1 is a schematic diagram of several attention mechanisms in related technologies.

[0041] FIG2 is a schematic diagram of several hole attention mechanisms in related technologies.

[0042] FIG3 is a flow chart of a sparse processing method of a sparse attention network according to an embodiment of the present disclosure.

[0043] FIG4 is a two-dimensional schematic diagram of a block attention mask in a Causal scenario according to an embodiment of the present disclosure.

[0044] FIG5 is a two-dimensional schematic diagram of a Block Attention Mask in a model fine-tuning scenario of a large language model according to an embodiment of the present disclosure.

[0045] FIG6 is another two-dimensional schematic diagram of a Block Attention Mask in a model fine-tuning scenario of a large language model according to an embodiment of the present disclosure.

[0046] FIG7 is another two-dimensional schematic diagram of a Block Attention Mask in a model fine-tuning scenario of a large language model according to an embodiment of the present disclosure.

[0047] FIG8 is a schematic diagram of performing an operation of constructing a mask sparse representation using constraints before calling the Block Attention mechanism according to an embodiment of the present disclosure.

[0048] FIG9 is a schematic diagram of an operation of constructing a mask sparse representation using constraint conditions in a data reading module according to an embodiment of the present disclosure.

[0049] FIG10 is a flowchart of a method for fine-tuning a large language model according to an embodiment of the present disclosure.

[0050] FIG11 is a flow chart of a method for processing data of a large language model according to an embodiment of the present disclosure.

[0051] Figure 12 is a structural diagram of a sparse processing device of a sparse attention network according to an embodiment of the present disclosure.

[0052] FIG13 is a schematic diagram of a fine-tuning apparatus for a large language model according to an embodiment of the present disclosure.

[0053] FIG14 is a schematic diagram of a data processing apparatus for a large language model according to an embodiment of the present disclosure.

[0054] Figure 15 is a scenario diagram of the sparse processing method of the sparse attention network according to an embodiment of the present disclosure.

[0055] FIG16 is a schematic diagram of a scenario of a method for fine-tuning a large language model according to an embodiment of the present disclosure.

[0056] FIG17 is a scenario diagram of a data processing method for a large language model according to an embodiment of the present disclosure.

[0057] Figure 18 is a structural diagram of an electronic device used to implement the sparse processing method of the sparse attention network and / or the fine-tuning method of the large language model and / or the data processing method of the large language model of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0059] The terms "first," "second," and "third," etc. in the description, embodiments, claims, and figures of the present disclosure are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, for example, including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to the process, method, product, or apparatus.

[0060] In related technologies, the Attention calculation is shown in formula (1): Y = softmax(QK T +Mask) V (1)

[0061] Generally, the shape of Q, K, V, and Y is [B, S, H], and the shape of Mask is [B, a, S, S]; where B represents the batch size, S represents the sequence length, H represents the hidden size, and a represents the number of heads.

[0062] However, in large model training, masks are usually sparse.

[0063] In existing technologies, common Attention solutions include:

[0064] 1. Vanilla Attention mechanism;

[0065] 2. Flash Attention mechanism;

[0066] 3. Sparse Attention mechanism;

[0067] 4. Random Attention mechanism;

[0068] 5. Window Attention Mechanism;

[0069] 6. Global Attention Mechanism;

[0070] 7. Dilated Attention mechanism;

[0071] 8.Combined attention (BigBird Attention) mechanism.

[0072] Figure 1 shows schematic diagrams of several attention mechanisms from related art. As shown in Figure 1, (a) shows a random attention mechanism, (b) shows a window attention mechanism, (c) shows a global sparse attention mechanism, and (d) shows a combined attention mechanism. The attention matrix A indicates which token pairs require attention. If A(i, j) = 1, it indicates that token i requires attention to be calculated with token j. For the random attention mechanism, the values ​​of A(i, j) may be generated based on a random process, resulting in certain positions being selected for attention calculation while others are not. For the window attention mechanism, the values ​​of A(i, j) are typically 1 only within a fixed local window and 0 outside the window. For the global attention mechanism, the values ​​of A(i, j) are typically all 1 in the matrix, indicating that attention is calculated between all token pairs. By combining local attention, global sparse attention, and possible randomness, the combined attention mechanism allows the model to capture key dependencies in the sequence while maintaining computational efficiency.

[0073] Figure 2 shows schematic diagrams of several dilation-based attention mechanisms in related art. By introducing dilation, the dilation-based attention mechanism increases the receptive field while maintaining sparsity, allowing it to capture more distant dependencies without increasing computational burden. As shown in Figure 2, the sequence length of (a) is 4 and the dilation ratio is 1; the sequence length of (b) is 8 and the dilation ratio is 2; and the sequence length of (c) is 16 and the dilation ratio is 4. Figures (a), (b), and (c) are fused to produce Figure (d). The dilation ratio is a parameter used to control the number and size of dilation holes in the convolution kernel, which determines the size of the kernel's receptive field. A larger dilation ratio results in a larger receptive field, and vice versa. Therefore, adjusting the dilation ratio can, to a certain extent, control the range and accuracy of the model's perception of the input data.

[0074] As shown in Figures 1 and 2, both the combined attention mechanism and the hole attention mechanism are examples of sparse attention. They achieve sparsity through different mechanisms, thereby optimizing the model's computational efficiency and performance. However, these sparse attention schemes have the following drawbacks: by introducing a mask that is proportional to the square of the sequence length into the attention calculation formula, memory pressure increases dramatically when the sequence length is very long.

[0075] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, the present disclosure proposes a sparse processing method for sparse attention networks. By adopting preset constraints to construct a masked sparse representation of the block attention mechanism, the memory consumption can be reduced from the square of the sequence length to the linear order of the sequence length, thereby significantly reducing the memory requirement during large model training and improving training efficiency. At the same time, through further optimization in Flash Attention, the computing performance of large models can be improved.

[0076] The present disclosure provides a sparse processing method for a sparse attention network, which is applied to a computing device comprising a computing unit and a memory unit. Figure 3 is a flow diagram of the sparse processing method for a sparse attention network according to an embodiment of the present disclosure. The sparse processing method for a sparse attention network can be applied to a sparse processing device for a sparse attention network. The sparse processing device for a sparse attention network is located in a computing device within an electronic device. The electronic device includes, but is not limited to, fixed devices and / or mobile devices. For example, fixed devices include, but are not limited to, servers, which can be cloud servers or ordinary servers. For example, mobile devices include, but are not limited to, mobile phones and tablets. In some possible implementations, the sparse processing method for a sparse attention network can also be implemented by a processor invoking computer-readable instructions stored in a memory. A sparse attention network is a special attention mechanism that reduces computational complexity and memory consumption by selecting a small portion of content relevant to the current location. This mechanism has broad application prospects in processing long sequences of data, such as text and images. The core concept of a sparse attention network is that, rather than giving equal attention to all input information, attention is focused on the portion most relevant to the current task, thereby improving processing efficiency and accuracy. As shown in Figure 3, the sparse processing method of the sparse attention network includes:

[0077] S301: Construct a masked sparse representation in the memory unit using constraints. The constraints are: the dimension of the mask matrix is ​​[B, a, S], where B represents the batch size, a represents the number of heads, and S represents the sequence length. Each element in the S dimension represents the starting row of each column in the mask matrix to be masked.

[0078] S302: The computing unit obtains the masked sparse representation from the memory unit, performs sparse processing on the input data using the masked sparse representation, and stores the sparse processing result in the memory unit.

[0079] In the embodiment of the present disclosure, the masked starting row can be understood as the row from which masking begins.

[0080] In the disclosed embodiments, masked sparse representation is used to describe a data sparsification representation method implemented by masking in an attention mechanism. In the attention mechanism, masking is a commonly used tool to indicate which parts should be ignored or not considered during the calculation process. Sparse representation refers to a data representation method in which most elements are zero or close to zero, and only a few elements have significant non-zero values.

[0081] In the embodiments of the present disclosure, masked sparse representation is often used in the attention mechanism to reduce computational complexity and memory consumption. Sparsification is achieved by pre-defining a mask that specifies which locations should be ignored (i.e., set to zero) when calculating attention weights. This sparsification not only improves computational efficiency but also helps the model focus on the most important information.

[0082] In some implementations, the attention module uses a constructed masked sparse representation to perform sparse processing on the input data. This involves applying a mask to the calculation of attention weights, allowing the model to ignore the masked portions when calculating the attention weights, thereby achieving attention sparsification. This sparse processing method can significantly improve the model's computational efficiency, especially when processing long sequences of data. Furthermore, because the model only focuses on the important portions, it can also improve model performance.

[0083] In some embodiments, whether each position should be masked can be determined based on the sequence length and the number of heads. This usually involves an in-depth understanding of the sequence content and task requirements. For example, in natural language processing tasks, it may be necessary to decide which word dependencies should be considered and which should be ignored based on the grammatical structure or semantic information of the sentence. Specifically, the granularity of the mask can be determined based on the length of the sequence. For longer sequences, a coarser-grained mask may be required to reduce the amount of computation, while for shorter sequences, a finer-grained mask can be used to retain more information. At the same time, the number of heads can also affect the design of the mask. Different heads may focus on different aspects or features in the sequence, so different masking strategies may be required.

[0084] In practical applications, designing mask sparse representation requires comprehensive consideration of multiple factors such as model performance, computational efficiency, and memory consumption. It usually requires a certain amount of experimentation and tuning to find the optimal masking strategy.

[0085] In the disclosed embodiment, the mask matrix has the dimensions [B, a, S]. In the sparse representation [B, a, S], B represents the batch size, i.e., the number of samples processed simultaneously; a represents the number of attention heads, i.e., the number of attention heads computed in parallel in the multi-head attention mechanism; and S represents the sequence length, where each element represents the row in a column from which all subsequent rows are masked. Since we only care about the mask in the lower left triangle, and this mask has a particular sparsity, it can be efficiently represented using the dimensions [B, a, S]. This representation enables more efficient memory utilization and reduces unnecessary computation when computing block attention. By focusing only on the locations that actually participate in the attention computation, the efficiency of large-scale sequence processing tasks can be significantly improved. When training deep learning models, we typically process batches of samples simultaneously, rather than processing one sample at a time. This speeds up training and leverages the parallel computing capabilities of the Graphics Processing Unit (GPU). B refers to the number of samples contained in the batch. In the multi-head attention mechanism, the model calculates multiple attention weights in parallel. Each attention weight is calculated by a "head," and the number of heads is the number of attention weights calculated in parallel. The multi-head attention mechanism helps the model capture different aspects of the input data, improving the model's expressiveness. In natural language processing tasks, the input data is usually in sequence form, such as sentences or paragraphs. The sequence length refers to the number of elements (such as words or characters) contained in the sequence. The sequence length may vary for different input samples.

[0086] Figure 4 shows a two-dimensional schematic diagram of the Block Attention Mask in the causal scenario. As shown in Figure 4, the Block Attention Mask consists of two main parts: the upper right triangular region, which is the causal mask; and the lower left triangular region, which is the newly added block mask. The Block Attention Mask performs a sparse representation in the lower left triangular region and fully masks the upper right triangular region. These two regions are combined to form a complete matrix. In this matrix, all elements in the upper right triangular region (including the diagonal) are masked (i.e., set to a value that does not affect the calculation, such as 0), while the lower left triangular region retains values ​​that may participate in the calculation. This mask design can be used to implement a form of causal or autoregressive attention mechanism, in which the model can only rely on information at a certain position and its previous position when calculating the output, without looking into future information. This is particularly useful when processing sequential data, such as in natural language processing tasks, where the model needs to predict the next word based on the current and past words.

[0087] Specifically, in the Block Attention Mask, the entire masked upper-right triangular area means that when calculating the attention weight for the current position, no information about future positions is considered. This helps the model maintain a causal understanding of the sequence and avoids "peeks" into future data. The sparse representation of the lower-left triangular area allows the model to consider information about the current position and its previous position when calculating attention. This helps the model capture dependencies in the sequence and generate reasonable outputs accordingly. By combining these two areas, the Block Attention Mask implements an attention mechanism that considers historical information while avoiding leakage of future information, thereby improving the efficiency and accuracy of the model when processing sequential data.

[0088] In the disclosed embodiments, Block Attention Mask can be used to implement causal masking, ensuring that the model does not see future information when predicting the current position. Block Attention Mask can also be used to implement local attention or windowed attention, where the model only focuses on specific local areas in the sequence.

[0089] As shown in Figure 4, the blank cells represent the masked cells. The mask corresponding to a head can be expressed as [18, 13, 20, 20, 11, 20, 10, 16, 12, 20, 20, 14, 15, 20, 17, 20, 20, 19, 20, 20]. Among them, 18 means that column 0 is masked from row 18; 13 means that column 1 is masked from row 13; 20 means that column 2 is masked from at least row 20; 20 means that column 3 is masked from at least row 20; 11 means that column 4 is masked from row 11; 20 means that column 5 is masked from at least row 20; 10 means that column 6 is masked from row 10; 16 means that column 7 is masked from row 16; 12 means that column 8 is masked from row 12; 20 means that column 9 is masked from at least row 20; 20 means that column 10 is masked from row 11; Column 10 is masked from at least row 20; 14 means column 11 is masked from row 14; 15 means column 12 is masked from row 15; 20 means column 13 is masked from at least row 20; 17 means column 14 is masked from row 17; 20 means column 15 is masked from at least row 20; 20 means column 16 is masked from at least row 20; 19 means column 17 is masked from row 19; 20 means column 18 is masked from at least row 20; 20 means column 19 is masked from at least row 20. It should be noted that rows and columns are usually numbered starting at 0.

[0090] Figure 5 shows a two-dimensional diagram of a Block Attention Mask during the fine-tuning phase of a large language model. As shown in Figure 5, blank cells represent masked locations, meaning that information at these locations is ignored or excluded when calculating attention. Non-blank cells represent unmasked locations, and the model considers information at these locations when calculating attention. Under the definition of a Block Attention Mask, the corresponding mask in this figure can be expressed as [10, 10, 10, 5, 5, 8, 8, 8, 10, 10].

[0091] Figure 6 shows another two-dimensional diagram of a Block Attention Mask during fine-tuning of a large language model. As shown in Figure 6, blank cells represent masked locations, while non-blank cells represent unmasked locations. Under the definition of a Block Attention Mask, the corresponding mask in this figure can be expressed as [3, 3, 3, 0, 0, 0, 0, 0, 0].

[0092] Figure 7 shows another two-dimensional diagram of a Block Attention Mask during fine-tuning of a large language model. As shown in Figure 7, blank cells represent masked locations, while non-blank cells represent unmasked locations. Under the definition of a Block Attention Mask, the corresponding mask in this diagram can be expressed as [2, 2, 0, 0, 0, 0, 0, 0, 0].

[0093] In some embodiments, a masked sparse representation is used to perform sparse processing on the input data, including: during the processing, the masked positions will be ignored when calculating the attention score or other related operations, thereby achieving sparsity.

[0094] The sparse processing method of this solution can be applied to Causal scenarios and can also be widely used in the following fields and scenarios:

[0095] 1. Causal Inference and Causal Modeling:

[0096] Understanding the causal relationships between variables is crucial in causal reasoning and causal modeling. By applying the sparse processing approach of this solution, causal graphs or causal models can be constructed more efficiently, focusing only on variables that have a direct impact on specific outcomes, thereby simplifying the model and improving the accuracy of inference.

[0097] 2. Generation tasks in natural language processing:

[0098] In natural language processing, generative tasks such as text summarization, machine translation, and conversational systems often require generating elements in a sequence in a specific order. This solution's sparse processing ensures that the model relies solely on previous contextual information when generating each word or sentence, thus adhering to the natural order of language processing.

[0099] 3. Time Series Analysis:

[0100] In time series analysis, such as stock price forecasting and weather pattern recognition, data points often have a clear temporal order. The sparse processing approach used in this solution ensures that the model only considers past information when analyzing time series data, thereby preventing the leakage of future information and improving forecast accuracy.

[0101] 4. Image and video processing:

[0102] Although image and video data themselves do not have a clear order, in some tasks, such as video prediction or image generation, we may want the model to process the data in a certain order. In this case, the sparse processing method of this scheme can be used to constrain the model's attention mechanism to focus on different parts of the image or video in a specific order.

[0103] 5. Recommendation System:

[0104] In recommendation systems, it is often necessary to predict the content that users may be interested in based on their historical behavior and preferences. By using the sparse processing method in this solution, we can ensure that the recommendation algorithm only considers the user's past behavior when making predictions, thereby avoiding the use of future information and improving the accuracy and credibility of recommendations.

[0105] It should be noted that the above scenarios are merely illustrative and non-limiting. Due to its flexibility and efficiency, the sparse processing approach of this solution can be widely applied to various scenarios that require processing sequence data or have sequential dependencies. By combining specific task requirements with data characteristics, the performance and efficiency of the model can be effectively improved.

[0106] In the embodiment of the present disclosure, the memory includes but is not limited to the memory of the central processing unit (CPU) and the memory of the graphics processing unit (GPU).

[0107] In practical applications, a large language model can be deployed on a computing device; based on training samples, the large language model deployed on the computing device is trained to obtain a target model.

[0108] In the process of training a large language model deployed on a computing device, the following processing is performed: a masked sparse representation is constructed using constraints; the constraints are: the dimension of the mask matrix is ​​[B, a, S], B represents the batch size, a represents the number of heads, S represents the sequence length, and each element in the S dimension represents the masked starting row of each column in the mask matrix; and the masked sparse representation is used to perform sparse processing on the input data.

[0109] In the embodiment of the present disclosure, the computing device may be a server device or a terminal device.

[0110] In an embodiment of the present disclosure, the target model may be a natural language processing model, which is used to perform at least one natural language processing including information extraction, text classification, text recognition, speech recognition, and question answering on the first type of data to be processed to obtain a natural language processing result. The target model may also be a computer vision processing model, which is used to perform at least one computer vision processing including image recognition, target detection, semantic segmentation, video understanding, and image generation on the second type of data to be processed to obtain a computer vision processing result. The above is only an exemplary explanation and is not intended to limit all possible types of target models, but it is not intended to be exhaustive here.

[0111] The technical solution of the embodiment of the present disclosure adopts constraints to construct a mask sparse representation, so that the mask of the lower left triangle area can be effectively sparsely represented, which greatly reduces the amount of calculation and memory consumption, and can significantly improve the efficiency of sequence data processing and reduce memory requirements. By adopting preset constraints to construct a mask sparse representation, the memory consumption can be reduced from the square of the sequence length to the linear order of the sequence length, thereby significantly reducing the memory requirements during large model training and improving training efficiency. This sparse representation method has broad application prospects in causal reasoning, natural language processing and other fields, and is expected to provide better performance and efficiency for large-scale model training and applications.

[0112] In the embodiment of the present disclosure, a mask sparse representation is used to perform sparse processing on the input data, including: inputting the query matrix, key matrix, value matrix of the input data and the mask matrix obtained based on the mask sparse representation into the attention calculation formula; wherein the attention calculation formula is: Y = softmax(QK T +Mask)V; where softmax() represents the normalization (softmax) function, Q represents the query matrix, K represents the key matrix, V represents the value matrix, Y represents the output matrix, T represents the transpose operation, and K T Indicates the transposition of the key matrix, Mask represents the mask matrix; the dimensions of Q, K, V and Y are [B, S, H], the dimension of Mask is [B, a, S], and H represents the hidden size.

[0113] In some embodiments, in the attention calculation formula Y=softmax(QK T +Mask)V, each letter represents a specific matrix or operation. The specific meaning of each letter is as follows:

[0114] Q (Query): Query matrix. In the attention mechanism, Q typically represents the information or position that needs to be focused on. Its dimension may depend on the specific model and task, but generally its dimension corresponds to the length of the sequence to be calculated.

[0115] K (Key): Key matrix. Similar to Q, K is also a matrix that represents some information or position. In attention calculations, K is often compared with Q to determine which parts should receive more attention. The dimensions of K are usually the same as Q.

[0116] V (Value): Value matrix. V contains the actual information or representation associated with each position. After attention weighting, these values ​​are used to generate the final output. V is typically of the same dimensions as K and Q (at least in some dimensions), but its specific content may differ from K and Q.

[0117] T: Transpose operation. It means to transpose the matrix K so that K T The rows of Q correspond to the columns of Q, so that the matrix multiplication QK can be performed T .

[0118] QK T : Matrix multiplication operation. This operation is used to calculate the dot product between the transpose of Q and K, generating a score matrix where each element represents the similarity or match between an element in Q and all elements in K.

[0119] Mask: Mask matrix. Mask is a matrix that is related to QK T A matrix of the same shape as the score matrix, used to indicate which positions should be ignored or given lower weights. In sequence processing, masks are often used to process sequences of different lengths (such as padding) or to ignore certain specific positions (such as padding in some tasks).

[0120] softmax(): softmax function. This function converts QK T The result of the +Mask is converted into a probability distribution, ensuring that all values ​​are in the range [0,1] and that the sum of all values ​​is 1. In this way, each position is given a weight indicating its importance relative to other positions.

[0121] Y: Output matrix. Y is the attention-weighted value matrix, computed by applying the result of the softmax function to V. Specifically, the value of each position in V is scaled by its corresponding softmax weight to generate Y.

[0122] In summary, the attention calculation formula describes how to calculate a weighted sum based on the similarity between Q and K and the mask information to generate a Y that incorporates important information. This attention mechanism is widely used in various deep learning models, especially when processing sequence data.

[0123] The technical solution of the disclosed embodiment inputs the query matrix, key matrix, value matrix, and mask matrix obtained based on masked sparse representation of the input data into the attention calculation formula. In this way, the masked sparse representation can reduce the number of elements involved in the attention calculation. Especially when processing long sequences or large data sets, by focusing only on the important elements, it can significantly reduce unnecessary calculations, thereby improving overall computational efficiency.

[0124] In the embodiment of the present disclosure, the mask sparse representation is constructed using constraints, including: executing the operation of constructing the mask sparse representation using constraints based on Flash Attention. Specifically, the operation of constructing the mask sparse representation using constraints is executed in the Flash Attention module.

[0125] In some implementations, Flash Attention is an attention mechanism that can simulate the human visual attention mechanism in a neural network. The attention mechanism focuses on different parts of the input by assigning different weights. It generates a new representation by calculating the weighted sum of the input data, making it more efficient when processing large amounts of data.

[0126] In some embodiments, the operation of constructing a masked sparse representation using constraints based on Flash Attention includes: when processing data using Flash Attention, a mask matrix is ​​generated based on preset constraints. This mask matrix is ​​used to implement a sparse representation of the data. Specific steps may include:

[0127] 1. Define pre-defined constraints: These constraints are typically set based on the characteristics and requirements of the task. For example, in a causal inference task, a constraint might require that the model not look at future words when predicting the next word; in a sequence generation task in natural language processing, a constraint might limit the model to focusing only on the input portion relevant to the current output.

[0128] 2. Construct a mask matrix: Based on these preset constraints, a mask matrix is ​​generated. The size of the mask matrix usually matches the dimension of the input data, and its element values ​​are used to indicate which parts of the data should be paid attention to (usually set to 1) and which parts should be ignored or reduced in weight (usually set to 0 or a small value).

[0129] 3. Apply the mask matrix: The constructed mask matrix is ​​combined with the query matrix, key matrix, and value matrix of the input data, and then calculated using the Flash Attention module. During the calculation process, the mask matrix plays a filtering and weighting role, allowing the model to focus on key information and ignore unimportant parts.

[0130] 4. Achieving sparse representation: By applying a mask matrix, the Flash Attention module can output a sparse representation. This sparse representation not only reduces the dimensionality and complexity of the data, but also improves the model's efficiency and accuracy in processing data.

[0131] The technical solution of the disclosed embodiments uses a Flash Attention module to construct a masked sparse representation using constraints. This sparse representation of the mask matrix allows the model to focus on the key components of the input data and skip unimportant information, reducing the waste of computing resources and significantly improving computational efficiency, especially when processing large datasets.

[0132] In an embodiment of the present disclosure, the operation of constructing a masked sparse representation using constraints is executed in the Flash Attention module, including: before the Flash Attention module calls the Block Attention mechanism, the operation of constructing a masked sparse representation using constraints is executed.

[0133] Here, before calling the Block Attention mechanism, it specifically refers to the time of constructing the mask sparse representation.

[0134] Figure 8 shows a schematic diagram of the operation of constructing a masked sparse representation using constraints before calling the Block Attention mechanism. As shown in Figure 8, each Transformer module contains an attention module (i.e., Attention Block) and a Feed Forward Networks (FFN) module; each Attention module includes two operations: constructing a masked sparse representation and calling Block Attention. Here, the program or component used to perform the operation of constructing a masked sparse representation can be called a sparse mask construction module; the program or component used to perform the operation of calling Block Attention is called a calling Block Attention module. Specifically, Q, K, V, and Block Attention Mask are input into the calling Block Attention module, and the masked area can be skipped in the actual calculation process. For example, those masked blocks can be skipped in the specific implementation of the Compute Unified Device Architecture Kernel (CUDA kernel). Constructing a mask before the Block Attention module can dynamically generate a mask based on the current state of the model or the characteristics of the input data, increasing flexibility. After the data has undergone a series of transformations and processing, some features or data points may no longer be important. In this case, constructing a mask before Block Attention can further reduce unnecessary calculations. In some embodiments, a masked sparse representation can be constructed in a memory unit; the computing unit obtains the masked sparse representation from the memory unit, uses the masked sparse representation to perform sparse processing on the input data, and stores the sparse processing results in the memory unit.

[0135] The technical solution of the embodiment of the present disclosure performs the operation of constructing a mask sparse representation using constraints before calling the Block Attention mechanism. In this way, by removing irrelevant information in advance, the computational complexity of the subsequent Block Attention mechanism is reduced, making the entire processing process more efficient.

[0136] In an embodiment of the present disclosure, an operation of constructing a masked sparse representation using constraints is performed in a Flash Attention module, including: in response to the Flash Attention module detecting a first construction instruction, constructing the same masked sparse representation for each Transformer module in the model based on the first construction instruction.

[0137] In some embodiments, the first construction instruction is used to generate the same mask sparse representation for all Transformer modules (ie, Transformer Block) in the model before calling the Block Attention mechanism.

[0138] In some embodiments, in each Transformer module, the system uses a predefined rule or algorithm to generate a masked sparse representation according to the first construction instruction, and applies the same masked sparse representation to each Transformer module in the model to ensure that they all use the same masked sparse representation.

[0139] The technical solution of the embodiment of the present disclosure, in response to detecting a first construction instruction, constructs the same masked sparse representation for each Transformer module in the model based on the first construction instruction. In this way, the same masked sparse representation is constructed for each Transformer module in the model, ensuring that the model applies the same sparsity pattern when processing data at different layers, which helps the model maintain consistent behavior at different levels and ensures the consistency of the model when processing data. Since all Transformer modules use the same masking rules, the model can maintain a unified perspective and logic when extracting features and making decisions, which helps to enhance the stability, interpretability and reliability of the model, reduce the complexity of the model, and make the model easier to understand and optimize.

[0140] In an embodiment of the present disclosure, an operation of constructing a mask sparse representation using constraints is performed in the Flash Attention module, including: in response to the Flash Attention module detecting a second construction instruction, constructing a different mask sparse representation for each Transformer module in the model based on the second construction instruction.

[0141] In some embodiments, the second construction instruction is used to provide a specific mask strategy for each Transformer module before calling the Block Attention mechanism.

[0142] In some embodiments, the system uses a predefined rule or algorithm to generate a unique mask for each Transformer module, ensuring that the masked sparse representations are different between them. This allows for flexible information processing based on the different characteristics of the input data and task requirements.

[0143] The technical solution of the disclosed embodiment constructs a different masked sparse representation for each Transformer module in the model based on the second construction instruction. These different masked sparse representations allow each Transformer module to focus on different parts of the input, enhancing the model's flexibility and expressiveness. Constructing a different masked sparse representation for each Transformer module in the model allows for customized masking strategies based on the needs and characteristics of different modules, better adapting to the demands of different data and tasks and helping to improve the model's adaptability and generalization capabilities.

[0144] In an embodiment of the present disclosure, a mask sparse representation of a Block Attention mechanism is constructed using constraint conditions, including: performing an operation of constructing the mask sparse representation using constraint conditions in a data reader module.

[0145] In some embodiments, the data reading module is a dedicated module for reading data from a data source. Its main function is to import data stored in an external data source (such as a database, file, application programming interface (API)), etc.) into the system for further processing, analysis, or calculation.

[0146] Figure 9 illustrates the operation of constructing a masked sparse representation using constraints in the data reading module. As shown in Figure 9, each Transformer module contains an Attention module and an FFN module; each Attention module calls a Block Attention module. The data reading module constructs the masked sparse representation. This step typically occurs before model input. The data reading module's primary responsibility is to load and process raw data to provide suitable input for the model. During this process, constructing a masked sparse representation based on pre-set constraints is part of data preprocessing, allowing for sparsity to be introduced during model training or inference. Specifically, after reading the raw data, the data reading module constructs a masked sparse representation for each data point (or data batch) based on pre-set constraints (such as specific rules, thresholds, or conditional expressions). This mask is applied to certain portions of the data, causing them to be ignored or have their weight reduced in subsequent computations. In this way, the data reading module ensures that the data has been sparsified according to specific requirements before the model begins processing it. The output of the data reading module (i.e., data processed using the masked sparse representation) serves as the model input and enters the model's training or inference process. This design allows sparsity to be introduced early in the model processing process, effectively impacting model performance and computational efficiency. Placing the construction of the mask as a separate step in the data preprocessing phase makes the entire data processing process more modular, easier to understand, and easier to maintain. By inputting the masked sparse representation along with the input data into the Transformer module and invoking Block Attention within the module based on the masked sparse representation, sparsity processing and selective attention on the input data can be achieved. Specifically, Q, K, V, and the Block Attention Mask are input to the Block Attention module. Masked areas can be skipped during actual computation. For example, masked blocks can be skipped in the implemented CUDA kernel. In some embodiments, the masked sparse representation can be read from disk into a memory unit; the computation unit retrieves the masked sparse representation from the memory unit, performs sparse processing on the input data using the masked sparse representation, and stores the sparse processing results in the memory unit.

[0147] The technical solution of the embodiment of the present disclosure performs an operation of constructing a mask sparse representation using constraint conditions in the data reading module, so that sparsity can be introduced in the early stage of model processing, thereby effectively affecting the performance and computational efficiency of the model. The data reading module can perform targeted screening and filtering when reading data, that is, only data that meets specific conditions or requirements will be read and processed, which helps to reduce the complexity and computational complexity of subsequent data processing and analysis. In addition, placing the construction of the Mask as an independent step in the data preprocessing stage can make the entire data processing process more modular, easy to understand and maintain.

[0148] In an embodiment of the present disclosure, an operation of constructing a masked sparse representation using constraints is performed in a data reading module, including: in response to the data reading module detecting a third construction instruction, constructing the same masked sparse representation for each Transformer module in the model based on the third construction instruction.

[0149] In some embodiments, the third construction instruction is used to execute in the data reading module to generate the same mask sparse representation for all Transformer modules in the model.

[0150] In some embodiments, in the data reading module, the system uses a predefined rule or algorithm to generate a masked sparse representation according to the third construction instruction, and applies the same masked sparse representation to each Transformer module in the model to ensure that they all use the same masked sparse representation. Since all modules share the same masked sparse representation, this consistency can be used in implementation to optimize calculations, for example, by only calculating outputs corresponding to non-zero weights when needed. Consistent mask patterns may help understand how the model makes decisions because it can be easier to identify which input features or model components have an important impact on the output, enhancing the interpretability of the decision.

[0151] In practice, the specific implementation details and effects depend on the specific content of the third construct and how it is defined and applied. This usually requires a deep understanding of the model architecture, data, and training objectives.

[0152] The technical solution of the embodiment of the present disclosure constructs the same masked sparse representation for each Transformer module in the model based on the third construction instruction in the data reading module. In this way, constructing the same masked sparse representation for each Transformer module in the model ensures the consistency of the model when processing data. Because all modules use the same masking rules, the model can maintain a unified perspective and logic when extracting features and making decisions, which helps to enhance the stability, interpretability and reliability of the model.

[0153] In an embodiment of the present disclosure, an operation of constructing a masked sparse representation using constraints is performed in a data reading module, including: in response to the data reading module detecting a fourth construction instruction, constructing a different masked sparse representation for each Transformer module in the model based on the fourth construction instruction.

[0154] In some embodiments, the fourth construction instruction is used to execute in the data reading module to provide a specific mask strategy for each Transformer module.

[0155] In some embodiments, in the data reading module, the system uses a predefined rule or algorithm to generate multiple different masked sparse representations according to the fourth construction instruction, and assigns a unique mask to each Transformer module to ensure that the masked sparse representations are different between them. This allows for flexible information processing based on the different characteristics of the input data and task requirements.

[0156] The technical solution of the embodiment of the present disclosure constructs a masked sparse representation through constraints, so that the data reading module can perform targeted screening and filtering when reading data, that is, only data that meets specific conditions or requirements will be read and processed, which helps to reduce the complexity and computational complexity of subsequent data processing and analysis. In this way, each Transformer module in the model will have a different masked sparse representation, so that it can focus on different features or patterns when processing input data. This design can improve the performance of the model because it allows the model to capture the complexity of the input data at multiple levels. At the same time, it may also increase the complexity and training difficulty of the model, so the masking strategy needs to be carefully adjusted and optimized to ensure optimal performance.

[0157] It should be understood that the schematic diagrams shown in Figures 1, 2, 4, 5, 6, 7, 8 and 9 are merely illustrative and not restrictive, and are expandable. Those skilled in the art can make various obvious changes and / or replacements based on the examples of Figures 1, 2, 4, 5, 6, 7, 8 and 9, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0158] An embodiment of the present disclosure provides a method for fine-tuning a large language model. FIG10 is a flow chart of the method for fine-tuning a large language model according to an embodiment of the present disclosure. The method for fine-tuning a large language model can be applied to a fine-tuning device for a large language model. The fine-tuning device for a large language model is located in an electronic device. The electronic device includes but is not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, and servers can be cloud servers or ordinary servers. For example, mobile devices include but are not limited to mobile phones, tablet computers, etc. In some possible implementations, the method for fine-tuning a large language model can also be implemented by a processor calling computer-readable instructions stored in a memory. As shown in FIG10 , the method for fine-tuning a large language model includes:

[0159] S1001: Obtain a dataset for the target task;

[0160] S1002: Fine-tune the large language model based on the dataset to adapt the large language model to the requirements of the target task.

[0161] Among them, fine-tuning the large language model based on the data set includes: combining the sparse attention mechanism to fine-tune the large language model based on the data set; among them, the sparse attention mechanism adopts the sparse processing method of the sparse attention network described above, and the sparse attention mechanism here is the Block Attention Mechanism.

[0162] Here, obtaining a dataset for the target task is one of the key steps in a machine learning project. The quality and diversity of the dataset are crucial for training an efficient model with strong generalization capabilities. For example, data can be obtained from public datasets. For another example, data can be obtained from websites, social media, or APIs using web crawlers. For another example, data can be collected and labeled independently. It should be noted that the above data acquisition methods are merely illustrative and non-restrictive, and this disclosure does not limit the methods for obtaining datasets.

[0163] When fine-tuning large language models, applying sparse processing to the Attention module in each Transformer module is an effective strategy that can significantly improve the efficiency and performance of the model. The following are the detailed processing steps:

[0164] First, we need to understand the basic structure of a large language model, which is usually composed of multiple stacked Transformer modules. Each Transformer module contains an Attention module, which enables the model to capture the key part of the dependency between different positions in the input sequence.

[0165] During fine-tuning training, the Attention module is masked using a sparse processing method. Sparse processing generally means focusing on only a portion of the key information in the input data, rather than all of it. This is done to reduce the computational burden of the model during training while maintaining or improving model performance.

[0166] There are many ways to implement sparse mask processing. One common approach is to use a local attention or block attention mechanism. Local attention means that each position only pays attention to a small number of nearby positions, rather than the entire sequence. Block attention divides the input sequence into multiple blocks, and attention calculations are performed only between positions within each block.

[0167] During fine-tuning, for the Attention module in each Transformer module, we can introduce a sparse mask matrix. The size of the sparse mask matrix is ​​the same as the Attention matrix, but most of its elements are 0, and only a few key positions have elements of 1. By multiplying the Attention matrix with the sparse mask matrix, we can achieve sparseness in the Attention calculation.

[0168] By doing this, we can effectively reduce the amount of computation required to train the model while keeping the model focused on key information. This not only speeds up the training process but also reduces the model's demand for computing resources, making it easier to deploy in resource-constrained environments.

[0169] It is important to note that the specific implementation of sparse mask processing may vary depending on the model structure and task requirements. Therefore, in practical applications, it is necessary to select an appropriate sparse processing method based on the specific situation and conduct sufficient experimental verification to ensure that the fine-tuned model can achieve the expected performance improvement.

[0170] During fine-tuning, certain model parameters, such as the learning rate, batch size, or regularization term, can be adjusted to optimize the model's performance on a specific task. At the same time, thanks to the sparse attention mechanism, the fine-tuning process can utilize computing resources more efficiently, as the model only needs to focus on the key parts of the input sequence.

[0171] Combining the sparse attention mechanism and fine-tuning the large language model based on the dataset has at least the following effects:

[0172] 1. Improved computational efficiency: The sparse attention mechanism enables the model to focus on important parts while ignoring irrelevant information when calculating attention weights. This reduces the amount of computation, speeds up the training process, and reduces the demand for computing resources.

[0173] 2. Reduced memory consumption: Due to sparse representation, the model only needs to save non-zero elements when storing attention weights, which greatly reduces memory usage. This is especially important for processing large models and long sequences, as it makes training more feasible with limited hardware resources.

[0174] 3. Model performance optimization: By combining sparse attention and fine-tuning, the model can better adapt to the needs of the target task. Sparse attention helps the model capture key information, while fine-tuning enables the model to utilize the characteristics of the specific dataset to improve performance on the target task.

[0175] 4. Improved generalization: Fine-tuning training enables the model to learn task-specific knowledge, thereby improving its generalization ability on the target task. By adjusting the model's parameters to adapt to the new dataset, the model is better able to handle unseen situations with a distribution similar to the training data.

[0176] 5. Enhanced model interpretability: The sparse attention mechanism makes the model’s decision-making more transparent and explainable. Because the model focuses on a few key elements, we can more easily understand how the model makes decisions, which is crucial for trust and reliability in real-world applications.

[0177] The technical solution of the disclosed embodiment, combined with the sparse attention mechanism to fine-tune the large language model, can not only improve computing efficiency and reduce memory consumption, but also optimize model performance, improve generalization ability, and enhance the interpretability of the model, bringing significant advantages to large language model tasks in practical applications.

[0178] An embodiment of the present disclosure provides a data processing method for a large language model. FIG11 is a flow chart of the data processing method for a large language model according to an embodiment of the present disclosure. The data processing method for a large language model can be applied to a data processing device for a large language model. The data processing device for a large language model is located in an electronic device. The electronic device includes but is not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, and servers can be cloud servers or ordinary servers. For example, mobile devices include but are not limited to mobile phones, tablet computers, etc. In some possible implementations, the data processing method for a large language model can also be implemented by a processor calling computer-readable instructions stored in a memory. As shown in FIG11 , the data processing method for a large language model includes:

[0179] S1101: Obtain input data;

[0180] S1102: Input the input data into the large language model so that the large language model processes the input data in combination with the sparse attention mechanism to generate output data; wherein the sparse attention mechanism adopts the sparse processing method of the sparse attention network described above, and the sparse attention mechanism here is the Block Attention Mechanism.

[0181] In the embodiments of the present disclosure, input data can be text data in various forms, which vary depending on the specific application scenario and task requirements. Input data includes, but is not limited to, natural language text, question query statements, conversation history, code snippets or related natural language descriptions, queries for specific knowledge or facts, and structured data.

[0182] In the disclosed embodiment, the output data is data obtained by the large language model based on the input data.

[0183] In the disclosed embodiment, input data is fed into a large language model and processed in conjunction with a sparse attention mechanism to generate output data. The beneficial effects are mainly reflected in the following aspects:

[0184] 1. Improved processing efficiency: The sparse attention mechanism uses sparse processing to enable the model to ignore unimportant parts while focusing on key information, thereby reducing the amount of computation. This efficient attention allocation method can significantly increase the speed at which large language models can process input data, especially when dealing with long sequences or large-scale datasets.

[0185] 2. Reduced resource consumption: Because the sparse attention mechanism focuses only on key information, the memory and computing resources required by the model are greatly reduced during storage and calculation. This is a key optimization for the hardware environment running large language models, enabling the processing of larger data or more complex models under limited resource conditions.

[0186] 3. Improved model performance: Large language models combined with sparse attention mechanisms can more accurately capture key information in input data, thereby generating more precise and valuable output data. This performance improvement can be seen in a variety of natural language processing tasks, such as text classification, sentiment analysis, and question-answering systems.

[0187] 4. Enhanced model generalization: Through the sparse attention mechanism, the model can better focus on the core features of the input data, making it more adaptable when processing text from different fields or styles. As a result, the model's generalization ability is improved, allowing it to be used in a wider range of tasks and scenarios.

[0188] 5. Improved interpretability: The sparse attention mechanism makes the model more transparent and interpretable in its decision-making process. By looking at the model’s attention distribution, we can understand how the model allocates attention when processing input data, which is important for understanding and optimizing the model’s behavior.

[0189] The technical solution of the disclosed embodiments combines a large language model with a sparse attention mechanism to process input data, which not only improves processing efficiency and reduces resource consumption, but also plays a significant role in enhancing model performance, generalization, and interpretability. This combination has broad prospects and significant value in practical applications, especially in scenarios that require processing large amounts of text data or performing complex natural language processing tasks.

[0190] The embodiment of the present disclosure provides a sparse processing device for a sparse attention network, which is applied to a computing device, wherein the computing device includes a computing unit and a memory unit. As shown in Figure 12, the sparse processing device for the sparse attention network may include: a construction module 1201, which is used to construct a masked sparse representation in the memory unit using a constraint condition; wherein the constraint condition is: the dimension of the mask matrix is ​​[B, a, S], B represents the batch size, a represents the number of heads, S represents the sequence length, and each element in the S dimension represents the starting row of each column in the mask matrix to be masked; a sparse processing module 1202, which is used to control the computing unit to obtain the masked sparse representation from the memory unit, use the masked sparse representation to perform sparse processing on the input data, and store the sparse processing result in the memory unit.

[0191] In some embodiments, the sparse processing module 1202 is used to input the query matrix, key matrix, value matrix of the input data and the mask matrix obtained based on the mask sparse representation into the attention calculation formula to obtain the output matrix of the input data; wherein the attention calculation formula is: Y = softmax(QK T +Mask)V; softmax() represents the normalization function, Q represents the query matrix, K represents the key matrix, V represents the value matrix, Y represents the output matrix, T represents the transpose operation, K T Indicates the transposition of the key matrix, Mask represents the mask matrix; the dimensions of Q, K, V and Y are [B, S, H], the dimension of Mask is [B, a, S], and H represents the hidden size.

[0192] In some embodiments, the construction module 1201 includes: a first construction submodule, configured to execute an operation of constructing a mask sparse representation using constraint conditions in a Flash Attention module.

[0193] In some embodiments, the first construction submodule is configured to: in response to the Flash Attention module detecting the first construction instruction, construct the same mask sparse representation for each Transformer module in the model based on the first construction instruction.

[0194] In some embodiments, the first construction submodule is configured to: in response to the Flash Attention module detecting the second construction instruction, construct a different mask sparse representation for each Transformer module in the model based on the second construction instruction.

[0195] In some embodiments, the construction module 1201 includes: a second construction submodule, configured to execute an operation of constructing a mask sparse representation using constraint conditions in a data reading module.

[0196] In some embodiments, the second construction submodule is configured to: in response to the data reading module detecting the third construction instruction, construct the same masked sparse representation for each Transformer module in the model based on the third construction instruction.

[0197] In some embodiments, the second construction submodule is configured to: in response to the data reading module detecting the fourth construction instruction, construct a different masked sparse representation for each Transformer module in the model based on the fourth construction instruction.

[0198] Those skilled in the art should understand that the functions of each processing module in the sparse processing device of the sparse attention network of the embodiment of the present disclosure can be understood with reference to the relevant description of the sparse processing method of the sparse attention network mentioned above. Each processing module in the sparse processing device of the sparse attention network of the embodiment of the present disclosure can be realized by a generation circuit that realizes the function of the embodiment of the present disclosure, and can also be realized by running software that executes the function of the embodiment of the present disclosure on an electronic device.

[0199] The sparse processing device of the sparse attention network of the embodiment of the present disclosure, by adopting constraint conditions to construct the masked sparse representation of the Block Attention mechanism, can reduce the memory consumption from the square of the sequence length to the linear order of the sequence length, thereby significantly reducing the memory requirement during large model training and improving training efficiency; at the same time, through further optimization in Flash Attention, the computing performance of large models can be improved.

[0200] Figure 13 is a schematic diagram of a fine-tuning device for a large language model according to an embodiment of the present disclosure. As shown in Figure 13, the fine-tuning device for a large language model includes: a first acquisition module 1301, used to acquire a data set for a target task; and a fine-tuning module 1302, used to perform fine-tuning training on the large language model based on the data set to adapt the large language model to the requirements of the target task.

[0201] Among them, the fine-tuning module 1302 is used to: combine the sparse attention mechanism to fine-tune the large language model based on the data set; among them, the sparse attention mechanism adopts the sparse processing method of the sparse attention network described above, and the sparse attention mechanism here is the Block Attention Mechanism.

[0202] Those skilled in the art should understand that the functions of each processing module in the fine-tuning device of the large language model in the embodiment of the present disclosure can be understood with reference to the relevant description of the fine-tuning method of the large language model described above. Each processing module in the fine-tuning device of the large language model in the embodiment of the present disclosure can be implemented by a generation circuit that implements the function of the embodiment of the present disclosure, or can be implemented by running software that executes the function of the embodiment of the present disclosure on an electronic device.

[0203] The fine-tuning device for a large language model in the embodiment of the present disclosure, combined with a sparse attention mechanism to perform fine-tuning training on the large language model, can not only improve computing efficiency and reduce memory consumption, but also optimize model performance, improve generalization ability, and enhance the interpretability of the model, bringing significant advantages to large language model tasks in practical applications.

[0204] Figure 14 is a schematic diagram of a data processing device for a large language model according to an embodiment of the present disclosure. As shown in Figure 14, the data processing device for the large language model includes: a second acquisition module 1401, used to acquire input data; a data processing module 1402, used to input the input data into the large language model, so that the large language model processes the input data in combination with the sparse attention mechanism to generate output data; wherein the sparse attention mechanism adopts the sparse processing method of the sparse attention network described above, and the sparse attention mechanism here is the Block Attention Mechanism.

[0205] Those skilled in the art should understand that the functions of each processing module in the data processing device of the large language model of the embodiment of the present disclosure can be understood with reference to the relevant description of the data processing method of the large language model described above. Each processing module in the data processing device of the large language model of the embodiment of the present disclosure can be implemented by a generation circuit that implements the function of the embodiment of the present disclosure, or can be implemented by running software that executes the function of the embodiment of the present disclosure on an electronic device.

[0206] The data processing device for a large language model in the disclosed embodiments combines a large language model with a sparse attention mechanism to process input data, significantly improving processing efficiency while also effectively reducing resource consumption. This combination offers broad prospects and significant value in practical applications, particularly in scenarios requiring processing large amounts of text data or performing complex natural language processing tasks.

[0207] An embodiment of the present disclosure provides a scenario diagram of sparse processing of a sparse attention network, as shown in FIG15 .

[0208] As previously mentioned, the sparse processing method for the sparse attention network provided by the embodiments of the present disclosure is applied to electronic devices. The term "electronic device" is intended to refer to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The term "electronic device" may also refer to various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0209] Specifically, the electronic device can perform the following operations:

[0210] A masked sparse representation is constructed in the memory unit using constraints. The constraints are as follows: the dimension of the mask matrix is ​​[B, a, S], where B represents the batch size, a represents the number of heads, and S represents the sequence length. Each element in the S dimension represents the starting row of each column in the mask matrix to be masked.

[0211] The computing unit obtains the masked sparse representation from the memory unit, performs sparse processing on the input data using the masked sparse representation, and stores the sparse processing result in the memory unit.

[0212] The input data can be obtained from a data source. The data source can be various forms of data storage devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The data source can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. Furthermore, the data source and the user terminal can be the same device.

[0213] Among them, the constraint conditions can be set by the large model training programmer or the large model training team.

[0214] It should be understood that the scene diagram shown in Figure 15 is merely illustrative and not restrictive. Those skilled in the art can make various obvious changes and / or substitutions based on the example of Figure 15, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0215] An embodiment of the present disclosure provides a schematic diagram of a scenario for fine-tuning a large language model, as shown in FIG16 .

[0216] As previously mentioned, the fine-tuning method for a large language model provided in the embodiments of the present disclosure is applied to electronic devices. The term "electronic device" is intended to refer to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The term "electronic device" may also refer to various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0217] Specifically, the electronic device can perform the following operations:

[0218] Obtain a dataset for the target task;

[0219] Combined with the sparse attention mechanism, the large language model is fine-tuned based on the dataset to adapt it to the needs of the target task. The sparse attention mechanism adopts the sparse processing method of the sparse attention network described above.

[0220] The data set can be obtained from a data source. The data source can be various forms of data storage devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The data source can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. Furthermore, the data source and the user terminal can be the same device.

[0221] It should be understood that the scene diagram shown in Figure 16 is merely illustrative and not restrictive. Those skilled in the art can make various obvious changes and / or substitutions based on the example of Figure 16, and the resulting technical solution still falls within the scope of the disclosure of the embodiments of the present disclosure.

[0222] An embodiment of the present disclosure provides a schematic diagram of a scenario for data processing of a large language model, as shown in FIG17 .

[0223] As previously mentioned, the data processing method for a large language model provided in the embodiments of the present disclosure is applied to electronic devices. The term "electronic device" is intended to refer to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The term "electronic device" may also refer to various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0224] Specifically, the electronic device can perform the following operations:

[0225] Get input data;

[0226] Input data is input into the large language model so that the large language model processes the input data in combination with the sparse attention mechanism to generate output data; wherein the sparse attention mechanism adopts the sparse processing method of the sparse attention network described above.

[0227] The input data can be obtained from a data source. The data source can be various forms of data storage devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The data source can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. Furthermore, the data source and the user terminal can be the same device.

[0228] It should be understood that the scene diagram shown in Figure 17 is merely illustrative and not restrictive. Those skilled in the art can make various obvious changes and / or substitutions based on the example of Figure 17, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0229] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0230] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0231] FIG18 shows a schematic block diagram of an example electronic device 1800 that can be used to implement an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0232] As shown in FIG18 , device 1800 includes a computing unit 1801 that can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 1802 or a computer program loaded from a storage unit 1808 into a random access memory (RAM) 1803. Various programs and data required for the operation of device 1800 may also be stored in RAM 1803. Computing unit 1801, ROM 1802, and RAM 1803 are connected to each other via a bus 1804. An input / output (I / O) interface 1805 is also connected to bus 1804.

[0233] Various components in device 1800 are connected to I / O interface 1805, including an input unit 1806, such as a keyboard and mouse; an output unit 1807, such as various types of displays and speakers; a storage unit 1808, such as a magnetic disk and optical disk; and a communication unit 1809, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1809 allows device 1800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0234] The computing unit 1801 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 1801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1801 performs the various methods and processes described above, such as the sparse processing method for the sparse attention network and / or the fine-tuning method for the large language model and / or the data processing method for the large language model. For example, in some embodiments, the sparse processing method for the sparse attention network and / or the fine-tuning method for the large language model and / or the data processing method for the large language model can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 1808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1800 via the ROM 1802 and / or the communication unit 1809. When the computer program is loaded into RAM 1803 and executed by the computing unit 1801, one or more steps of the sparse processing method for the sparse attention network and / or the fine-tuning method for the large language model and / or the data processing method for the large language model described above may be performed. Alternatively, in other embodiments, the computing unit 1801 may be configured to execute the sparse processing method for the sparse attention network and / or the fine-tuning method for the large language model and / or the data processing method for the large language model by any other appropriate means (e.g., by means of firmware).

[0235] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0236] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0237] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0238] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0239] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0240] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0241] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0242] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A sparse processing method for a sparse attention network, applied to a computing device, the computing device comprising a computing unit and a memory unit, the method comprising: Constructing a masked sparse representation in the memory unit using constraints; wherein the constraints are: the dimension of the mask matrix is ​​[B, a, S], B represents the batch size, a represents the number of heads, S represents the sequence length, and each element in the S dimension represents the starting row of each masked column in the mask matrix; The computing unit obtains the masked sparse representation from the memory unit, performs sparse processing on input data using the masked sparse representation, and stores the sparse processing result in the memory unit.

2. The method according to claim 1, wherein The step of performing sparse processing on input data by using the mask sparse representation includes: Inputting the query matrix, key matrix, value matrix of the input data and the mask matrix obtained based on the mask sparse representation into an attention calculation formula to obtain an output matrix of the input data; The attention calculation formula is: Y = softmax(QK T +Mask)V; softmax() represents a normalization function, Q represents the query matrix, K represents the key matrix, V represents the value matrix, Y represents the output matrix, T represents a transpose operation, K T Indicates transposing the key matrix, Mask indicates the mask matrix; the dimensions of Q, K, V, and Y are [B, S, H], the dimension of Mask is [B, a, S], and H indicates the hidden size.

3. The method according to claim 1, wherein The method of constructing a mask sparse representation using constraint conditions includes: The operation of constructing a masked sparse representation using constraints is performed in the transient attention module.

4. The method according to claim 3, wherein: The operation of constructing a mask sparse representation using constraints in the instantaneous attention module includes: In response to the transient attention module detecting a first construction instruction, constructing the same masked sparse representation for each Transformer module in the model based on the first construction instruction.

5. The method according to claim 3, wherein The operation of constructing a mask sparse representation using constraints in the instantaneous attention module includes: In response to the transient attention module detecting a second construction instruction, a different masked sparse representation is constructed for each Transformer module in the model based on the second construction instruction.

6. The method according to claim 1, wherein The method of constructing a mask sparse representation using constraint conditions includes: The operation of constructing the mask sparse representation using constraints is performed in the data reading module.

7. The method according to claim 6, wherein: The operation of constructing a mask sparse representation using constraint conditions in the data reading module includes: In response to the data reading module detecting a third construction instruction, the same masked sparse representation is constructed for each Transformer module in the model based on the third construction instruction.

8. The method according to claim 6, wherein: The operation of constructing a mask sparse representation using constraint conditions in the data reading module includes: In response to the data reading module detecting a fourth construction instruction, a different masked sparse representation is constructed for each Transformer module in the model based on the fourth construction instruction.

9. A method for fine-tuning a large language model, comprising: Obtain a dataset for the target task; Fine-tune the large language model based on the data set to adapt the large language model to the requirements of the target task; The fine-tuning training of the large language model based on the data set includes: In combination with a sparse attention mechanism, fine-tuning the large language model based on the dataset; The sparse attention mechanism adopts the sparse processing method of the sparse attention network described in any one of claims 1 to 8.

10. A data processing method for a large language model, comprising: Get input data; Inputting the input data into the large language model, so that the large language model processes the input data in combination with a sparse attention mechanism to generate output data; The sparse attention mechanism adopts the sparse processing method of the sparse attention network described in any one of claims 1 to 8.

11. A sparse processing device for a sparse attention network, applied to a computing device, the computing device comprising a computing unit and a memory unit, the device comprising: A construction module, configured to construct a mask sparse representation in the memory unit using constraints; Wherein, the constraint condition is: the dimension of the mask matrix is ​​[B, a, S], B represents the batch size, a represents the number of heads, S represents the sequence length, and each element in the S dimension represents the starting row of each column in the mask matrix to be masked; The sparse processing module is used to control the computing unit to obtain the mask sparse representation from the memory unit, perform sparse processing on the input data using the mask sparse representation, and store the sparse processing result in the memory unit.

12. The device according to claim 11, wherein The sparse processing module is used to: Inputting the query matrix, key matrix, value matrix of the input data and the mask matrix obtained based on the mask sparse representation into an attention calculation formula to obtain an output matrix of the input data; The attention calculation formula is: Y = softmax(QK T +Mask)V; softmax() represents a normalization function, Q represents the query matrix, K represents the key matrix, V represents the value matrix, Y represents the output matrix, T represents a transpose operation, K T Indicates transposing the key matrix, Mask indicates the mask matrix; the dimensions of Q, K, V, and Y are [B, S, H], the dimension of Mask is [B, a, S], and H indicates the hidden size.

13. The device according to claim 11, wherein The construction module comprises: The first construction submodule is used to perform an operation of constructing a mask sparse representation using constraints in the instantaneous attention module.

14. The device according to claim 13, wherein The first construction submodule is used to: In response to the transient attention module detecting a first construction instruction, constructing the same masked sparse representation for each Transformer module in the model based on the first construction instruction.

15. The device according to claim 13, wherein The first construction submodule is used to: In response to the transient attention module detecting a second construction instruction, a different masked sparse representation is constructed for each Transformer module in the model based on the second construction instruction.

16. The device according to claim 11, wherein The construction module comprises: The second construction submodule is used to execute the operation of constructing the mask sparse representation using the constraint conditions in the data reading module.

17. The device according to claim 16, wherein The second construction submodule is used to: In response to the data reading module detecting a third construction instruction, the same masked sparse representation is constructed for each Transformer module in the model based on the third construction instruction.

18. The device according to claim 16, wherein The second construction submodule is used to: In response to the data reading module detecting a fourth construction instruction, a different masked sparse representation is constructed for each Transformer module in the model based on the fourth construction instruction.

19. A fine-tuning device for a large language model, comprising: The first acquisition module is used to obtain a data set for a target task; A fine-tuning module, configured to perform fine-tuning training on the large language model based on the data set, so as to adapt the large language model to the requirements of the target task; The fine-tuning training of the large language model based on the data set includes: In combination with a sparse attention mechanism, fine-tuning the large language model based on the dataset; The sparse attention mechanism adopts the sparse processing method of the sparse attention network described in any one of claims 1 to 8.

20. A data processing device for a large language model, comprising: A second acquisition module is used to acquire input data; a data processing module, configured to input the input data into the large language model, so that the large language model processes the input data in combination with a sparse attention mechanism to generate output data; The sparse attention mechanism adopts the sparse processing method of the sparse attention network described in any one of claims 1 to 8.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: Computer instructions for causing a computer to perform a method according to any one of claims 1 to 10.

23. A computer program product comprising a computer program stored on a storage medium, the computer program implementing the method according to any one of claims 1 to 10 when executed by a processor.

Images