Sparse processing method for sparse attention networks, fine-tuning method for large-scale language models, data processing method for large-scale language models, apparatus, electronic device, and program

Abstract

This disclosure relates to a sparse processing method for a sparse attention network. Methods for fine-tuning large-scale language models, methods for processing data of large-scale language models, Device 、 Electronic devices and program This disclosure provides a solution relating to the field of artificial intelligence technology, particularly to the fields of large-scale models, deep learning, and natural language processing. The specific solution involves constructing a masked sparse representation in a memory unit using constraints, where the constraints are that the dimensions of the mask matrix are [B, a, S], where B represents the batch size, a represents the number of heads, S represents the sequence length, and each element in dimension S represents the starting row masked in each column of the mask matrix; and the computing unit obtains the masked sparse representation from the memory unit, performs sparse processing on the input data using the masked sparse representation, and stores the sparse processing results in the memory unit. According to the solution of this disclosure, by constructing a masked sparse representation using constraints, memory consumption can be reduced from a squared scale of the sequence length to a linear scale of the sequence length, thereby significantly reducing memory requirements during large-scale model training and improving training efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and particularly to technical fields such as large-scale models, deep learning, and natural language processing.

Background Art

[0002] In the training of large-scale models based on transformers, the attention module is an important component of the Transformer block, and the computational complexity and memory usage of attention are proportional to the square of the sequence length. Flash Attention proposed in the industry aims to reduce the memory complexity of causal attention and make it closer to a linear relationship with the sequence length. However, when it is necessary to input a custom mask, the memory usage of the mask itself still correlates with the square of the sequence length, which limits the effect of reducing the memory consumption of Flash Attention. Therefore, especially when using a custom mask, how to further reduce the memory consumption while maintaining the performance of the attention mechanism is one of the important issues in current research and practice.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The present disclosure provides a sparse processing method, apparatus, and electronic device for a sparse attention network.

Means for Solving the Problems

[0004] According to a first aspect of the present disclosure, a sparse processing method for a sparse attention network is provided, which is applied to a computing device, the computing device includes a computing unit and a memory unit, and the method includes The method involves constructing a masked sparse representation in a memory unit using constraints, wherein the constraints are that the dimensions of the mask matrix are [B, a, S], where B represents the batch size, a represents the number of heads, S represents the sequence length, and each element in dimension S represents the starting row masked in each column of the mask matrix. The computing unit includes obtaining a masked sparse representation from the memory unit, performing sparse processing on the input data using the masked sparse representation, and storing the sparse processing result in the memory unit.

[0005] According to a second aspect of this disclosure, a method for fine-tuning a large-scale language model is provided, and the method, Obtaining a dataset for the target task, This includes fine-tuning and training a large-scale language model based on a dataset to adapt it to the requirements of a target task, Fine-tuning and training large-scale language models based on datasets is This includes combining sparse attention mechanisms to fine-tune and train large-scale language models based on datasets. The sparse attention mechanism employs a sparse processing method for the sparse attention network according to the first embodiment.

[0006] According to a third aspect of this disclosure, a method for processing data of a large-scale language model is provided, and the method, Obtaining input data, This includes inputting input data into a large-scale language model, which then processes the input data in conjunction with a sparse attention mechanism to generate output data. The sparse attention mechanism employs a sparse processing method for the sparse attention network according to the first embodiment.

[0007] According to a fourth aspect of this disclosure, a sparse processing device for a sparse attention network is provided, which is applied to a computing device, the computing device comprising a computing unit and a memory unit, and the sparse processing device for the sparse attention network is A construction module for constructing a masked sparse representation in a memory unit using constraints, wherein the constraints are that the dimensions of the mask matrix are [B, a, S], where B represents the batch size, a represents the number of heads, S represents the sequence length, and each element in dimension S represents the starting row masked in each column of the mask matrix; The system includes a sparse processing module for controlling the computing unit to obtain a masked sparse representation from the memory unit, perform sparse processing on the input data using the masked sparse representation, and store the sparse processing results in the memory unit.

[0008] According to a fifth aspect of this disclosure, a device for fine-tuning a large-scale language model is provided, and the device is The first acquisition module is for obtaining a dataset for the target task, To adapt a large-scale language model to the requirements of a target task, it includes a fine-tuning module for training a large-scale language model based on a dataset, Fine-tuning and training large-scale language models based on datasets is This includes combining sparse attention mechanisms to fine-tune and train large-scale language models based on datasets. The sparse attention mechanism employs a sparse processing method for the sparse attention network according to the first embodiment.

[0009] According to the sixth embodiment of this disclosure, a data processing device for a large-scale language model is provided, and the device is A second acquisition module for obtaining input data, The system includes a data processing module for inputting input data into a large-scale language model, which then processes the input data in conjunction with a sparse attention mechanism to generate output data. The sparse attention mechanism employs a sparse processing method for the sparse attention network according to the first embodiment.

[0010] According to a seventh aspect of this disclosure, an electronic device is provided, said device is At least one processor, The system comprises at least one processor and memory that is communicated with, The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, they cause the execution of a sparse processing method for a sparse attention network according to a first embodiment and / or a fine-tuning method for a large-scale language model according to a second embodiment and / or a data processing method for a large-scale language model according to a third embodiment. According to the eighth aspect of this disclosure, a non-temporary computer-readable storage medium is provided that stores computer instructions for causing a computer to execute a sparse processing method for a sparse attention network according to the first aspect and / or a fine-tuning method for a large-scale language model according to the second aspect and / or a data processing method for a large-scale language model according to the third aspect.

[0011] According to the ninth aspect of this disclosure, a program product is provided which, when executed by a processor, stores in a storage medium and implements a sparse processing method for a sparse attention network according to the first aspect and / or a fine-tuning method for a large-scale language model according to the second aspect and / or a data processing method for a large-scale language model according to the third aspect.

[0012] According to the method of this disclosure, by constructing a mask sparse representation using constraints, memory consumption can be reduced from a scale of the square of the sequence length to a linear scale of the sequence length, thereby significantly reducing memory requirements during large-scale model training and improving training efficiency.

[0013] It should be understood that the content described here does not intend to describe the key points or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. For other features of the present disclosure, understanding is promoted through the following specification.

Brief Description of the Drawings

[0014] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the plurality of accompanying drawings indicate the same or similar components or elements. These accompanying drawings are not necessarily drawn to scale. It should be understood that these drawings show only some embodiments provided by the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0015] [Figure 1] It is a schematic diagram of some attention mechanisms in the related art. [Figure 2] It is a schematic diagram of some extended attention mechanisms in the related art. [Figure 3] It is a flowchart of the sparse processing method of the sparse attention network according to the embodiment of the present disclosure. [Figure 4] It is a two-dimensional schematic diagram of the Block Attention Mask in the Causal scenario according to the embodiment of the present disclosure. [Figure 5] It is a two-dimensional schematic diagram of the Block Attention Mask in the scenario of the model fine-tuning stage of the large language model according to the embodiment of the present disclosure. [Figure 6] It is another two-dimensional schematic diagram of the Block Attention Mask in the scenario of the model fine-tuning stage of the large language model according to the embodiment of the present disclosure. [Figure 7] It is yet another two-dimensional schematic diagram of the Block Attention Mask in the scenario of the model fine-tuning stage of the large language model according to the embodiment of the present disclosure. [Figure 8]This is a schematic diagram illustrating the operation of constructing a mask sparse representation using constraints before calling the Block Attention mechanism, according to an embodiment of the present disclosure. [Figure 9] This is a schematic diagram illustrating the operation of constructing a mask sparse representation using constraints in a data reading block according to an embodiment of the present disclosure. [Figure 10] This is a flowchart illustrating a method for fine-tuning a large-scale language model according to the embodiments of this disclosure. [Figure 11] This is a flowchart of a data processing method for a large-scale language model according to an embodiment of the present disclosure. [Figure 12] This is a schematic diagram of the configuration of a sparse processing unit for a sparse attention network according to an embodiment of the present disclosure. [Figure 13] This is a schematic diagram of a fine-tuning device for a large-scale language model according to an embodiment of the present disclosure. [Figure 14] This is a schematic diagram of a data processing device for a large-scale language model according to an embodiment of the present disclosure. [Figure 15] This is a schematic diagram of a scenario for a sparse processing method for a sparse attention network according to an embodiment of the present disclosure. [Figure 16] This is a schematic diagram of a scenario for a method for fine-tuning a large-scale language model according to an embodiment of the present disclosure. [Figure 17] This is a schematic diagram of a scenario for a data processing method for a large-scale language model according to an embodiment of the present disclosure. [Figure 18] This is a schematic diagram of the configuration of an electronic device for realizing the sparse processing method for a sparse attention network and / or the fine-tuning method for a large-scale language model and / or the data processing method for a large-scale language model according to the embodiments of the present disclosure. [Modes for carrying out the invention]

[0016] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. Various details of the embodiments of the present disclosure are included herein for the sake of clarity and should be considered as illustrative only. Accordingly, those skilled in the art should be aware that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, in order to be clear and concise, descriptions of known functions and structures will be omitted in the following description.

[0017] The terms “First,” “Second,” and “Third,” etc., in the embodiments and claims of this disclosure and in the drawings above are for distinguishing similar subjects and are not necessary to describe a specific order or priority. Furthermore, the terms “includes” and “have,” and their variations, are intended to describe non-exclusive inclusion, e.g., inclusion of a set of things or units. Methods, systems, products, or devices are not limited to the things or units expressly enumerated and may include other things or units not expressly enumerated or specific to these processes, methods, products, or devices.

[0018] In related technologies, the calculation of Attention is given by the following equation (1).

number

[0019] Generally, the dimensions (shape) of Q, K, V, and Y are [B, S, H], and the dimensions of Mask are [B, a, S, S], where B is the batch size, S is the sequence length, H is the hidden size, and a is the number of heads.

[0020] However, in training large-scale models, the Mask is generally sparse. Conventional technologies typically include the following attention strategies: 1. Vanilla Attention Mechanism 2. Flash Attention Mechanism 3. Sparse Attention Mechanism 4. Random Attention Mechanism 5. Window Attention Mechanism 6. Global Attention Mechanism 7. Dilated Attention Mechanism 8. Combinatorial attention (BigBird Attention) mechanism.

[0021] Figure 1 shows schematic diagrams of several attention mechanisms of the related technology. As shown in Figure 1, (a) shows a schematic diagram of a random attention mechanism, (b) shows a schematic diagram of a window attention mechanism, (c) shows a schematic diagram of a global sparse attention mechanism, and (d) shows a schematic diagram of a combinational attention mechanism. The attention matrix A is used to indicate which pairs of tokens need to have attention calculated between them. If A(i,j)=1, it means that token i needs to have attention calculated between token j. In the case of a random attention mechanism, the value of A(i,j) may be generated based on some random process such that some locations are selected for attention calculation, while others are not. In the case of a window attention mechanism, the value of A(i,j) is usually 1 within a fixed local window, and 0 outside the window. In the case of a global attention mechanism, the value of A(i,j) is usually all 1 in the matrix, indicating that attention is calculated between all pairs of tokens. By combining local attention, global sparse attention, and possible randomness, combinatorial attention mechanisms can enable models to capture critical dependencies in sequences while maintaining computational efficiency.

[0022] Figure 2 shows schematic diagrams of several augmentation attention mechanisms of related technologies. These mechanisms broaden the receptive field while maintaining sparsity by introducing dilation, allowing for the capture of more distant dependencies without increasing computational burden. As shown in Figure 2, (a) has a sequence length of 4 and a dilation ratio of 1. (b) has a sequence length of 8 and a dilation ratio of 2. (c) has a sequence length of 16 and a dilation ratio of 4. Combining (a), (b), and (c) yields (d). Here, the dilation ratio is a parameter used to control the number and magnitude of dilations in the convolutional nucleus, which determines the size of the receptive field of the convolutional nucleus. A larger dilation ratio results in a larger receptive field, and vice versa. Therefore, by adjusting the dilation ratio, the perceived range and accuracy of the model for input data can be controlled to some extent.

[0023] As can be seen from Figures 1 and 2, both the combinational attention mechanism and the extended attention mechanism are examples of sparse attention, achieving sparsity through different mechanisms and thereby optimizing the computational efficiency and performance of the model. However, the above-mentioned sparse attention approach has the drawback that, by introducing a Mask, which is proportional to the square of the sequence length, into the attention calculation formula, the memory load increases dramatically when the sequence length is very large.

[0024] This disclosure proposes a sparse processing method for sparse attention networks that, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, can reduce memory consumption from a squared scale of the sequence length to a linear scale of the sequence length by constructing a masked sparse representation of the block attention mechanism using pre-set constraints, thereby significantly reducing memory requirements during large-scale model training and improving training efficiency. At the same time, further optimization in flash attention can improve the computational performance of large-scale models.

[0025] Embodiments of this disclosure provide a sparse processing method for a sparse attention network applicable to a computing device including a computing unit and a memory unit. Figure 3 is a flowchart of a sparse processing method for a sparse attention network applicable to a sparse attention network processing device according to an embodiment of this disclosure. This sparse attention network processing device resides in a computing device within an electronic device. The electronic device includes, but is not limited to, a fixed device and / or a mobile device. For example, a fixed device includes, but is not limited to, a server, which may be a cloud server or a regular server. For example, a mobile device includes, but is not limited to, a mobile phone, a tablet, etc. In some possible implementations, the sparse processing method for a sparse attention network can also be implemented by a processor calling computer-readable instructions stored in memory. A sparse attention network is a special attention mechanism that reduces computational complexity and memory consumption by selecting only a small portion of content relevant to the current location, and this mechanism has the potential for widespread application in processing long sequence data such as text and images. The core idea of ​​a sparse attention network is to improve processing efficiency and accuracy by focusing attention on the parts most relevant to the current task, rather than giving equal attention to all input information. As shown in Figure 3, the sparse processing method of this sparse attention network includes the following:

[0026] In S301, a mask sparse representation is constructed in the memory unit using constraints, the constraint being that the dimensions of the mask matrix are [B, a, S], where B represents the batch size, a represents the number of heads, S represents the sequence length, and each element in dimension S represents the starting row masked in each column of the mask matrix.

[0027] In S302, the computing unit obtains a masked sparse representation from the memory unit, performs sparse processing on the input data using the masked sparse representation, and stores the sparse processing result in the memory unit.

[0028] In embodiments of this disclosure, the masked starting row can be understood as the row from which the mask begins.

[0029] In the embodiments of this disclosure, masked sparse representation is used to describe a sparse representation of data realized by a mask in an attention mechanism. In an attention mechanism, a mask is a common tool for indicating parts that should be ignored or not considered in the computation process. Sparse representation, on the other hand, refers to a data representation method in which most elements are zero or close to zero, and only a few elements have significant non-zero values.

[0030] In embodiments of this disclosure, mask sparse representations in attention mechanisms are typically used to reduce computational complexity and memory consumption. Sparsification is achieved by predefining a mask that specifies locations to be ignored (i.e., set to zero) when calculating attention weights. Such sparseness not only improves computational efficiency but also helps the model focus on the most important information.

[0031] In some embodiments, applying sparse processing to the input data using a constructed masked sparse representation in the attention block involves applying a mask to the attention weight calculation process. This allows the model to ignore the masked portion when calculating the attention weights, thereby achieving attention sparsity. This sparse processing method can significantly improve the computational efficiency of the model, especially when processing long sequence data. It can also improve the model's performance because the model only focuses on the important parts.

[0032] In some embodiments, it is possible to determine whether each position should be masked based on the sequence length and the number of heads. Generally, this relates to a deep understanding of the sequence content and task requirements. For example, in natural language processing tasks, it may be necessary to determine which inter-word dependencies should be considered and which should be ignored based on the syntactic structure and semantic information of the sentence. Specifically, the granularity of the mask can be determined according to the sequence length. For relatively long sequences, a coarser-grained mask may be needed to reduce computational complexity. On the other hand, for short sequences, a finer-grained mask can be used to retain more information. At the same time, the number of heads can also influence the mask design. Different heads may focus on different aspects or features within the sequence, and therefore, different masking strategies may be required.

[0033] In practical applications, designing a masked sparse representation requires comprehensive consideration of multiple factors such as model performance, computational efficiency, and memory consumption, and typically requires some experimentation and tuning to find the optimal masking strategy.

[0034] In the embodiments of this disclosure, the dimensions of the mask matrix are [B, a, S], where in the sparse representation of [B, a, S], B is the batch size, i.e., the number of samples processed simultaneously; a is the number of attention heads, i.e., the number of attention heads computed in parallel in a multi-head attention mechanism; and S is the sequence length, where each element represents which row in the column and all subsequent rows are masked. We are only interested in the Mask in the lower left triangular portion, and since this Mask has a specific sparsity, it can be effectively represented using the dimensions of [B, a, S]. This representation method allows for more efficient use of memory when computing Block Attention and reduces unnecessary computations. By focusing only on the positions that actually participate in the attention computation, the efficiency of large-scale sequence processing tasks can be greatly improved. Here, when training a deep learning model, we can speed up training by processing one batch of samples simultaneously rather than just one sample at a time, and by utilizing the parallel computing capabilities of a graphics processing unit (GPU), B refers to the number of samples in this batch. In a multi-head attention mechanism, the model computes multiple attention weights in parallel, with each attention weight computed from a single "head." The number of heads is the number of these parallel-computed attentions. The multi-head attention mechanism helps the model capture information from different aspects of the input data, thereby improving the model's expressive power. In natural language processing tasks, input data is typically in sequence form, such as sentences or paragraphs. Sequence length refers to the number of elements (words, characters, etc.) contained in the sequence, and different input samples may have different sequence lengths.

[0035] Figure 4 shows a two-dimensional schematic diagram of a Block Attention Mask in a causal scenario. As shown in Figure 4, the Block Attention Mask mainly consists of two parts: the upper right triangular region, which is the first part, the causal mask, and the lower left triangular region, which is the newly added block mask. The Block Attention Mask is masked in the upper right triangular region by performing a sparsified representation in the lower left triangular region, and these two regions are combined to form a single complete matrix. In this matrix, all elements in the upper right triangular region (including the diagonal) are masked (i.e., set to values ​​that do not affect the calculation, such as 0), while the lower left triangular region holds values ​​that may participate in the calculation. Such a mask design can be used to implement certain forms of causal or autoregressive attention mechanisms, in which the model relies only on that location and the information before it when calculating the output at a given location, and cannot see future information. This is particularly useful when processing sequence data, for example in natural language processing tasks, where the model needs to predict the next word based on current and past words.

[0036] Specifically, in a Block Attention Mask, all masks in the upper right triangle region mean that no information about future positions is considered when calculating the attention weights for the current position. This helps the model maintain a causal understanding of the sequence and avoid "peeking" at future data. The lower left triangle region is a sparsified representation, allowing the model to consider the current position and previous information when calculating attention. This helps the model capture dependencies within the sequence and generate reasonable outputs based on them. By combining these two regions, the Block Attention Mask provides an attention mechanism that considers historical information while avoiding leakage of future information, thereby improving the efficiency and accuracy of the model's processing of sequence data.

[0037] In embodiments of this disclosure, a Block Attention Mask can be used to implement a causal mask that ensures future information is not visible when a model predicts its current position. A Block Attention Mask can also be used to implement local attention or windowed attention, where a model focuses only on specific local regions within a sequence.

[0038] As shown in Figure 4, the blank squares indicate that they are masked, and the Mask corresponding to one head can be represented as [18, 13, 20, 20, 11, 20, 10, 16, 12, 20, 20, 14, 15, 20, 17, 20, 20, 19, 20, 20]. Here, 18 indicates that column 0 is masked from row 18, 13 indicates that column 1 is masked from row 13, 20 indicates that column 2 is masked from at least row 20, 20 indicates that column 3 is masked from at least row 20, 11 indicates that column 4 is masked from row 11, 20 indicates that column 5 is masked from at least row 20, 10 indicates that column 6 is masked from row 10, 16 indicates that column 7 is masked from row 16, 12 indicates that column 8 is masked from row 12, 20 indicates that column 9 is masked from at least row 20, and 20 indicates that column 10 is The numbers indicate that the columns are masked from at least row 20. 14 indicates that the 11th column is masked from row 14, 15 indicates that the 12th column is masked from row 15, 20 indicates that the 13th column is masked from at least row 20, 17 indicates that the 14th column is masked from row 17, 20 indicates that the 15th column is masked from at least row 20, 20 indicates that the 16th column is masked from at least row 20, 19 indicates that the 17th column is masked from row 19, 20 indicates that the 18th column is masked from at least row 20, and 20 indicates that the 19th column is masked from at least row 20. Note that rows and columns are usually counted from 0.

[0039] Figure 5 shows a two-dimensional schematic diagram of a Block Attention Mask in a model fine-tuning scenario for a large-scale language model. As shown in Figure 5, blank squares represent masked positions, meaning that information about these positions is ignored or excluded when calculating attention. Non-blank squares represent unmasked positions, and the model considers information about these positions when calculating attention. Under the definition of Block Attention Mask, the corresponding Mask in this figure can be represented as [10, 10, 10, 5, 5, 8, 8, 8, 10, 10].

[0040] Figure 6 shows another two-dimensional schematic diagram of a Block Attention Mask in a scenario of the model fine-tuning phase of a large-scale language model, where, as shown in Figure 6, blank squares represent masked positions and non-blank squares represent unmasked positions. Under the definition of Block Attention Mask, the corresponding Mask in this diagram can be represented as [3, 3, 3, 0, 0, 0, 0, 0, 0, 0].

[0041] Figure 7 shows yet another two-dimensional schematic diagram of a Block Attention Mask in a scenario of the model fine-tuning phase of a large-scale language model, where, as shown in Figure 7, blank squares represent masked positions and non-blank squares represent unmasked positions. Under the definition of Block Attention Mask, the corresponding Mask in this figure can be represented as [2, 2, 0, 0, 0, 0, 0, 0, 0, 0].

[0042] In some embodiments, sparse processing of input data using a masked sparse representation involves achieving sparseness by ignoring masked locations when calculating attention scores or other related operations during the processing.

[0043] The sparse processing method of this solution can also be applied to causal scenarios and can be broadly applied to the following fields and scenarios.

[0044] 1. Causal Inference and Causal Modeling In causal inference and causal modeling, understanding the causal relationships between variables is crucial. By applying the sparse processing method of this approach, causal diagrams or causal models can be constructed more effectively. By focusing only on variables that directly affect specific outcomes, the model can be simplified, and the accuracy of inference can be improved.

[0045] 2. Generative tasks in natural language processing In natural language processing, generation tasks such as text summarization, machine translation, and dialogue systems typically require the generation of elements in a sequence in a specific order. By using the sparse processing method of this solution, it is possible to ensure that the model relies only on the preceding context information when generating each word or sentence, thus conforming to the natural order of language processing.

[0046] 3. Time-sequence analysis In time sequence analysis, data points typically have a clear chronological order, such as in stock price forecasting or weather pattern identification. The sparse processing method of this solution ensures that the model considers only past information when analyzing time sequence data, thereby avoiding the leakage of future information and improving the accuracy of predictions.

[0047] 4. Image and video processing While image and video data themselves do not have a clear order, in some tasks, such as video prediction or image generation, it may be desirable for the model to process the data in a certain order. In this case, the sparse processing method of this solution can be used to constrain the model's attention mechanism to focus on different parts of an image or video in a specific order.

[0048] 5. Recommendation System Recommendation systems typically need to predict content that users may be interested in, based on their behavioral history and preferences. By using the sparse processing method of this solution, we can ensure that the recommendation algorithm considers only the user's past actions when making predictions, thereby avoiding the use of future information and improving the accuracy and reliability of recommendations.

[0049] The above scenarios are merely examples and not limiting. Due to its flexibility and efficiency, the sparse processing method of this approach can be broadly applied to various scenarios requiring the processing of sequential data or involving order-dependent data. By matching specific task requirements with data characteristics, the model's performance and efficiency can be effectively improved.

[0050] In the embodiments of this disclosure, memory includes, but is not limited to, central processing unit (CPU) memory and graphics processing unit (GPU) display memory.

[0051] In practical applications, large-scale language models can be deployed to computing devices, and based on training samples, these deployed models are trained to obtain the target model.

[0052] Here, in the process of training a large-scale language model deployed on a computing device, the following steps are performed: construct a masked sparse representation using constraints, where the dimensions of the mask matrix are [B, a, S], B represents the batch size, a represents the number of heads, and S represents the sequence length, and each element in dimension S represents the starting row masked in each column of the mask matrix, and sparse processing is performed on the input data using the masked sparse representation.

[0053] In the embodiments of this disclosure, the computing device may be a server device or a terminal device.

[0054] In embodiments of this disclosure, the target model may be a natural language processing model for performing at least one natural language processing operation, including information extraction, text classification, text identification, speech identification, and question answering, on a first class of data to be processed, and obtaining a natural language processing result. The target model may also be a computer vision processing model for performing at least one computer vision processing operation, including image identification, target detection, semantic segmentation, video understanding, and image generation, on a second class of data to be processed, and obtaining a computer vision processing result. The above is merely an illustrative description and does not limit all possible types of target models, nor is it listed here.

[0055] The technical solution of the embodiment of this disclosure effectively sparsely represents the Mask in the lower left triangular region by constructing a mask sparse representation using constraints, thereby significantly reducing computational complexity and memory consumption, greatly improving the efficiency of sequence data processing, and reducing memory requirements. By constructing a mask sparse representation using pre-set constraints, memory consumption can be reduced from a squared scale of the sequence length to a linear scale of the sequence length, thereby significantly reducing memory demand during large-scale model training and improving training efficiency. Such sparse representation methods have the potential for widespread application in fields such as causal inference and natural language processing, and are expected to provide better performance and efficiency for large-scale model training and application.

[0056] In embodiments of this disclosure, performing sparse processing on input data using a masked sparse representation includes inputting the query matrix, key matrix, value matrix, and mask matrix obtained based on the masked sparse representation of the input data into an attention calculation formula, wherein the attention calculation formula is:

number

[0057] In some embodiments, the attention calculation formula is:

number

[0058] Q (Query) represents the query matrix. In the attention mechanism, Q typically represents the information or location currently under attention. Its dimension may depend on the specific model or task, but it usually corresponds to the sequence length for which attention is being calculated.

[0059] K (Key) represents the key matrix. Similar to Q, K is a matrix that represents certain information or positions. In attention calculations, K is usually used to determine which parts should receive more attention compared to Q. The dimension of K is usually the same as that of Q.

[0060] V (Value) represents a value matrix. V contains the actual information or representation associated with each position. When attention is weighted, these values ​​are used to generate the final output. The dimensions of V are usually the same as K and Q (at least in some dimensions), but the specific content may differ from that of K and Q.

[0061] T represents the transpose operation. We show that we can perform a transpose on matrix K such that the rows of KT correspond to the columns of Q, thereby enabling matrix multiplication QKT.

[0062] QKT represents a matrix multiplication operation. This operation calculates the point product between the transposes of Q and K and is used to generate a score matrix in which each element represents the similarity or consistency between one element in Q and all elements in K.

[0063] A mask represents a mask matrix. A mask is a matrix with the same shape as the QKT score matrix and is used to indicate which positions should be ignored or given a relatively low weight. In sequence processing, masks are typically used to handle sequences of different lengths (such as padding portions) or to ignore specific locations (such as padding for some tasks).

[0064] `softmax()` represents the softmax function. This function converts the result of QKT+Mask into a probability distribution, ensuring that all values ​​are within the range [0, 1] and that the sum of all values ​​is 1. This assigns weights to each position, indicating its importance relative to the others.

[0065] Y represents the output matrix. Y is the attention-weighted value matrix, calculated by applying the result of the softmax function to V. Specifically, the values ​​in V at each position are scaled based on the corresponding softmax weights to generate Y.

[0066] In summary, this attention calculation formula describes a method for calculating a weighted sum based on the similarity between Q and K and mask information, generating a Y that fuses the important information. This attention mechanism is widely used in various deep learning models, particularly when processing sequence data.

[0067] The technical solution of the embodiments of this disclosure inputs a query matrix, key matrix, value matrix, and a mask matrix obtained based on a mask sparse representation of the input data into the attention calculation formula. In this way, a mask-based sparse representation can reduce the number of elements involved in the attention calculation, and especially when processing long sequences or large datasets, unnecessary calculations can be greatly reduced by focusing only on important elements, thereby improving overall computational efficiency.

[0068] In embodiments of this disclosure, constructing a mask sparse representation using constraints includes performing an operation to construct a mask sparse representation using constraints based on Flash Attention. Specifically, the operation to construct a mask sparse representation using constraints is performed in a Flash Attention block.

[0069] In some embodiments, Flash Attention is an attention mechanism that can simulate the human visual attention mechanism in a neural network. The attention mechanism focuses on different parts of the input by assigning different weights, generates a new representation by calculating a weighted sum of the input data, and becomes more efficient when processing large amounts of data.

[0070] In some embodiments, performing an operation to construct a masked sparse representation using constraints based on Flash Attention generates a mask matrix used to realize a sparse representation of data based on predefined constraints when processing data using Flash Attention. Specific steps include:

[0071] 1. Pre-set constraints. These constraints are typically set based on the characteristics and requirements of the task. For example, in a causal inference task, constraints might require the model not to see future words when predicting the next word. In a natural language processing sequence generation task, constraints might limit the model to focusing only on the input portion relevant to the current output.

[0072] 2. Construct the mask matrix. A mask matrix is ​​generated based on these predefined constraints. The size of the mask matrix is ​​usually consistent with the dimensions of the input data, and its element values ​​are used to indicate which parts of the data should be given attention (usually set to 1), which parts should be ignored, or which parts should have their weight reduced (usually set to 0 or a relatively small value).

[0073] 3. Apply the mask matrix. The structured mask matrix is ​​joined with the input data query matrix, key matrix, and value matrix, and computed through the Flash Attention block. During the computation process, the mask matrix acts as a filter and weight, allowing the model to focus on important information and ignore unimportant parts.

[0074] 4. Achieve sparse representation. By applying a mask matrix, the Flash Attention block can output a sparse representation result. This sparse representation result not only reduces the dimensionality and complexity of the data, but also improves the efficiency and accuracy of the model when processing the data.

[0075] The technical solutions of the embodiments of this disclosure perform an operation to construct a sparse representation of a mask using constraints based on a Flash Attention block. In this way, the sparse representation of the mask matrix allows the model to focus on the important parts of the input data and skip over and not compute non-important information, thereby reducing the waste of computational resources and significantly improving computational efficiency, especially when processing large datasets.

[0076] In embodiments of the present disclosure, performing an operation to construct a mask sparse representation using constraints in a Flash Attention block includes performing an operation to construct a mask sparse representation using constraints before the Flash Attention block invokes the Block Attention mechanism.

[0077] Here, "before calling the Block Attention mechanism" specifically refers to the timing when the mask sparse representation is constructed.

[0078] Figure 8 shows a schematic diagram for performing the operation of constructing a masked sparse representation using constraints before calling the Block Attention mechanism. As shown in Figure 8, each Transformer block contains an attention block (i.e., Attention Block) and a feedforward neural network (FFN) module, and each Attention module contains two operations: constructing a masked sparse representation and calling Block Attention. Here, the program or component that performs the operation of constructing a masked sparse representation can be called the construct sparse mask module, and the program or component that performs the operation of calling Block Attention can be called the call Block Attention module. Specifically, Q, K, V, and Block Attention Mask are input to the call Block Attention module, and the masked region can be skipped without computation in the actual computation process. For example, the masked block can be skipped without computation in a specifically implemented CUDA kernel (Compute Unified Device Architecture Kernel). By constructing a Mask before the Block Attention module, the Mask can be dynamically generated based on the current state of the model and the characteristics of the input data, allowing for flexibility. After the data has undergone a series of transformations and processing, some features or data points may become irrelevant. By constructing a Mask before Block Attention, unnecessary computations can be further reduced. In some embodiments, a mask sparse representation can be constructed in a memory unit. The computing unit retrieves the mask sparse representation from the memory unit, performs sparse processing on the input data using the mask sparse representation, and stores the sparse processing results in the memory unit.

[0079] The technical solution in the embodiments of this disclosure performs an operation to construct a mask sparse representation using constraints before calling the Block Attention mechanism. By removing irrelevant information in advance in this way, the computational complexity of the subsequent Block Attention mechanism is reduced, making the entire processing process more efficient.

[0080] In embodiments of the present disclosure, performing an operation in a Flash Attention block to construct a mask sparse representation using constraints includes, in response to the Flash Attention block detecting a first construct instruction, constructing the same mask sparse representation in each Transformer block in the model based on the first construct instruction.

[0081] In some embodiments, the first construct instruction is used to generate the same mask sparse representation for all Transformer blocks (i.e., Transformer Blocks) in the model before calling the Block Attention mechanism.

[0082] In some embodiments, within each Transformer block, the system uses a predefined rule or algorithm to generate a mask sparse representation based on a first construction instruction, and applies the same mask sparse representation to each Transformer block in the model to ensure that they all use the same mask sparse representation.

[0083] The technical solution of the embodiments of this disclosure constructs the same mask sparse representation for each Transformer block in the model based on the first construct instruction in response to the detection of a first construct instruction. By constructing the same mask sparse representation for each Transformer block in the model in this way, it is ensured that the model applies the same sparse pattern when processing data in different layers, which helps the model maintain consistent actions across different layers and ensures consistency in how the model processes data. Because all Transformer blocks use the same masking rules, the model can maintain a unified perspective and logic when extracting features and making decisions, which enhances the stability, interpretability, and reliability of the model, reduces the complexity of the model, and facilitates understanding and optimization of the model.

[0084] In embodiments of the present disclosure, performing an operation in a Flash Attention block to construct a mask sparse representation using constraints includes, in response to the Flash Attention block detecting a second construct instruction, constructing a different mask sparse representation for each Transformer block in the model based on the second construct instruction.

[0085] In some embodiments, this second construct instruction is used to provide each Transformer block with a specific masking strategy before calling the Block Attention mechanism.

[0086] In some embodiments, within each Transformer block, the system generates its own mask using a predefined rule or algorithm, ensuring that the sparse representation of the masks differs between them. This enables flexible processing of information according to different characteristics of the input data and task requirements.

[0087] The technical solutions of the embodiments of this disclosure construct different mask sparse representations for each Transformer block in the model based on the second construct instruction, and these different mask sparse representations allow each Transformer block to focus on different parts of the input, thereby enhancing the flexibility and expressiveness of the model. By constructing different mask sparse representations for each Transformer block in the model, the mask strategy can be customized based on the requirements and characteristics of different modules, which helps to better adapt to the requirements of different data and tasks, and enhances the adaptability and generalization ability of the model.

[0088] In embodiments of this disclosure, constructing a mask sparse representation of a Block Attention mechanism using constraints includes performing an operation in a data reader module to construct a mask sparse representation using constraints.

[0089] In some embodiments, this data reading block is a dedicated module for reading data from a data source. Its primary role is to introduce data stored in an external data source (e.g., a database, file, or application interface (API)) into the system for further processing, analysis, or computation.

[0090] Figure 9 shows a schematic diagram of the operation to construct a masked sparse representation using constraints in the data reading block. As shown in Figure 9, each Transformer block contains one Attention module and one FFN module, and each Attention module contains a calling Block Attention module. The data reading block performs the operation to construct a masked sparse representation. The data reading block performs the operation to construct a masked sparse representation using constraints, which typically occurs before model input. The main role of the data reading block is to load and process the original data and provide appropriate input to the model. In this process, constructing a masked sparse representation based on pre-defined constraints is part of the data preprocessing to introduce sparsity into the model training or inference process. Specifically, when the data reading block reads the original data, it constructs a masked sparse representation for each data point (or data batch) based on pre-defined constraints (e.g., specific rules, thresholds, conditional expressions). This mask is applied to parts of the data, which are ignored or have reduced weight in subsequent calculations. In this way, the data reading block can ensure that the data is sparsified according to specific requirements before the model begins processing the data. The output of the data reading block (i.e., the data processed with a masked sparse representation) enters the model training or inference flow as input to the model. This design allows sparsity to be introduced in the early stages of model processing, effectively impacting the model's performance and computational efficiency. Placing the Mask structure as an independent step in the data preprocessing stage makes the entire data processing process more modular, easier to understand, and maintain. Sparsity processing and selective attention to the input data can be achieved by inputting the masked sparse representation along with the input data into a Transformer block and calling Block Attention based on the masked sparse representation within the module.Specifically, Q, K, V, and Block Attention Mask are called and input to the Block Attention module, and the masked region can be skipped without computation in the actual computation process. For example, a masked block can be skipped without computation in a specifically implemented CUDA kernel. In some embodiments, the masked sparse representation can be read from disk to a memory unit, the computing unit retrieves the masked sparse representation from the memory unit, performs sparse processing on the input data using the masked sparse representation, and stores the sparse processing result in the memory unit.

[0091] The technical solutions of the embodiments of this disclosure introduce sparsity in the early stages of model processing by performing an operation in the data reading block to construct a mask sparse representation using constraints, thereby effectively impacting the model's performance and computational efficiency. The data reading block can perform accurate screening and filtering when reading data, meaning that only data that matches specific conditions or requirements is read and processed, thus helping to reduce the complexity and computational cost of subsequent data processing and analysis. Furthermore, by placing the Mask structure as an independent step in the data preprocessing stage, the entire data processing process can be made more modular, easier to understand, and easier to maintain.

[0092] Embodiments of the present disclosure include performing an operation in a data read block to construct a mask sparse representation using constraints, and in response to the data read block detecting a third construct instruction, constructing the same mask sparse representation in each Transformer block in the model based on the third construct instruction.

[0093] In some embodiments, this third construct instruction is used in the data read block to generate the same mask sparse representation for all Transformer blocks in the model.

[0094] In some embodiments, in the data reading block, the system uses a predefined rule or algorithm to generate a single mask sparse representation based on a third construction instruction, and applies the same mask sparse representation to each Transformer block in the model to ensure they all use the same mask sparse representation. Because all modules share the same mask sparse representation, implementations can leverage this consistency to optimize computations, for example, by computing outputs corresponding to non-zero weights only when necessary. A consistent mask pattern makes it easier to identify which input features or model components have a significant impact on the output, which can help understand how the model makes decisions, thereby enhancing the interpretability of decision-making.

[0095] In actual implementation, the specific details and effects depend on the concrete content of the third build instruction and how it is defined and applied. This usually requires a deep understanding of the model's architecture, data, and training objectives.

[0096] The technical solution of the embodiments of this disclosure constructs the same mask sparse representation for each Transformer block in the model based on a third construct instruction in the data reading block. By constructing the same mask sparse representation for each Transformer block in the model in this way, consistency of the model can be ensured when processing data. Because all modules use the same masking rules, the model can maintain a unified perspective and logic when extracting features and making decisions, which helps to enhance the stability, interpretability, and reliability of the model.

[0097] In embodiments of the present disclosure, performing an operation in a data read block to construct a mask sparse representation using constraints includes, in response to the data read block detecting a fourth construct instruction, constructing a different mask sparse representation for each Transformer block in the model based on the fourth construct instruction.

[0098] In some embodiments, this fourth construct instruction is used to provide a specific masking strategy to each Transformer block in the data read block.

[0099] In some embodiments, the data reading block uses a predefined rule or algorithm to generate multiple different mask sparse representations based on the fourth construct instruction, ensuring that the mask sparse representations are different among them by assigning one unique mask to each Transformer block. This enables flexible information processing in response to different characteristics of the input data and task requirements.

[0100] The technical solutions of the embodiments of this disclosure construct a masked sparse representation through constraints, allowing data reading blocks to perform accurate screening and filtering when reading data. That is, only data matching specific conditions or requirements is read and processed, thus reducing the complexity and computational cost of subsequent data processing and analysis. In this way, each Transformer block in the model has a different masked sparse representation, allowing it to focus on different features and modes when processing input data. Because the model can capture the complexity of the input data in multiple ways, this design can improve the model's performance. At the same time, it may increase the complexity and training difficulty of the model, so the masking strategy needs to be carefully adjusted and optimized to ensure optimal performance.

[0101] Furthermore, the schematic diagrams shown in Figures 1, 2, 4, 5, 6, 7, 8, and 9 are illustrative and expandable, not limiting, and those skilled in the art can make various obvious changes and / or substitutions based on the examples in Figures 1, 2, 4, 5, 6, 7, 8, and 9, and the resulting technical solutions still fall within the scope of the embodiments disclosed herein.

[0102] Embodiments of this disclosure provide a method for fine-tuning a large-scale language model, and Figure 10 is a flowchart of the method for fine-tuning a large-scale language model according to an embodiment of this disclosure, and this method for fine-tuning a large-scale language model can be applied to a device for fine-tuning a large-scale language model. This device for fine-tuning a large-scale language model is located on an electronic device. This electronic device includes, but is not limited to, a fixed device and / or a mobile device. For example, a fixed device includes, but is not limited to, a server, which may be a cloud server or a regular server. For example, a mobile device includes, but is not limited to, a mobile phone, a tablet, etc. In some possible implementations, this method for fine-tuning a large-scale language model can also be implemented by a processor calling computer-readable instructions stored in memory. As shown in Figure 10, this method for fine-tuning a large-scale language model includes the following:

[0103] In S1001, obtain the dataset for the target task.

[0104] In S1002, the large-scale language model is fine-tuned and trained based on the dataset to adapt it to the requirements of the target task.

[0105] Here, fine-tuning and training a large-scale language model based on a dataset includes fine-tuning and training a large-scale language model based on a dataset in conjunction with a sparse attention mechanism, where the sparse attention mechanism uses the sparse processing method of the sparse attention network described above, and the sparse attention mechanism here is a Block Attention Mechanism.

[0106] Obtaining a dataset for the target task is a crucial step in any machine learning project. The quality and diversity of the dataset are important for training an efficient and highly generalizable model. For example, data can be obtained from publicly available datasets. Alternatively, data can be obtained from websites, social media, and APIs via web crawlers. Another method is to collect and mark up the data yourself. The above data acquisition methods are merely illustrative and not restrictive, and this disclosure does not limit the methods of dataset acquisition.

[0107] When fine-tuning and training large-scale language models, masking the Attention modules within each Transformer block using a sparse processing method is an effective strategy that can significantly improve the model's efficiency and performance. The detailed processing steps are shown below.

[0108] First, it's necessary to understand the basic structure of large-scale language models. Typically, multiple Transformer blocks are stacked together. Each Transformer block contains an Attention module, which allows the model to capture important parts of the dependencies between different positions in the input sequence.

[0109] During the fine-tuning training phase, the Attention module is masked using a sparse processing method. Sparse processing generally means focusing on only a portion of the important information in the input data, rather than all of it. The purpose of this is to reduce the computational burden on the model during training while maintaining or improving the model's performance.

[0110] There are several ways to implement sparse mask processing, but one common method is to use a local attention or block attention mechanism. Local attention means that each position focuses only on a subset of positions near the entire sequence, rather than the entire sequence. Block attention divides the input sequence into multiple blocks and performs attention calculations between positions within each block.

[0111] In the fine-tuning process, a sparse mask matrix can be introduced for each Attention module in each Transformer block. The sparse mask matrix is ​​the same size as the Attention matrix, but most of its elements are 0, with only a few important elements being 1. Sparsification of the Attention matrix can be achieved by multiplying the Attention matrix and the sparse mask matrix.

[0112] This method effectively reduces the computational load during model training while maintaining focus on the model's critical information. This not only accelerates the training process but also reduces the model's demands on computational resources, facilitating deployment in resource-constrained environments.

[0113] The specific implementation method for sparse mask processing may vary depending on the model structure and task requirements. Therefore, in actual application, it is possible to select an appropriate sparse mask processing method according to the specific situation and conduct thorough experimental validation so that the fine-tuned model achieves the expected performance improvement.

[0114] During the fine-tuning process, model parameters such as the learning rate, batch size, and normalized items can be adjusted to optimize the model's performance on specific tasks. Simultaneously, the presence of a sparse attention mechanism means the model only needs to focus on critical parts of the input sequence, allowing the fine-tuning process to utilize computational resources more efficiently.

[0115] Combining sparse attention mechanisms with fine-tuning and training large-scale language models based on datasets has at least the following effects:

[0116] 1. Improved computational efficiency. The sparse attention mechanism allows the model to focus on important parts when calculating attention weights, ignoring irrelevant information. This reduces computational load, accelerates the training process, and lowers the demands on computational resources.

[0117] 2. Reduced memory consumption. Sparse representations mean that models only need to store non-zero elements when recording attention weights, significantly reducing memory usage. This is especially important for processing large models and long sequences, as it makes training more feasible under limited hardware resources.

[0118] 3. Optimizing model performance. By combining sparse attention mechanisms and fine-tuning training, the model can better adapt to the requirements of the target task. Sparse attention helps the model capture important information, and fine-tuning training allows the model to leverage features of a specific dataset to improve its performance on the target task.

[0119] 4. Improved generalization ability. Through fine-tuning training, the model can learn knowledge about specific tasks, thereby improving its generalization ability on target tasks. By adapting the model's parameters to new datasets, the model can better handle unfamiliar situations similar to the distribution of the training data.

[0120] 5. Increased interpretability of the model. The sparse attention mechanism makes the model more transparent and interpretable during decision-making. Because the model focuses primarily on a few key elements, it is easier to understand how the model makes decisions, which is crucial for reliability and trustworthiness in real-world applications.

[0121] The technical solutions of the embodiments of this disclosure, when combined with a sparse attention mechanism, can not only perform fine-tuning training on large language models to improve computational efficiency and reduce memory consumption, but also optimize model performance, enhance generalization ability, and strengthen model interpretability, providing a significant advantage in the task of large language models in practical applications.

[0122] Embodiments of this disclosure provide a data processing method for large-scale language models, and Figure 11 is a flowchart of the data processing method for large-scale language models according to an embodiment of this disclosure, and this data processing method for large-scale language models can be applied to a data processing device for large-scale language models. This data processing device for large-scale language models is located on an electronic device. This electronic device includes, but is not limited to, a fixed device and / or a mobile device. For example, a fixed device includes, but is not limited to, a server, which may be a cloud server or a regular server. For example, a mobile device includes, but is not limited to, a mobile phone, a tablet, etc. In some possible implementations, this data processing method for large-scale language models can also be implemented by a processor calling computer-readable instructions stored in memory. As shown in Figure 11, this data processing method for large-scale language models includes the following:

[0123] In S1101, input data is acquired.

[0124] In S1102, this input data is input to a large-scale language model, and the large-scale language model processes this input data in conjunction with a sparse attention mechanism to generate output data, where the sparse attention mechanism uses the sparse processing method of the sparse attention network described above, and the sparse attention mechanism here is a Block Attention Mechanism.

[0125] In the embodiments of this disclosure, the input data may be text data in various forms depending on the specific adaptation scenario and task requirements. The input data may include, but is not limited to, natural language text, problem query phrases, conversation history, code snippets or related natural language descriptions, queries for specific knowledge or facts, and structured data.

[0126] In the embodiments of this disclosure, the output data is data obtained by a large-scale language model based on the input data.

[0127] In the embodiments of this disclosure, inputting input data into a large-scale language model and processing it in conjunction with a sparse attention mechanism to generate output data has beneficial effects in several main areas, including the following:

[0128] 1. Improved processing efficiency. The sparse attention mechanism reduces computational complexity by allowing the model to ignore unimportant parts when focusing on important information through sparse processing. Such an efficient attention distribution method can significantly improve the speed at which large language models process input data, and the effect is even more pronounced when processing long sequences or large datasets.

[0129] 2. Reduced resource consumption. Because the sparse attention mechanism focuses only on essential information, the memory and computational resources required for the model are significantly reduced during the memory and computation processes. This is a significant optimization for hardware environments running large-scale language models, enabling the processing of larger data or more complex models under limited resource conditions.

[0130] 3. Improved model performance. Large-scale language models combined with sparse attention mechanisms can more accurately capture important information in input data, thereby generating more accurate and valuable output data. This performance improvement is reflected in various natural language processing tasks, such as text classification, sentiment analysis, and question-and-answer systems.

[0131] 4. Enhanced model generalization ability. The sparse attention mechanism allows the model to focus better on core features in the input data, thereby enabling the model to be more adaptable when processing text in different domains or styles. As a result, the model's generalization ability improves, allowing it to perform a role in a wider range of tasks and scenarios.

[0132] 5. Improved Interpretability. The sparse attention mechanism makes the model more transparent and interpretable in the decision-making process. By looking at the model's attention distribution, we can understand how the model allocates attention when processing input data, which is of great significance in understanding and optimizing the model's behavior.

[0133] The technical solutions of the embodiments of this disclosure enable large-scale language models to process input data in conjunction with sparse attention mechanisms, thereby increasing processing efficiency and reducing resource consumption. This also plays a crucial role in improving model performance, enhancing model generalization capabilities, and improving interpretability. This coupling method has broad potential and significant value in practical applications, particularly in scenarios requiring the processing of large amounts of text data or complex natural language processing tasks.

[0134] Embodiments of the present disclosure provide a sparse processing device for a sparse attention network, which is applied to a computing device comprising a computing unit and a memory unit. As shown in Figure 12, the sparse processing device for a sparse attention network comprises a construction module 1201 for constructing a masked sparse representation in the memory unit using constraints, wherein the dimensions of the mask matrix are [B, a, S], where B represents the batch size, a represents the number of heads, S represents the sequence length, and each element in the S dimension represents a starting row masked in each column of the mask matrix; and a sparse processing module 1202 for controlling the computing unit to obtain the masked sparse representation from the memory unit, perform sparse processing on input data using the masked sparse representation, and store the sparse processing results in the memory unit.

[0135] In some embodiments, the sparse processing module 1202 is used to obtain an output matrix of the input data by inputting the query matrix, key matrix, value matrix, and mask matrix obtained based on the mask sparse representation of the input data into an attention calculation formula.

[0136] Here, the attention calculation formula is:

number

[0137] In some embodiments, the construction module 1201 includes a first construction submodule for performing an operation in the flash attention block to construct a mask sparse representation using constraints.

[0138] In some embodiments, the first build submodule is used to build the same mask sparse representation for each Transformer block in the model based on the first build instruction, in response to the flash attention block detecting the first build instruction.

[0139] In some embodiments, the first build submodule is used to construct different mask sparse representations for each Transformer block in the model based on the second build instruction, in response to the flash attention block detecting the second build instruction.

[0140] In some embodiments, the construction module 1201 includes a second construction submodule for performing an operation in the data reading block to construct a mask sparse representation using constraints.

[0141] In some embodiments, the second build submodule is used to build the same mask sparse representation for each Transformer block in the model based on the third build instruction, in response to the data read block detecting the third build instruction.

[0142] In some embodiments, the second build submodule is used to construct different mask sparse representations for each Transformer block in the model based on the fourth build instruction, in response to the data read block detecting the fourth build instruction.

[0143] Those skilled in the art will understand that the functions of each processing module in the sparse processing device for a sparse attention network according to the embodiments of this disclosure can be understood by referring to the relevant descriptions of the sparse processing methods for sparse attention networks described above, and that each processing module in the sparse processing device for a sparse attention network according to the embodiments of this disclosure can be realized by a generation circuit that implements the function according to the embodiments of this disclosure, and can be realized by the operation of software on an electronic device that performs the function according to the embodiments of this disclosure.

[0144] The sparse processing device for the sparse attention network provided in the embodiments of this disclosure allows for the construction of a masked sparse representation of the Block Attention mechanism using constraints, thereby reducing memory consumption from a squared scale of the sequence length to a linear scale of the sequence length. This significantly reduces memory requirements during training of large-scale models and improves training efficiency. Furthermore, further optimization in Flash Attention can improve the computational performance of large-scale models.

[0145] Figure 13 is a schematic diagram of a fine-tuning device for a large-scale language model according to an embodiment of the present disclosure. As shown in Figure 13, the fine-tuning device for a large-scale language model comprises a first acquisition module 1301 for acquiring a dataset for a target task, and a fine-tuning module 1302 for fine-tuning and training the large-scale language model based on the dataset in order to adapt the large-scale language model to the requirements of the target task.

[0146] Here, the fine-tuning module 1302 includes fine-tuning and training a large language model based on a dataset in conjunction with a sparse attention mechanism, the sparse attention mechanism using the sparse processing method of the sparse attention network described above, and the sparse attention mechanism here is a Block Attention Mechanism.

[0147] Those skilled in the art will understand that the functions of each processing module in the large-scale language model fine-tuning device according to the embodiments of this disclosure can be understood by referring to the relevant descriptions of the large-scale language model fine-tuning methods described above, and that each processing module in the large-scale language model fine-tuning device according to the embodiments of this disclosure can be realized by a generation circuit that implements the function according to the embodiments of this disclosure, and can be realized by the operation of software on an electronic device that performs the function according to the embodiments of this disclosure.

[0148] The fine-tuning device for large-scale language models provided in the embodiments of this disclosure can be coupled with a sparse attention mechanism to perform fine-tuning training on large-scale language models, improving computational efficiency and reducing memory consumption. It can also optimize model performance, enhance generalization ability, and strengthen model interpretability, providing a significant advantage in the task of large-scale language models in practical applications.

[0149] Figure 14 is a schematic diagram of a data processing device for a large-scale language model according to an embodiment of the present disclosure. As shown in Figure 14, the data processing device for the large-scale language model comprises a second acquisition module 1401 for acquiring input data, and a data processing module 1402 for inputting the input data into a large-scale language model, which in turn processes the input data in conjunction with a sparse attention mechanism to generate output data. The sparse attention mechanism uses the sparse processing method of the sparse attention network described above, and the sparse attention mechanism here is a Block Attention Mechanism.

[0150] Those skilled in the art will understand that the functions of each processing module in the data processing device for large-scale language models according to the embodiments of this disclosure can be understood by referring to the relevant descriptions of the data processing methods for large-scale language models described above, and that each processing module in the data processing device for large-scale language models according to the embodiments of this disclosure can be realized by a generation circuit that implements the function according to the embodiments of this disclosure, and can be realized by the operation of software on an electronic device that performs the function according to the embodiments of this disclosure.

[0151] The data processing apparatus for large-scale language models provided in the embodiments of this disclosure allows for the significant improvement of processing efficiency and effective reduction of resource consumption by combining the large-scale language model with a sparse attention mechanism to process input data. Therefore, this combined approach is promising and of significant value in practical applications, particularly in scenarios where large amounts of text data need to be processed or where complex natural language processing tasks need to be performed.

[0152] Figure 15 is a schematic diagram of a scenario for a sparse processing method for a sparse attention network according to an embodiment of the present disclosure.

[0153] As described above, the sparse processing method for sparse attention networks provided in the embodiments of this disclosure is applicable to electronic devices. Electronic devices refer to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, large computers, and other suitable computers. Electronic devices further refer to various forms of mobile devices, such as personal digital assistants, cellular phones, intelligent phones, wearable devices, and other similar computer devices.

[0154] Specifically, electronic devices perform the following operations, namely, In the memory unit, a masked sparse representation is constructed using constraints, where the dimensions of the mask matrix are [B, a, S], where B represents the batch size, a represents the number of heads, and S represents the sequence length, and each element in dimension S represents the starting row masked in each column of the mask matrix. The computing unit obtains a masked sparse representation from the memory unit, performs sparse processing on the input data using the masked sparse representation, and stores the sparse processing result in the memory unit.

[0155] Here, input data can be obtained from a data source. The data source may be various data storage devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The data source further refers to each type of mobile device, such as personal digital assistants, cellular phones, intelligent phones, wearable devices, and other similar computer devices. The data source may also be the same device as the user terminal.

[0156] Here, the constraints may be set by the large-scale model training programmer or the large-scale model training team.

[0157] The scenario diagram shown in Figure 15 is merely illustrative and not restrictive, and those skilled in the art can make various obvious changes and / or substitutions based on the example in Figure 15, and the resulting technical solutions still fall within the scope of the embodiments of this disclosure.

[0158] Figure 16 is a schematic diagram of a scenario for a method for fine-tuning a large-scale language model according to an embodiment of the present disclosure.

[0159] As described above, the methods for fine-tuning large-scale language models provided in the embodiments of this disclosure are applicable to electronic devices. Electronic devices refer to each type of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, large computers, and other suitable computers. Electronic devices further refer to each type of mobile device, such as personal digital assistants, cellular phones, intelligent phones, wearable devices, and other similar computer devices.

[0160] Specifically, electronic devices perform the following operations, namely, Obtaining a dataset for the target task, In conjunction with a sparse attention mechanism, this involves fine-tuning and training a large-scale language model based on a dataset to adapt it to the requirements of a target task. Here, the sparse attention mechanism uses the sparse processing method of the sparse attention network described above.

[0161] Here, the dataset can be obtained from a data source. The data source may be various data storage devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The data source further refers to each type of mobile device, such as personal digital assistants, cellular phones, intelligent phones, wearable devices, and other similar computer devices. The data source may also be the same device as the user terminal.

[0162] The scenario diagram shown in Figure 16 is merely illustrative and not restrictive, and those skilled in the art can make various obvious changes and / or substitutions based on the example in Figure 16, and the resulting solutions still fall within the scope of the embodiments of this disclosure.

[0163] Figure 17 is a schematic diagram of a scenario for a data processing method for a large-scale language model according to an embodiment of the present disclosure.

[0164] As described above, the methods for fine-tuning large-scale language models provided in the embodiments of this disclosure are applicable to electronic devices. Electronic devices refer to each type of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, large computers, and other suitable computers. Electronic devices further refer to each type of mobile device, such as personal digital assistants, cellular phones, intelligent phones, wearable devices, and other similar computer devices.

[0165] Specifically, electronic devices perform the following operations, namely, Obtaining input data, The process involves inputting input data into a large-scale language model, which then processes the input data in conjunction with a sparse attention mechanism to generate output data. Here, the sparse attention mechanism uses the sparse processing method of the sparse attention network described above. Here, input data can be obtained from a data source. The data source may be various data storage devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The data source further refers to each type of mobile device, such as personal digital assistants, cellular phones, intelligent phones, wearable devices, and other similar computer devices. The data source may also be the same device as the user terminal.

[0166] The scenario diagram shown in Figure 17 is merely illustrative and not restrictive, and those skilled in the art can make various obvious changes and / or substitutions based on the example in Figure 17, and the resulting solutions still fall within the scope of the embodiments of this disclosure.

[0167] In the technical scheme described herein, the acquisition, storage, and application of users' personal information comply with the provisions of applicable laws and regulations and do not violate public order and morals.

[0168] According to embodiments of the present disclosure, the present disclosure further provides electronic devices, non-temporary computer-readable storage media, and program products.

[0169] Figure 18 is a block diagram of an electronic device 1800 for implementing an embodiment of the present disclosure. The electronic device refers to various types of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, large computers, and other compatible computers. The electronic device further refers to various types of mobile devices, such as personal digital assistants, cellular phones, intelligent phones, wearable devices, and other similar computer devices. The components, their connections, and functions described in this disclosure are illustrative and do not limit the implementation of anything described or specified in this disclosure.

[0170] As shown in Figure 18, device 1800 includes a computing unit 1801 that can perform various appropriate operations and processes based on computer program instructions stored in read-only memory (ROM) 1802 or computer program instructions loaded from storage unit 1808 into random access memory (RAM) 1803. RAM 1803 can further store various programs and data necessary for the operation of device 1800. The computing unit 1801, ROM 1802, and RAM 1803 are connected to each other via bus 1804. An input / output (I / O) interface 1805 is also connected to bus 1804.

[0171] Multiple components in device 1800 are connected to an I / O interface 1805, which includes an input unit 1806 such as a keyboard and mouse, an output unit 1807 such as various displays and speakers, a storage unit 1808 such as a magnetic disk or optical disk, and a communication unit 1809 such as a network card, modem, or wireless communication transceiver. The communication unit 1809 allows device 1800 to exchange information / data with other devices via computer networks such as the Internet and / or various carrier networks.

[0172] The computing unit 1801 may be a variety of general-purpose and / or dedicated processing components having processing and computing capabilities. Some examples of the computing unit 1801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, a computing unit that executes various machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1801 performs each of the methods and processes described above, for example, a sparse processing method for a sparse attention network and / or a fine-tuning method for a large language model and / or a data processing method for a second large language model. For example, in some embodiments, the sparse processing method for a sparse attention network and / or a fine-tuning method for a large language model and / or a data processing method for a second large language model can be implemented as a computer software program tangibly contained in a machine-readable medium such as a storage unit 1808. In some embodiments, part or all of the computer program can be loaded and / or installed into the device 1800 via ROM 1802 and / or communication unit 1809. When a computer program is loaded into RAM 1803 and executed by computing unit 1801, one or more of the sparse processing methods for the sparse attention network and / or the fine-tuning methods for the large language model and / or the data processing methods for the second large language model described above can be performed. Additionally, in other embodiments, computing unit 1801 may be configured to perform the sparse processing methods for the sparse attention network and / or the fine-tuning methods for the large language model and / or the data processing methods for the second large language model by any other suitable method (e.g., firmware).

[0173] Various embodiments of the systems or technologies described in this disclosure can be implemented by digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standards (ASSPs), systems-on-a-chip (SOCs), complex-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. Each of these embodiments may be implemented by one or more computer programs that run and / or interpret on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0174] Program code for performing the methods of this disclosure can be written in any combination of one or more programming languages. These program codes are provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programming data processing device, so that when the program code is executed by the processor or controller, it can perform the functions / operations defined in the flowcharts and / or block diagrams. The program code may run entirely on a mainframe, partially on a mainframe, as a standalone software package partially on a mainframe and partially on a remote mainframe, or entirely on a remote mainframe or server.

[0175] In this disclosure, machine-readable media may be tangible media containing or storing programs used by or in conjunction with instruction execution systems, devices, or equipment. Machine-readable media may be machine-readable signal media or machine-readable storage media. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any suitable combination of the foregoing. Further specific examples of machine-readable storage media include electrical connections by one or more wires, portable computer disk cartridges, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any combination of the foregoing.

[0176] To provide user interaction, a computer may implement the systems and technologies described herein, which may include a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor), a keyboard and pointing device for the user to provide input to the computer (e.g., a mouse or trackball). Other types of devices may also be used to provide user interaction; for example, the feedback provided to the user may be any form of sensor feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user may be accepted in any form (e.g., acoustic input, voice input, haptic input).

[0177] The systems and technologies described herein can be implemented in computing systems that include background components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include front-end components (e.g., user computers having a graphics user interface or network browser, through which users can interact with embodiments of the systems and technologies described herein), or in any combination of such background components, middleware components, or front-end components. Components of the system can be connected to one another via digital data communication in any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0178] A computer system can include a client and a server. Typically, the client and server are geographically separated and interact via a communication network. The client-server relationship is created by a computer program that operates on the corresponding computer. The server may be a cloud server, a server in a distributed system, or a server incorporating blockchain technology.

[0179] It should be understood that it is possible to reorder, add, or delete items using the various forms of flows shown above. For example, each item described in this disclosure may be performed in parallel, sequentially, or in a different order. This disclosure is not limited to this, as long as the technical solutions disclosed herein can achieve the desired results.

[0180] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, subcombinations, and substitutions are possible due to design considerations and other factors. Any changes, equivalent substitutions, and improvements within the gist and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A sparse processing method for a sparse attention network applied to a computing device, wherein the computing device includes a computing unit and a memory unit, and the sparse processing method for the sparse attention network is: The memory unit constructs a mask sparse representation using constraints, wherein the constraints are that the dimensions of the mask matrix are [B, a, S], where B represents the batch size, a represents the number of heads, S represents the sequence length, and each element in dimension S represents the starting row masked in each column of the mask matrix. The computing unit includes obtaining the mask sparse representation from the memory unit, performing sparse processing on the input data using the mask sparse representation, and storing the sparse processing result in the memory unit. A method for handling sparseness in a sparse attention network.

2. Performing sparse processing on input data using the aforementioned mask sparse representation means The process includes inputting the query matrix, key matrix, value matrix, and mask matrix obtained based on the mask sparse representation of the input data into an attention calculation formula to obtain the output matrix of the input data, Here, the attention calculation formula is: [Math 1] Here, softmax() represents the normalization function, Q represents the query matrix, K represents the key matrix, V represents the value matrix, Y represents the output matrix, T represents the transpose operation, K Ｔ represents transposing the key matrix, Mask represents the mask matrix, the dimensions of Q, K, V, Y are [B, S, H], the dimension of Mask is [B, a, S], and H represents the hidden size. A method for sparse processing of a sparse attention network according to claim 1.

3. Constructing a mask sparse representation using the aforementioned constraints is, In the flash attention block, the operation to construct a mask sparse representation using constraints is performed, A method for sparse processing of a sparse attention network according to claim 1.

4. In the aforementioned flash attention block, performing the operation to construct a mask sparse representation using constraint conditions is: In response to the flash attention block detecting a first construct instruction, the process includes constructing the same mask sparse representation in each Transformer block in the model based on the first construct instruction. A method for sparse processing of a sparse attention network according to claim 3.

5. In the aforementioned flash attention block, performing the operation to construct a mask sparse representation using constraint conditions is: In response to the flash attention block detecting a second construct instruction, the flash attention block includes constructing different mask sparse representations for each Transformer block in the model based on the second construct instruction. A method for sparse processing of a sparse attention network according to claim 3.

6. Constructing a mask sparse representation using the aforementioned constraints is, The data reading block includes performing an operation to construct a mask sparse representation using constraints. A method for sparse processing of a sparse attention network according to claim 1.

7. In the aforementioned data reading block, performing the operation to construct a mask sparse representation using constraint conditions means that In response to the data reading block detecting a third construction instruction, the process includes constructing the same mask sparse representation in each Transformer block in the model based on the third construction instruction. A method for sparse processing of a sparse attention network according to claim 6.

8. In the aforementioned data reading block, performing the operation to construct a mask sparse representation using constraint conditions means that In response to the data reading block detecting a fourth construction instruction, the data reading block includes constructing different mask sparse representations for each Transformer block in the model based on the fourth construction instruction. A method for sparse processing of a sparse attention network according to claim 6.

9. A method for fine-tuning large-scale language models, Obtaining a dataset for the target task, This includes fine-tuning and training the large-scale language model based on the dataset in order to adapt the large-scale language model to the requirements of the target task, Training the large-scale language model based on the aforementioned dataset is: This includes, in conjunction with a sparse attention mechanism, fine-tuning and training the large-scale language model based on the dataset, The sparse attention mechanism uses the sparse processing method for a sparse attention network described in any one of claims 1 to 8. Methods for fine-tuning large-scale language models.

10. A data processing method for large-scale language models, Obtaining input data, This includes inputting the aforementioned input data into the large-scale language model, and the large-scale language model processing the input data in conjunction with a sparse attention mechanism to generate output data, The sparse attention mechanism uses the sparse processing method for a sparse attention network described in any one of claims 1 to 8. Data processing methods for large-scale language models.

11. A sparse processing unit for a sparse attention network applied to a computing device, wherein the computing device includes a computing unit and a memory unit, and the sparse processing unit for the sparse attention network is A construction module for constructing a mask sparse representation in the memory unit using constraints, wherein the constraints are that the dimensions of the mask matrix are [B, a, S], where B represents the batch size, a represents the number of heads, S represents the sequence length, and each element in dimension S represents the starting row masked in each column of the mask matrix; The computing unit includes a sparse processing module for controlling the system to acquire the mask sparse representation from the memory unit, perform sparse processing on the input data using the mask sparse representation, and store the sparse processing result in the memory unit. Sparse processing unit for sparse attention networks.

12. The aforementioned sparse processing module is The query matrix, key matrix, value matrix of the input data, and the mask matrix obtained based on the mask sparse representation are input into an attention calculation formula to obtain the output matrix of the input data. Here, the attention calculation formula is: [Math 2] Here, softmax() represents the normalization function, Q represents the query matrix, K represents the key matrix, V represents the value matrix, Y represents the output matrix, T represents the transpose operation, K Ｔ represents transposing the key matrix, Mask represents the mask matrix, the dimensions of Q, K, V, Y are [B, S, H], the dimension of Mask is [B, a, S], and H represents the hidden size. A sparse processing device for a sparse attention network according to claim 11.

13. The aforementioned construction module is The flash attention block includes a first construction submodule for performing an operation to construct a mask sparse representation using constraints, A sparse processing device for a sparse attention network according to claim 11.

14. The first construction submodule described above is In response to the flash attention block detecting a first construction instruction, it is used to construct the same mask sparse representation in each Transformer block in the model based on the first construction instruction. A sparse processing device for a sparse attention network according to claim 13.

15. The first construction submodule described above is In response to the flash attention block detecting a second construction instruction, it is used to construct different mask sparse representations in each Transformer block in the model based on the second construction instruction. A sparse processing device for a sparse attention network according to claim 13.

16. The aforementioned construction module is The data reading block includes a second construction submodule for performing an operation to construct a mask sparse representation using constraints, A sparse processing device for a sparse attention network according to claim 11.

17. The aforementioned second construction submodule is, In response to the data reading block detecting a third construction instruction, it is used to construct the same mask sparse representation in each Transformer block in the model based on the third construction instruction. A sparse processing device for a sparse attention network according to claim 16.

18. The aforementioned second construction submodule is, In response to the data reading block detecting a fourth construction instruction, it is used to construct different mask sparse representations in each Transformer block in the model based on the fourth construction instruction. A sparse processing device for a sparse attention network according to claim 16.

19. A fine-tuning device for large-scale language models, The first acquisition module for obtaining the dataset for the target task, The system includes a fine-tuning module for fine-tuning and training the large-scale language model based on the dataset in order to adapt the large-scale language model to the requirements of the target task, Training the large-scale language model based on the aforementioned dataset is: This includes, in conjunction with a sparse attention mechanism, fine-tuning and training the large-scale language model based on the dataset, The sparse attention mechanism uses the sparse processing method for a sparse attention network described in any one of claims 1 to 8. A fine-tuning tool for large-scale language models.

20. A data processing device for large-scale language models, A second acquisition module for obtaining input data, The system includes a data processing module for inputting the aforementioned input data into the large-scale language model, and for the large-scale language model to process the input data in conjunction with a sparse attention mechanism to generate output data, The sparse attention mechanism uses the sparse processing method for a sparse attention network described in any one of claims 1 to 8. A data processing unit for large-scale language models.

21. At least one processor, The system comprises at least one processor and a memory that is communicated with by it, An electronic device wherein the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor causes the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-temporary computer-readable storage medium storing computer instructions that cause a computer to perform the method described in any one of claims 1 to 10.

23. A program product stored on a storage medium, which, when executed by a processor in a computer, includes a program for realizing the control method described in any one of claims 1 to 10.

Images