Large language model sparse method, device, electronic device and storage medium
By configuring different predictors for different network layers of the large language model, the sparsity is flexibly selected according to the input features and the network layer feature distribution differences, the efficiency and accuracy problems caused by fixed spars are solved, and the efficient processing of the large language model is achieved. and balanced accuracy.
Patent Information
- Application Number
- CN202411155461.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-22
AI Technical Summary
Existing methods for sparseizing large language models usually adopt fixed sparseness, resulting in insufficient or excessive sparseness, affecting processing efficiency and accuracy.
By configuring different predictors for different network layers, the sparsity is flexibly selected according to the difference in input features and network layer feature distribution, higher sparsity is used to improve processing efficiency, lower sparsity is used to ensure accuracy, and achieve processing efficiency and accuracy. balance between.
It effectively improves the processing efficiency of large language models, while ensuring accuracy, avoiding information loss during sparseness, and achieving a balance of efficiency and accuracy.
Smart Images

Figure CN118673394B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for sparsifying a large language model. Background Art
[0002] In recent years, with the tremendous success of ChatGPT, large language models have been widely used and attracted widespread attention. Due to the enormous scale and number of parameters of large language models, optimizing their inference process to reduce energy consumption, operating costs, and user wait time has become a crucial research and application topic for large language models.
[0003] Sparsification of large language models is a key research area in large language model optimization. Existing methods for sparsifying large language models typically use a fixed sparsity level. This approach can easily lead to insufficient or excessive sparsity, severely impacting the processing efficiency and effectiveness of large language models. Summary of the Invention
[0004] In view of this, an object of the present invention is to provide a method, device, electronic device and storage medium for sparsifying a large language model, which can effectively balance the processing efficiency and accuracy of a large language model.
[0005] In order to achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows:
[0006] In a first aspect, the present invention provides a method for sparsifying a large language model, wherein the large language model includes multiple network layers, each of which includes an activation function; the method includes:
[0007] Split the input text into multiple tokens and generate a feature vector for each token;
[0008] Using the feature vector as input feature and the first network layer of the large language model as the target network layer;
[0009] When the target network layer does not have a predictor, using the weight matrix of the target network layer as a sparse weight;
[0010] When the target network layer has a predictor, the input features are input into the predictor of the target network layer to obtain the importance corresponding to multiple preset sparsities of each input feature, a target sparsity is determined according to the importance corresponding to the multiple preset sparsities of each input feature, and a sparse calculation is performed on the weight matrix of the target network layer according to a mask corresponding to the target sparsity to obtain a sparse weight; the mask is used to sparsify the weight matrix;
[0011] The product of each input feature and the sparse weight is input into the activation function of the target network layer, and the output result of the activation function is used as the input feature of the next network layer, and the next network layer is used as the target network layer. The process is performed layer by layer until the last network layer of the large language model, and the output result of the activation function of the last network layer is used as the output result of the large language model.
[0012] In an optional embodiment, determining the target sparsity according to the importance corresponding to the multiple preset sparsities of each input feature includes:
[0013] For each of the input features, the preset sparsity corresponding to the importance exceeding the threshold is used as the preliminary sparsity, and the preliminary sparsity with the largest sparsity is used as the candidate sparsity;
[0014] The number of the candidate sparsities is counted, and the candidate sparsity with the largest number is determined as the target sparsity.
[0015] In an optional embodiment, performing sparse calculation on the weight matrix of the target network layer according to the mask corresponding to the target sparsity to obtain sparse weights includes:
[0016] The mask corresponding to the target sparsity is multiplied by the weight matrix of the target network layer to obtain the sparse weight.
[0017] In an optional embodiment, each network layer is provided with a weight matrix, and the predictor is obtained in the following manner:
[0018] For each network layer requiring sparsity adjustment, a weight importance matrix is obtained based on a plurality of training features and the weight matrix; the weight importance matrix is used to characterize the importance of each weight in the weight matrix; the training features of the first network layer are obtained by extracting features based on the received training text; the training features of network layers other than the first network layer are the outputs of the previous network layer;
[0019] Obtaining a mask corresponding to each preset sparsity according to the weight importance matrix and a plurality of preset sparsities;
[0020] Obtaining the theoretical importance of each preset sparsity corresponding to each token vector according to the token vector corresponding to the training feature, the weight matrix, and the mask corresponding to each preset sparsity;
[0021] The fully connected layer to be trained is iteratively trained according to the theoretical importance of each of the token vectors and the corresponding preset sparsity to obtain the predicted importance of each preset sparsity, and the fully connected layer whose predicted importance meets the acceptance condition is used as the predictor.
[0022] In an optional embodiment, obtaining a weight importance matrix based on the plurality of training features and the weight matrix includes:
[0023] Generating a corresponding training matrix according to each of the training features, and generating a three-dimensional training matrix according to all of the training matrices;
[0024] generating a three-dimensional weight matrix according to the weight matrix;
[0025] Taking the absolute value of each element of the three-dimensional training matrix and the three-dimensional weight matrix and performing dot product thereon to obtain an activation matrix;
[0026] The activation matrix is summed over the number dimension of the training matrix to obtain the weight importance matrix.
[0027] In an optional embodiment, obtaining a mask corresponding to each preset sparsity according to the weight importance matrix and a plurality of preset sparsities includes:
[0028] For each of the preset sparsity, when the sparse format is a ratio, calculating the sparse quantity according to the preset sparsity, and obtaining the coordinates of the elements with small weights in the weight importance matrix according to the sparse quantity;
[0029] When the sparse format is a ratio, the coordinates of the elements with smaller weights in the weight importance matrix are obtained in sequence according to the ratio;
[0030] An all-one matrix is generated according to the weight importance matrix, and the elements corresponding to the element coordinates are set to 0 to obtain a mask corresponding to the preset sparsity.
[0031] In an optional embodiment, obtaining the theoretical importance of each preset sparsity corresponding to each token vector according to the token vector corresponding to the training feature, the weight matrix, and the mask corresponding to each preset sparsity includes:
[0032] For each of the token vectors, an original importance matrix is obtained based on the token vector and the weight matrix; the token vectors of the first network layer are obtained by performing feature extraction on the tokens obtained by splitting the training text; the training features of the network layers other than the first network layer are output after the token vectors are processed by the previous network layer;
[0033] Obtaining a theoretical importance matrix corresponding to each of the preset sparsities according to the original importance matrix and the mask corresponding to each of the preset sparsities;
[0034] The theoretical importance of each preset sparsity corresponding to each token vector is obtained according to each of the theoretical importance matrices and the original importance matrix.
[0035] In a second aspect, the present invention provides a large language model sparsification device, wherein the large language model includes multiple network layers, each of which includes an activation function; the device includes:
[0036] A preprocessing module is used to split the input text into multiple tokens and generate a feature vector for each token;
[0037] A sparse module is used to take the feature vector as the input feature and the first network layer of the large language model as the target network layer; when the target network layer does not have a predictor, the weight matrix of the target network layer is used as the sparse weight; when the target network layer has a predictor, the input feature is input into the predictor of the target network layer to obtain the importance corresponding to multiple preset sparsities of each input feature, the target sparsity is determined according to the importance corresponding to the multiple preset sparsities of each input feature, and the weight matrix of the target network layer is sparsely calculated according to the mask corresponding to the target sparsity to obtain the sparse weight; the mask is used to sparsify the weight matrix; the product of each input feature and the sparse weight is input into the activation function of the target network layer, and the output result of the activation function is used as the input feature of the next network layer, the next network layer is used as the target network layer, and processing is carried out layer by layer until the last network layer of the large language model, and the output result of the activation function of the last network layer is used as the output result of the large language model.
[0038] In a third aspect, the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor can execute the machine-executable instructions to implement the large language model sparsification method described in any of the aforementioned embodiments.
[0039] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the large language model sparsification method as described in any one of the aforementioned embodiments.
[0040] Compared with the prior art, the large language model sparsification method, device, electronic device and storage medium provided by the embodiments of the present invention configure different predictors for different network layers. The predictors make predictions based on different input features, fully considering the differences in the distribution of different input features and features of different network layers. The appropriate sparsity can be flexibly selected according to the input features and network features. For inputs with a high degree of sparsification, a higher sparsity is used for faster calculations to improve the processing efficiency of the large language model. For inputs with a low degree of sparsification, a lower sparsity is used to ensure the accuracy of the large language model, thereby achieving a balance between the processing efficiency and accuracy of the large language model.
[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.
[0043] Figure 1 A schematic diagram showing the prior art of using a fixed sparsity for thinning is shown.
[0044] Figure 2 A schematic flow chart of a large language model sparsification method provided by an embodiment of the present invention is shown.
[0045] Figure 3 A schematic diagram of generating sparse weights provided by an embodiment of the present invention is shown.
[0046] Figure 4 Another flowchart of the large language model sparsification method provided by an embodiment of the present invention is shown.
[0047] Figure 5 A schematic diagram of generating a mask provided by an embodiment of the present invention is shown.
[0048] Figure 6 Another schematic diagram of generating a mask provided by an embodiment of the present invention is shown.
[0049] Figure 7 A schematic diagram of generating a theoretical importance matrix provided by an embodiment of the present invention is shown.
[0050] Figure 8A block diagram of a large language model sparsification device provided by an embodiment of the present invention is shown.
[0051] Figure 9 A block diagram of an electronic device provided by an embodiment of the present invention is shown.
[0052] Icon: 100 - electronic device; 110 - memory; 120 - processor; 130 - communication module; 300 - large language model sparsification device; 301 - preprocessing module; 302 - sparsification module. DETAILED DESCRIPTION
[0053] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.
[0054] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely intended to represent selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.
[0055] It should be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not preclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.
[0056] Sparsification of large language models includes static sparsification and dynamic sparsification. Static sparsification includes amplitude pruning, SparseGPT, and Wanda, while dynamic sparsification includes DejaVu and PowerInfer. Static sparsification refers to pre-pruning model weights before forward inference, setting some weights to 0. This reduces computational effort and accelerates inference during model forward inference. Dynamic sparsification refers to selecting only a subset of input features or weights for calculation during forward inference, thereby reducing computational effort and accelerating inference.
[0057] In static sparsification or dynamic sparsification, the sparsity is the proportion of elements that are sparse (values become 0 or do not participate in the calculation). Existing large language model sparsification usually sets a fixed sparsity. For example, based on experience, the sparsity is set to 50%. For static sparsification, 50% of the weights are sparsified to 0. For dynamic sparsification, 50% of the elements are selected for calculation, such as Figure 1 shown. Figure 1 In the figure, the left matrix is the dense weight matrix before sparsification, and the right matrix is the sparse weight matrix after sparsification with a sparsity of 50%. The white box in the right matrix indicates that the weight becomes 0.
[0058] The inventors discovered that if different network layers could adapt their operations to different sparsities for different inputs, they could achieve the desired sparsity while minimizing the impact on the accuracy of the large language model. However, due to the sheer size and number of network layers in large language models, it is difficult to find the appropriate sparsity for each layer. This makes some global search methods difficult to implement.
[0059] Even without considering inter-layer correlations, independently adjusting each layer still requires hundreds or even thousands of network layers. The problem becomes even more complex when considering different inputs matching different sparsities. Therefore, mainstream large-scale language models all use a fixed sparsity as a hyperparameter, such as setting it to 0.5. This fixed sparsity can be manually adjusted and tested repeatedly to find the optimal value.
[0060] When a large language model uses fixed sparsity sparsification, all network layers requiring sparsification in the large language model use the preset fixed sparsity for sparsification calculations. Because large language models include multiple network layers, the weight feature distribution of each network layer varies. Some network layers have a large amount of space available for sparsification, and even after sparsification using a fixed sparsity, there is still some space left unused. However, other network layers have a small amount of space available for sparsification, and sparsification using a fixed sparsity inevitably results in the loss of important information, affecting the accuracy of the large language model. Even within the same network layer, the degree of sparsification available for different input sequences varies.
[0061] Based on this, the core idea of the large language model sparsification method and device provided by the embodiment of the present invention is that different network layers are configured with different predictors, and the predictors make predictions based on different input features, fully considering the differences in different input features and feature distributions of different network layers, and can flexibly select appropriate sparsity according to input features and network features. For inputs with a high degree of sparsification, a higher sparsity is used for faster calculations to improve the processing efficiency of the large language model. For inputs with a low degree of sparsification, a lower sparsity is used to ensure the accuracy of the large language model, thereby achieving a balance between the processing efficiency and accuracy of the large language model.
[0062] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0063] Please refer to Figure 2 , Figure 2 A schematic flow chart of a large language model sparsification method provided by an embodiment of the present invention is shown. The method is applied to a large language model, which includes multiple network layers, each of which includes an activation function. The method includes the following steps:
[0064] Step S101: split the input text into multiple tokens and generate a feature vector for each token.
[0065] In this embodiment of the present invention, the large language model segments input text into multiple tokens and performs feature extraction on each token to obtain a feature vector corresponding to each token. The input text can be in various forms, such as e-books, articles, web pages, and prompts, and the tokens can be words, punctuation marks, or other language units.
[0066] Step S102: Use the feature vector as input feature and the first network layer of the large language model as the target network layer.
[0067] In an embodiment of the present invention, the inference calculation is started from the first network layer according to the hierarchy of the network layers in the large language model. The first network layer is first used as the target network layer, and the feature vector is used as the input feature of the target network layer.
[0068] Step S103: determine whether there is a predictor in the target network layer.
[0069] In the embodiment of the present invention, it is queried whether the target network layer is provided with a predictor. If the target network layer does not have a predictor, step S104 is executed; if the target network layer has a predictor, step S105 is executed.
[0070] It's important to note that large language models typically consist of a stacked Transformer architecture, with the primary parameters and computation concentrated in the attention and multilayer perceptron (MLP) modules within the Transformer. The attention module includes a linear layer that calculates the query, key, and value matrices, as well as a linear transformation layer that merges the multi-head attention features. The MLP module typically consists of two linear layers, which can be sparsified. In practice, sparsification can be performed on all of these layers within the Transformer architecture, or the first few Transformer modules can be skipped. Generally, for large language models (e.g., 70B or larger), all Transformer modules can be sparsified. For smaller models, the first few Transformer modules can be skipped to maintain higher accuracy.
[0071] Step S104: When there is no predictor in the target network layer, the weight matrix of the target network layer is used as the sparse weight.
[0072] In an embodiment of the present invention, when the target network layer does not have a predictor, it indicates that the target network layer does not need to be sparsified, and the weight matrix of the target network layer is directly used as the sparse weight, that is, the sparsity is 0.
[0073] Step S105: When a predictor exists in the target network layer, the input features are input into the predictor of the target network layer to obtain the importance corresponding to multiple preset sparsities of each input feature, the target sparsity is determined according to the importance corresponding to the multiple preset sparsities of each input feature, and the weight matrix of the target network layer is sparsely calculated according to the mask corresponding to the target sparsity to obtain the sparse weight.
[0074] Among them, the mask is used to sparsify the weight matrix.
[0075] In an embodiment of the present invention, when a predictor exists in a target network layer, indicating that the sparsity of the target network layer needs to be adjusted, each input feature is sequentially input into the predictor of the target network layer, or multiple input features are input into the predictor of the target network layer together. The predictor predicts each input feature according to multiple preset sparsities, and obtains the importance corresponding to the multiple preset sparsities of each input feature. The preset sparsities correspond to masks one-to-one, and each target network layer is provided with a mask corresponding to each preset sparsity.
[0076] It should be noted that the input features can be input one by one into the predictor of the target network layer for prediction to obtain the importance corresponding to multiple preset sparsities corresponding to the input features. Alternatively, multiple input features can be input together into the predictor of the target network layer for prediction to obtain the importance corresponding to multiple preset sparsities corresponding to the multiple input features. The present invention is not limited to this.
[0077] The preset sparsity can be configured in combination with the sparse computing framework and hardware support actually used. For example, if the GPU chip supports 2x, 4x, 8x and 16x sparsity, that is, the sparse ratios are 1:2, 3:4, 7:8 and 15:16, the preset sparsity and the corresponding mask can be set to 4 levels, corresponding to the sparse ratios supported. If the sparse computing framework used supports unstructured sparsity of arbitrary ratios, the preset sparsity can be set to 0.25, 0.5 and 0.75, or 0.1, 0.2, ..., 0.9. Even if the sparse computing framework or hardware only supports one sparse format, the large language model sparsification method provided in the embodiment of the present invention can be used to generate different masks based on different input features, or to determine the network layer that needs to adjust the sparsity based on different input features, thereby improving the processing efficiency of the large language model.
[0078] Step S106: Input the product of each input feature and the sparse weight into the activation function of the target network layer.
[0079] In an embodiment of the present invention, the product of each input feature and the sparse weight is passed through the activation function of the target network layer, so that the large language model can capture more complex features.
[0080] Step S107 , determining whether the target network layer is the last network layer of the large language model.
[0081] In an embodiment of the present invention, it is determined whether the current target network layer is the last network layer of the large language model. If the current target network layer is not the last network layer of the large language model, step S108 is executed; if the current target network layer is the last network layer of the large language model, step S109 is executed.
[0082] Step S108: Use the output result of the activation function as the input feature of the next network layer, and use the next network layer as the target network layer.
[0083] In this embodiment of the present invention, after executing step S108, a new target network layer and the input features corresponding to the new target network layer are obtained. Steps S103 to S108 are continued based on the new target network layer and the corresponding input features until the last network layer of the large language model is processed, and then step S109 is continued.
[0084] Step S109: The output result of the activation function of the last network layer is used as the output result of the large language model.
[0085] In an embodiment of the present invention, the output result of the activation function of the last network layer is used as the output result of the large language model, and the output result of the large language model is sent to the user.
[0086] In summary, the large language model sparsification method provided by the embodiment of the present invention configures different predictors for different network layers, and the predictors make predictions based on different input features, fully considering the differences in the distribution of different input features and features of different network layers. The appropriate sparsity can be flexibly selected according to the input features and network features. For inputs with a high degree of sparsification, a higher sparsity is used for faster calculations to improve the processing efficiency of the large language model. For inputs with a low degree of sparsification, a lower sparsity is used to ensure the accuracy of the large language model, thereby achieving a balance between the processing efficiency and accuracy of the large language model.
[0087] Optionally, in practical applications, a threshold of information loss is set in combination with the application scenario, and target sparsity that meets the threshold condition is screened according to different input features for sparsification. Figure 2 The sub-step of determining the target sparsity according to the importance corresponding to the multiple preset sparsities of each input feature in step S105 may include:
[0088] For each input feature, the preset sparsity corresponding to the importance exceeding the threshold is used as the preliminary sparsity, and the preliminary sparsity with the largest sparsity is used as the candidate sparsity; the number of candidate sparsities is counted, and the candidate sparsity with the largest number is determined as the target sparsity.
[0089] As a possible implementation, assume that the preset sparsity is 25%, 50%, and 75%, respectively, and the importance threshold is set to 0.6. Input feature A is input to the predictor, and the importance of the predictor output is 0.88, 0.65, and 0.25, respectively. Among them, 0.88 is the importance of input feature A when it is sparsely spaced at the preset sparsity of 0.25, 0.65 is the importance of input feature A when it is sparsely spaced at the preset sparsity of 0.5, and 0.25 is the importance of input feature A when it is sparsely spaced at the preset sparsity of 0.75.
[0090] Since 0.65 and 0.88 are greater than the threshold value of 0.6, the preset sparsities of 0.25 and 0.5 are used as the preliminary sparsities, and the preset sparsity of 0.5 is used as the candidate sparsity. Similarly, the candidate sparsity corresponding to each input feature is determined, and the number of each candidate sparsity is counted. For example, if the candidate sparsities are 0.25 and 0.5, the number of candidate sparsity 0.25 is 500, and the number of candidate sparsity 0.5 is 600, then the candidate sparsity 0.5 is determined as the target sparsity. Each target network layer performs sparse operations according to its own target sparsity, and the target network layer corresponds to the target sparsity one by one.
[0091] It can be seen that the embodiment of the present invention utilizes the differences in input features in each network layer to dynamically adjust the sparsity of the network layer, which can better adaptively adjust the sparsity according to the input features, thereby reducing information loss and improving the processing efficiency of the large language model through sparsification as much as possible while ensuring the accuracy of the large language model.
[0092] Optionally, in practical applications, the weight matrix is sparsely processed using a sparsity suitable for different input features and different network features, effectively reducing the weights involved in the calculation. Figure 2 The sub-step of performing sparse calculation on the weight matrix of the target network layer according to the mask corresponding to the target sparsity to obtain sparse weights in step S105 may include:
[0093] The sparse weights are obtained by multiplying the mask corresponding to the target sparsity by the weight matrix of the target network layer.
[0094] In an embodiment of the present invention, a target mask is obtained according to the target sparsity, and the weight matrix of the target network layer is sparsified according to the target mask to obtain sparse weights.
[0095] As a possible implementation method, Figure 3 For example, assuming the target sparsity is 0.5, a dot product operation is performed based on the mask corresponding to the sparsity of 0.5 and the weight matrix. This means that the weight matrix elements corresponding to the positions where the elements in the mask are 1 are copied to the positions corresponding to the sparse weights, and the elements in the sparse weights corresponding to the positions where the elements in the mask are 0 are set to 0. Compared to the weight matrix, the number of elements involved in the sparse weight operation is reduced from 15 to 8, effectively reducing the computational complexity of the large language model inference process and maximizing the processing efficiency of the large language model while ensuring its accuracy.
[0096] Optionally, before using the predictor to perform sparsification on a large language model, it is necessary to set a fully connected layer to be trained for each network layer that needs to adjust the sparsity, and train each fully connected layer to be trained to obtain a trained predictor. The following describes the training process of the predictor. Please refer to Figure 4 , Figure 4Another flow chart of a large language model sparsification method provided by an embodiment of the present invention is shown. The predictor is obtained by:
[0097] Step S201 : For each network layer that needs to adjust sparsity, a weight importance matrix is obtained based on multiple training features and a weight matrix.
[0098] Among them, the weight importance matrix is used to characterize the importance of each weight in the weight matrix; the training features of the first network layer are obtained by extracting features based on the received training text; the training features of the network layers other than the first network layer are the output of the previous network layer.
[0099] Step S202 : obtaining a mask corresponding to each preset sparsity according to the weight importance matrix and a plurality of preset sparsities.
[0100] Step S203 , obtaining the theoretical importance of each preset sparsity corresponding to each token vector based on the token vector corresponding to the training feature, the weight matrix, and the mask corresponding to each preset sparsity.
[0101] In this embodiment of the present invention, taking the first network layer as an example, the training text is segmented into multiple tokens, and features are extracted from the multiple tokens to obtain multiple token vectors corresponding to the training text, i.e., multiple token vectors corresponding to the training features. The token vectors of the second network layer are the token vectors output by the first network layer.
[0102] For each network layer requiring sparsity adjustment, a weight importance matrix is generated based on the training features received by the network layer. Multiple masks are generated for each training feature based on multiple preset sparsities. The theoretical importance of each token vector based on the preset sparsity is then calculated based on the mask and the token vectors corresponding to each training feature.
[0103] Step S204, iteratively train the fully connected layer to be trained according to the theoretical importance of each token vector and the corresponding preset sparsity, obtain the predicted importance of each preset sparsity, and use the fully connected layer whose predicted importance meets the acceptance condition as the predictor.
[0104] In an embodiment of the present invention, the token vector and the corresponding theoretical importance of each preset sparsity are input into the fully connected layer to be trained in turn to obtain the predicted importance of each preset sparsity corresponding to the token vector. When the difference between the predicted importance and the theoretical importance of each preset sparsity is less than the acceptance threshold, the fully connected layer is used as a predictor.
[0105] It is worth mentioning that the prediction principle of the predictor is essentially a regression task. In addition to training a fully connected layer as a predictor, regression models such as support vector regression models, decision tree regression models, and gradient boosting tree models can also be trained as predictors. This is not limited to the present invention.
[0106] It should be noted that the embodiment of the present invention determines the mask corresponding to each preset sparsity based on the weight importance matrix of each network layer, and uses the mask of each preset sparsity to perform sparsification. When the information loss rate of the network layer after sparsification exceeds the preset loss threshold, it indicates that the network layer does not need to be sparsified. In other words, the network layer whose information loss rate after sparsification exceeds the preset loss threshold is not set with a predictor, and the network layer whose information loss rate after sparsification does not exceed the preset loss threshold is set with a predictor. Among them, the preset loss threshold can be set based on experience.
[0107] It can be seen that the embodiment of the present invention uses the training features received by each network layer in the large language model to generate masks of different sparsities corresponding to each training feature, and uses the masks of different sparsities to generate token vectors based on the theoretical importance of different sparsities, and finally uses different token vectors and corresponding theoretical importance to iteratively train to obtain a predictor. The predictor can be used to predict the importance or information retention of different input features corresponding to different sparsities. In other words, the predictor can be used to measure the size of the loss after sparsification using different sparsities, thereby dynamically adjusting the sparsity of the large language model and improving the processing efficiency of the large language model as much as possible while ensuring the accuracy of the large language model.
[0108] Optionally, in practical applications, the importance of weights is determined by combining the training features and the weight matrix, and the weights are as important as possible when performing weight sparsification, thereby reducing information loss. Figure 4 The sub-step of obtaining a weight importance matrix based on the multiple training features and the weight matrix in step S201 may include:
[0109] Generate a corresponding training matrix according to each training feature, and generate a three-dimensional training matrix according to all training matrices; generate a three-dimensional weight matrix according to the weight matrix; take the absolute value of each element of the three-dimensional training matrix and the three-dimensional weight matrix and perform dot product to obtain an activation matrix; sum the activation matrix over the number dimension of the training matrix to obtain a weight importance matrix.
[0110] In the embodiment of the present invention, it is assumed that the preset feature length is D, the number of training features is N, and the D-dimensional training features , The weight matrix . Each training feature is copied and expanded to The training matrix , stack all the training matrices to get The three-dimensional training matrix The weight matrix is copied according to the number of training features, and the copied weight matrices are stacked to obtain The three-dimensional weight matrix .Will and Perform element-wise multiplication to get The activation matrix , sum the activation matrix A over the number dimension of the training matrix, and get The weight importance matrix The dimension of the weight importance matrix is the same as that of the weight matrix, that is, the number of rows and columns of the weight importance matrix is the same as that of the weight matrix.
[0111] It is worth mentioning that when generating the weight importance matrix, the activation matrix A can also be averaged or regularized in the number dimension of the training matrix. The present invention is not limited to this. As a possible implementation, assuming that the preset feature length is 4, the number of training features is 2, and the two training features are and , the weight matrix w is:
[0112]
[0113] According to the training characteristics and Copy and expand to obtain the corresponding training matrix and , and according to the training matrix and Generate a three-dimensional training matrix X.
[0114]
[0115]
[0116] X=
[0117] According to the number of training features, a three-dimensional weight matrix W is generated according to the weight matrix, and the absolute value of each element of the three-dimensional training matrix and the three-dimensional weight matrix is taken and dot multiplication is performed to obtain the activation matrix A.
[0118] W=
[0119] A=
[0120] in, Is the training matrix and the weight matrix The result is obtained by taking the absolute value of each element and performing dot multiplication. Is the training matrix and the weight matrix Obtained by taking the absolute value of each element and performing dot multiplication.
[0121]
[0122]
[0123] Sum the activation matrix over the number dimension of the training matrix, that is, according to and The weight importance matrix S is obtained by summing each element.
[0124]
[0125] It should be noted that the three-dimensional training matrix, the three-dimensional weight matrix, and the activation matrix are calculated using matrix parallelism to obtain the weight importance matrix. When calculating the weight importance matrix, the number of training matrices N can be set based on experience, for example, N can be any integer between 128 and 512. The training features of the first network layer correspond one-to-one to the received training text. Each training text is divided into multiple tokens, and feature extraction is performed on each token to obtain a token vector. The eigenvalues of the token vector corresponding to a training text are averaged to obtain the corresponding training feature.
[0126] As can be seen, the embodiments of the present invention fully consider the differences in the distribution of different input features and features of different network layers to construct a weight importance matrix suitable for each network layer, and then obtain a mask suitable for each network layer based on the weight importance matrix. When generating the weight importance matrix, the training features and weight matrix of each network layer input are used for parallel calculation, so as to utilize GPU chips for acceleration, effectively improving the inference speed and processing efficiency of large language models.
[0127] Optionally, in practical applications, dynamically generating masks for each network layer based on the weight importance matrix can effectively reduce information loss. Figure 4 The sub-steps of step S202 may include:
[0128] For each preset sparsity, when the sparse format is a ratio, the sparse quantity is calculated according to the preset sparsity, and the coordinates of the elements with small weights in the weight importance matrix are obtained according to the sparse quantity; when the sparse format is a proportion, the coordinates of the elements with small weights in the weight importance matrix are obtained in sequence according to the proportion; an all-1 matrix is generated according to the weight importance matrix, and the elements corresponding to the element coordinates are set to 0 to obtain the mask corresponding to the preset sparsity.
[0129] In an embodiment of the present invention, if the sparse format is a ratio, that is, the preset sparsity is unstructured data, the sparse quantity is calculated according to the number of elements in the weight importance matrix and the preset sparsity, and the element coordinates corresponding to the first sparse quantity weights are obtained in ascending order of the weights in the weight importance matrix. An all-1 matrix is generated according to the rows and columns of the weight importance matrix, and the elements corresponding to the element coordinates are updated to 0 to obtain a mask corresponding to the preset sparsity.
[0130] In one possible implementation, Figure 5 For example, assume that the weight importance matrix S is Matrix, with a preset sparsity of 50%. First, determine the number of sparsity elements (i.e., 8) based on the number of elements in the weight importance matrix S (i.e., 16) and the preset sparsity (i.e., 50%). Second, start searching from the smallest weight in the weight importance matrix S and find the coordinates of the 8 elements with small weights (i.e., Figure 5 The white box position in the middle matrix). Finally, generate a The full 1 matrix updates the elements corresponding to the element coordinates to 0, and we get Figure 5 The mask on the right.
[0131] Note that a 1 in the mask indicates that the element at that coordinate is to be retained, while a 0 indicates that the element at that coordinate is to be thinned. Similarly, replace 50% with the corresponding ratio and save the masks corresponding to the desired sparsity. For example, save the masks corresponding to the preset sparsities of 25%, 50%, and 75%.
[0132] In an embodiment of the present invention, if the sparse format is a ratio, that is, the preset sparsity is semi-structured data, for example, the preset sparsity is K:L, then when generating the mask, K elements out of every L elements are set to 0. The element coordinates of the K elements are the coordinates of the elements with the smallest weights selected from the corresponding L weights in the weight importance matrix in ascending order.
[0133] In another possible embodiment, Figure 6 For example, assume that the weight importance matrix S is Matrix, preset sparsity is 50%, 2:4 and 3:4. Figure 6 In
[15] , masks corresponding to each preset sparsity are generated based on the weight importance matrix S and three preset sparsities (50%, 2:4, and 3:4, respectively). It can be seen that although the preset sparsities of 50% and 2:4 both retain common elements, the preset sparsity of 50% thins out the coordinates of elements with small weights in the weight importance matrix S, while the preset sparsity of 2:4 thins out the coordinates of elements with small weights for every 4 weights.
[0134] It can be seen that the embodiment of the present invention constructs a weight importance matrix suitable for each network layer based on the differences in different input features and feature distributions of different network layers, and generates masks of preset sparsity in each network layer based on the weight importance matrix. By setting masks that meet the distribution of different input features and different network layer features, more zero values are introduced, effectively reducing the number of weights actually involved in the calculation, thereby improving the speed of matrix operations in large language models. Since only non-zero weights need to be stored in large language models, the memory space occupied is effectively reduced through sparse processing, further optimizing the processing efficiency of large language models.
[0135] Optionally, in practical applications, different token vectors are used to generate theoretical importance of each preset sparsity, effectively enriching the training data of the predictor, thereby improving the prediction accuracy of the predictor. Figure 4 The sub-steps of step S203 may include:
[0136] For each token vector, the original importance matrix is obtained based on the token vector and the weight matrix; the token vector of the first network layer is obtained by feature extraction of the tokens obtained by splitting the training text; the training features of the network layers other than the first network layer are the output of the token vector after processing the previous network layer; based on the original importance matrix and the mask corresponding to each preset sparsity, the theoretical importance matrix corresponding to each preset sparsity is obtained; based on each theoretical importance matrix and the original importance matrix, the theoretical importance of each preset sparsity corresponding to each token vector is obtained.
[0137] In a possible implementation, for each token vector, the token vector is copied and expanded to obtain a token matrix, and the original importance matrix is obtained based on the token matrix and the weight matrix. Figure 7 For example, assuming the token vector is , the weight matrix is the weight matrix in the previous example , the preset sparsity is 25%, 50% and 75%. The original importance matrix T is calculated as follows:
[0138]
[0139] According to the original importance matrix T, the total weight of the token vector before sparsification is 128. According to the preset sparsity and the original importance matrix T, the masks corresponding to the sparsity of 25%, the masks corresponding to the sparsity of 50%, and the masks corresponding to the sparsity of 75% are obtained respectively. Then, according to the masks corresponding to the preset coefficients and the original importance matrix T, the theoretical importance matrix corresponding to the sparsity of 25%, the theoretical importance matrix corresponding to the sparsity of 50%, and the theoretical importance matrix corresponding to the sparsity of 50% are obtained respectively. Finally, according to each theoretical importance matrix, the weight sum corresponding to the sparsity of 25%, the weight sum corresponding to the sparsity of 50%, and the weight sum corresponding to the sparsity of 75% are obtained respectively. The three weight sums are 115, 90, and 58 respectively.
[0140] According to the sum of the weights after sparseness and the sum of the weights before sparseness, we get the theoretical importance of 0.9 for a sparsity of 25%, 0.7 for a sparsity of 50%, and 0.45 for a sparsity of 75%. Thus, we get a set of training data, namely the token vector and theoretical importance Assuming that approximately 1,000 training texts are used when training the predictor, splitting them by token will yield approximately 100,000 sets of training data. For each network layer that requires adjustment of sparsity, the corresponding training data for each network layer must be obtained using the aforementioned method for calculating theoretical importance.
[0141] Based on the same inventive concept, an embodiment of the present invention also provides a large language model sparsification device. Its basic principles and technical effects are the same as those of the above embodiment. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiment.
[0142] Please refer to Figure 8 , Figure 8 A block diagram of a large language model sparsification apparatus 300 provided by an embodiment of the present invention is shown. The large language model includes multiple network layers, each of which includes an activation function. The large language model sparsification apparatus 300 includes a preprocessing module 301 and a sparsification module 302.
[0143] The preprocessing module 301 is used to split the input text into multiple tokens and generate a feature vector for each token;
[0144] The sparse module 302 is used to use the feature vector as the input feature and the first network layer of the large language model as the target network layer; when the target network layer does not have a predictor, the weight matrix of the target network layer is used as the sparse weight; when the target network layer has a predictor, the input feature is input into the predictor of the target network layer to obtain the importance corresponding to multiple preset sparsities of each input feature, the target sparsity is determined according to the importance corresponding to the multiple preset sparsities of each input feature, and the weight matrix of the target network layer is sparsely calculated according to the mask corresponding to the target sparsity to obtain the sparse weight; the mask is used to sparsify the weight matrix; the product of each input feature and the sparse weight is input into the activation function of the target network layer, and the output result of the activation function is used as the input feature of the next network layer, and the next network layer is used as the target network layer. The process is performed layer by layer until the last network layer of the large language model, and the output result of the activation function of the last network layer is used as the output result of the large language model.
[0145] In summary, the large language model sparsification device provided by the embodiment of the present invention has different predictors configured for different network layers. The predictors make predictions based on different input features, fully considering the differences in the distribution of different input features and features of different network layers. The appropriate sparsity can be flexibly selected according to the input features and network features. For inputs with a high degree of sparsification, a higher sparsity is used for faster calculations to improve the processing efficiency of the large language model. For inputs with a low degree of sparsification, a lower sparsity is used to ensure the accuracy of the large language model, thereby achieving a balance between the processing efficiency and accuracy of the large language model.
[0146] Optionally, the sparsity module 302 is specifically used to, for each input feature, use the preset sparsity corresponding to the importance exceeding the threshold as the preliminary sparsity, and use the preliminary sparsity with the largest sparsity as the candidate sparsity; count the number of candidate sparsities, and determine the candidate sparsity with the largest number as the target sparsity.
[0147] Optionally, the sparse module 302 is specifically configured to perform a dot multiplication of a mask corresponding to the target sparsity and a weight matrix of the target network layer to obtain a sparse weight.
[0148] Optionally, each network layer is provided with a weight matrix. The preprocessing module 301 is further used to obtain a weight importance matrix for each network layer that needs to adjust the sparsity based on multiple training features and the weight matrix; the weight importance matrix is used to characterize the importance of each weight in the weight matrix; the training features of the first network layer are obtained based on the features extracted from the received training text; the training features of the network layers other than the first network layer are the outputs of the previous network layer; based on the weight importance matrix and multiple preset sparsities, a mask corresponding to each preset sparsity is obtained; based on the token vector corresponding to the training feature, the weight matrix and the mask corresponding to each preset sparsity, the theoretical importance of each preset sparsity corresponding to each token vector is obtained; based on each token vector and the theoretical importance of each corresponding preset sparsity, the fully connected layer to be trained is iteratively trained to obtain the predicted importance of each preset sparsity, and the fully connected layer whose predicted importance meets the acceptance criteria is used as a predictor.
[0149] Optionally, the preprocessing module 301 is specifically used to generate a corresponding training matrix according to each training feature, and generate a three-dimensional training matrix according to all the training matrices; generate a three-dimensional weight matrix according to the weight matrix; take the absolute value of each element of the three-dimensional training matrix and the three-dimensional weight matrix and perform dot product to obtain an activation matrix; sum the activation matrix in the quantity dimension of the training matrix to obtain a weight importance matrix.
[0150] Optionally, the preprocessing module 301 is specifically used to calculate the sparse quantity according to the preset sparsity for each preset sparsity when the sparse format is a ratio, and obtain the coordinates of the elements with small weights in the weight importance matrix according to the sparse quantity; when the sparse format is a proportion, obtain the coordinates of the elements with small weights in the weight importance matrix in sequence according to the proportion; generate an all-1 matrix according to the weight importance matrix, and set the elements corresponding to the element coordinates to 0 to obtain a mask corresponding to the preset sparsity.
[0151] Optionally, the preprocessing module 301 is specifically used to obtain, for each token vector, an original importance matrix based on the token vector and the weight matrix; the token vector of the first network layer is obtained by feature extraction of the tokens obtained by splitting the training text; the training features of the network layers other than the first network layer are output after the previous network layer processes the token vector; based on the original importance matrix and the mask corresponding to each preset sparsity, a theoretical importance matrix corresponding to each preset sparsity is obtained; based on each theoretical importance matrix and the original importance matrix, the theoretical importance of each preset sparsity corresponding to each token vector is obtained.
[0152] Please refer to Figure 9 , Figure 9This is a block diagram of an electronic device 100 according to an embodiment of the present invention. The electronic device 100 can be any device with data processing capabilities, such as a personal computer, laptop computer, or server. The electronic device 100 includes a memory 110, a processor 120, and a communication module 130. The memory 110, processor 120, and communication module 130 are electrically connected to each other, directly or indirectly, to enable data transmission or exchange. For example, these components can be electrically connected via one or more communication buses or signal lines.
[0153] Memory 110 is used to store programs or data. Memory 110 can be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM).
[0154] The processor 120 is used to read / write data or programs stored in the memory 110 and execute corresponding functions. For example, when the computer program stored in the memory 110 is executed by the processor 120, the large language model sparsification method disclosed in the above embodiments can be implemented.
[0155] The communication module 130 is used to establish a communication connection between the electronic device 100 and other communication terminals through a network, and to send and receive data through the network.
[0156] It should be understood that Figure 9 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may also include Figure 9 More or fewer components than shown, or with Figure 9 Different configurations shown. Figure 9 Each component shown in the figure can be implemented by hardware, software or a combination thereof.
[0157] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by the processor 120, the large language model sparsification method disclosed in the above embodiments is implemented.
[0158] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.
[0159] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.
[0160] If the functions are implemented as software modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.
[0161] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.
Claims
1. A method for sparseening a large language model, characterized in that: The large language model includes a plurality of network layers, each of which includes an activation function; and the method includes: Split the input text into multiple tokens and generate a feature vector for each token; Using the feature vector as input feature and the first network layer of the large language model as the target network layer; When the target network layer does not have a predictor, using the weight matrix of the target network layer as a sparse weight; When the target network layer has a predictor, the input features are input into the predictor of the target network layer to obtain the importance corresponding to multiple preset sparsities of each input feature, and for each input feature, the preset sparsity corresponding to the importance exceeding the threshold is used as the preliminary sparsity, and the preliminary sparsity with the largest sparsity is used as the candidate sparsity; the number of each candidate sparsity is counted, and the candidate sparsity with the largest number is determined as the target sparsity, and the weight matrix of the target network layer is sparsely calculated according to the mask corresponding to the target sparsity to obtain sparse weights; the mask is used to sparse the weight matrix; The product of each input feature and the sparse weight is input into the activation function of the target network layer, and the output result of the activation function is used as the input feature of the next network layer, and the next network layer is used as the target network layer, and the process is performed layer by layer until the last network layer of the large language model, and the output result of the activation function of the last network layer is used as the output result of the large language model.
2. The method for sparseening a large language model according to claim 1, characterized in that: The step of performing sparse calculation on the weight matrix of the target network layer according to the mask corresponding to the target sparsity to obtain sparse weights includes: The mask corresponding to the target sparsity is multiplied by the weight matrix of the target network layer to obtain the sparse weight.
3. The method for sparseening a large language model according to claim 1, characterized in that: Each network layer is provided with a weight matrix, and the predictor is obtained in the following way: For each network layer that needs to adjust sparsity, a weight importance matrix is obtained according to a plurality of training features and the weight matrix; the weight importance matrix is used to characterize the importance of each weight in the weight matrix; the training features of the first network layer are obtained by extracting features based on the received training text; The training features of the network layers except the first network layer are the output of the previous network layer; According to the weight importance matrix and a plurality of preset sparsities, obtaining a mask corresponding to each of the preset sparsities; Obtaining the theoretical importance of each preset sparsity corresponding to each token vector according to the token vector corresponding to the training feature, the weight matrix and the mask corresponding to each preset sparsity; The fully connected layer to be trained is iteratively trained according to the theoretical importance of each of the token vectors and the corresponding preset sparsity to obtain the predicted importance of each preset sparsity, and the fully connected layer whose predicted importance meets the acceptance condition is used as the predictor.
4. The method for sparseening a large language model according to claim 3, characterized in that: The step of obtaining a weight importance matrix according to the plurality of training features and the weight matrix comprises: Generate a corresponding training matrix according to each of the training features, and generate a three-dimensional training matrix according to all of the training matrices; Generate a three-dimensional weight matrix according to the weight matrix; Taking the absolute values of the three-dimensional training matrix and the three-dimensional weight matrix element by element and performing dot multiplication to obtain an activation matrix; The activation matrix is summed over the number dimension of the training matrix to obtain the weight importance matrix.
5. The method for sparseening a large language model according to claim 3, characterized in that: The step of obtaining a mask corresponding to each preset sparsity according to the weight importance matrix and a plurality of preset sparsities includes: For each of the preset sparsity, when the sparse format is a ratio, the sparse quantity is calculated according to the preset sparsity, and the coordinates of the elements with small weights in the weight importance matrix are obtained according to the sparse quantity; When the sparse format is a ratio, the coordinates of the elements with small weights in the weight importance matrix are obtained in sequence according to the ratio; An all-one matrix is generated according to the weight importance matrix, and the elements corresponding to the element coordinates are set to 0, so as to obtain a mask corresponding to the preset sparsity.
6. The method for sparseening a large language model according to claim 3, characterized in that: The obtaining, according to the token vector corresponding to the training feature, the weight matrix and the mask corresponding to each preset sparsity, the theoretical importance of each preset sparsity corresponding to each token vector comprises: For each of the token vectors, an original importance matrix is obtained according to the token vector and the weight matrix; the token vector of the first network layer is obtained by extracting features from the tokens obtained by splitting the training text; the training features of the network layers other than the first network layer are output after the previous network layer processes the token vector; Obtaining a theoretical importance matrix corresponding to each of the preset sparsities according to the original importance matrix and the masks corresponding to each of the preset sparsities; The theoretical importance of each preset sparsity corresponding to each token vector is obtained according to each of the theoretical importance matrices and the original importance matrix.
7. A large language model sparse device, characterized in that: The large language model includes a plurality of network layers, each of which includes an activation function; the device includes: A preprocessing module, used to split the input text into multiple tokens and generate a feature vector for each token; A sparse module is used to take the feature vector as an input feature and the first network layer of the large language model as a target network layer; when the target network layer does not have a predictor, the weight matrix of the target network layer is used as a sparse weight; when the target network layer has a predictor, the input feature is input into the predictor of the target network layer to obtain the importance corresponding to multiple preset sparsities of each input feature, for each input feature, the preset sparsity corresponding to the importance exceeding the threshold is used as a preliminary sparsity, and the preliminary sparsity with the largest sparsity is used as a candidate sparsity; the number of each candidate sparsity is counted The candidate sparsity with the largest number is determined as the target sparsity, and the weight matrix of the target network layer is sparsely calculated according to the mask corresponding to the target sparsity to obtain the sparse weight; the mask is used to sparsify the weight matrix; the product of each input feature and the sparse weight is input into the activation function of the target network layer, and the output result of the activation function is used as the input feature of the next network layer, and the next network layer is used as the target network layer, and the processing is performed layer by layer until the last network layer of the large language model, and the output result of the activation function of the last network layer is used as the output result of the large language model.
8. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the large language model sparsification method described in any one of claims 1-6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the large language model sparsification method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Network model calculation method, electronic equipment and storage medium
CN114861867A
Vehicle traffic light detection method and device, vehicle and storage medium
CN115861964A
Detection network lightweight method combining progressive guide distillation and structure reconstruction
CN117557857A
