Neural network sparsity methods, systems, devices, and media

Abstract

A neural network sparse method, by acquiring the input and output similarity of the attention module and the multi-layer perception module in each Transformer layer; according to the model target sparsity, the module preset quota ratio and the input and output similarity of the attention module and the multi-layer perception module in each Transformer layer, the module target sparsity of the attention module and the multi-layer perception module in each Transformer layer is obtained; according to the module target sparsity of the attention module and the multi-layer perception module in each Transformer layer, the global pruning template of each weight matrix of the attention module and the multi-layer perception module in each Transformer layer is generated; according to the corresponding global pruning template, the weight pruning and reconstruction of the weight matrix of the attention module and the multi-layer perception module in each Transformer layer are carried out. By once weight pruning, the large language model is compressed to a high sparse state, without retraining, and excellent performance can be maintained. Compared with the sparseGPT method, the accuracy of the benchmark test performance is more excellent in high sparsity.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a sparse neural network method, system, device, and medium. Background Technology

[0002] With the rapid development of deep learning technology, large language models (LLMs) have demonstrated remarkable achievements in fields such as natural language processing (NLP). Large language models such as LLaMA, Palm, GLM, BLOOM, and GPT have achieved outstanding performance in many NLP tasks, including text generation, machine translation, question answering systems, text summarization, and sentiment analysis.

[0003] Currently popular Large Language Models (LLMs) have large dimensionality in their weight matrices. For example, in LLaMA 3.1 8b, the V weight matrix of an attention module has a dimension of 4k x 4k. To reduce storage and computational costs for easier deployment, large language models can be lightweighted by pruning (or sparsifying) them.

[0004] Existing pruning techniques for large language models still have limitations in their application. They either cannot achieve a sufficient compression ratio or cannot guarantee the model's performance after pruning. This is especially true when dealing with the complex architecture and massive parameters of large language models, where it is difficult to find a satisfactory balance between resource requirements and performance maintenance.

[0005] Therefore, a new pruning method suitable for large language models is needed, which is an urgent problem to be solved in the practical application and promotion of current large language models. Summary of the Invention

[0006] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a neural network sparse method, system, device and medium that can achieve a high compression ratio (effectively reducing the storage and computing costs of the model) while maintaining or even not significantly impairing the original performance of the model, thereby finding a satisfactory balance between resource requirements and performance preservation, which facilitates the promotion and miniaturized deployment of large language models.

[0007] To achieve the above and other related objectives, this application provides a sparse neural network method, wherein the neural network includes several stacked Transformer layers, and each Transformer layer includes an attention module and a multilayer perception module;

[0008] The method includes:

[0009] Calibration data is injected into the neural network to obtain the input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer;

[0010] Based on the input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer, obtain the input-output similarity of the attention module and the multilayer perception module in each Transformer layer;

[0011] Based on the model target sparsity, the module preset quota ratio, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer, the module target sparsity of the attention module and the multilayer perception module in each Transformer layer is obtained.

[0012] Based on the module target sparsity of the attention module and the multilayer perception module in each Transformer layer, a global pruning template for each weight matrix of the attention module and the multilayer perception module in each Transformer layer is generated.

[0013] Based on the global pruning template of each weight matrix of the attention module and the multilayer perception module in each Transformer layer, the weight matrices of the attention module and the multilayer perception module in each Transformer layer are pruned and reconstructed to update each weight matrix in the attention module and the multilayer perception module in each Transformer layer.

[0014] In an optional embodiment of this application, the input-output similarity of the attention module and the multilayer perception module in each Transformer layer is obtained based on the input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer, including:

[0015] Based on the input and output feature data of the attention module and the multilayer perception module in each Transformer layer, obtain the final output feature data of the attention module and the multilayer perception module in each Transformer layer;

[0016] Based on the input feature data and final output feature data of the attention module and the multilayer perception module in each Transformer layer, the input-output similarity of the attention module and the multilayer perception module in each Transformer layer is obtained.

[0017] In an optional embodiment of this application, the input-output similarity of the attention module and the multilayer perception module in each Transformer layer is obtained based on the input feature data and final output feature data of the attention module and the multilayer perception module in each Transformer layer, including:

[0018] Based on the input feature data and final output feature data of the attention module and the multilayer perception module in each Transformer layer, a preset similarity algorithm is used to obtain the input-output similarity of the attention module and the multilayer perception module in each Transformer layer;

[0019] The preset similarity algorithms include cosine similarity algorithm, Euclidean similarity algorithm, Manhattan similarity algorithm, and Pearson correlation coefficient algorithm.

[0020] In an optional embodiment of this application, the module target sparsity of the attention module and the multilayer perception module in each Transformer layer is obtained based on the model target sparsity, the module preset quota ratio, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer, including:

[0021] Based on the model target sparsity and the module preset quota ratio, the sparsity quota of the attention module and the multilayer perception module is calculated and obtained.

[0022] Based on the sparse quotas of the attention module and the multilayer perception module, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer, the module target sparsity of the attention module and the multilayer perception module in each Transformer layer is obtained.

[0023] In an optional embodiment of this application, the module target sparsity of the attention module and the multilayer perception module in each Transformer layer is obtained based on the sparsity quota of the attention module and the multilayer perception module, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer, including:

[0024] Based on the preset amplification parameters, the sparsity quotas of the attention module and the multilayer perception module, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer, the target module sparsity of the attention module and the multilayer perception module in each Transformer layer is obtained.

[0025] In an optional embodiment of this application, the target module sparsity T of the attention module and the multilayer perception module in each Transformer layer is calculated using the following formula:

[0026]

[0027] In the formula, S represents the input-output similarity of the current attention module or multilayer perception module; turbo represents the preset amplification parameter; Mean(S) is the average input-output similarity of the attention module or multilayer perception module in all Transformer layers; and base is the sparse quota of the attention module or multilayer perception module.

[0028] In one alternative embodiment of this application, turbo is between 1.0 and 1.5.

[0029] In an optional embodiment of this application, a global pruning template for each weight matrix of the attention module and the multilayer perception module in each Transformer layer is generated based on the module target sparsity of the attention module and the multilayer perception module in each Transformer layer, including:

[0030] Obtain the input correlation Hesse matrix of the target weight matrix, wherein the target weight matrix is ​​any weight matrix of the attention module or the multilayer perception module;

[0031] Based on the input related Hessian matrix of the target weight matrix, a Hessian inverse matrix is ​​generated, wherein the lower half of the Hessian inverse matrix is ​​a matrix with zeros.

[0032] The error introduced by setting each weight in the target weight matrix to zero is calculated based on the Hesse inverse matrix.

[0033] Based on the error introduced by resetting each weight to zero in the target weight matrix, and the module target sparsity corresponding to the target weight matrix, a global pruning template for the target weight matrix is ​​generated.

[0034] In an optional embodiment of this application, a global pruning template for the target weight matrix is ​​generated based on the error introduced by resetting each weight to zero in the target weight matrix and the target sparsity of the corresponding modules, including:

[0035] The weights of the target weight matrix are sorted according to the error introduced by resetting each weight to zero in the target weight matrix;

[0036] The number of weights to be pruned is determined based on the module target sparsity corresponding to the target weight matrix and the total number of weights in the target weight matrix;

[0037] The weights to be clipped in the target weight matrix are determined based on the order of the weights in the target weight matrix and the number of weights to be clipped.

[0038] Generate a global clipping template for the target weight matrix based on the weights that need to be clipped in the target weight matrix.

[0039] In an optional embodiment of this application, weight pruning and reconstruction are performed on the weight matrices of the attention module and the multilayer perception module in each Transformer layer according to the global pruning template of each weight matrix of the attention module and the multilayer perception module in each Transformer layer, so as to update each weight matrix in the attention module and the multilayer perception module in each Transformer layer, including:

[0040] Based on the global pruning template and Hesse inverse matrix of each weight matrix of the attention module and the multilayer perception module in each Transformer layer, the weight matrices of the attention module and the multilayer perception module in each Transformer layer are pruned column by column, and the weights of the remaining pruned columns are reconstructed using the OBS method to update each weight matrix of the attention module and the multilayer perception module in each Transformer layer.

[0041] To achieve the above and other related objectives, this application provides a neural network sparsification system, wherein the neural network includes several stacked Transformer layers, and each Transformer layer includes an attention module and a multilayer perception module;

[0042] The system includes:

[0043] The data acquisition module is used to inject calibration data into the neural network to obtain the input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer;

[0044] The similarity calculation module is used to obtain the input-output similarity of the attention module and the multilayer perception module in each Transformer layer based on the input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer.

[0045] The sparsity calculation module is used to obtain the module target sparsity of the attention module and the multilayer perception module in each Transformer layer based on the model target sparsity, the module preset quota ratio, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer.

[0046] The template acquisition module is used to generate a global pruning template for each weight matrix of the attention module and the multilayer perception module in each Transformer layer based on the module target sparsity of the attention module and the multilayer perception module in each Transformer layer.

[0047] The pruning and reconstruction module is used to prune and reconstruct the weight matrices of the attention module and the multilayer perception module in each Transformer layer according to the global pruning template of each weight matrix of the attention module and the multilayer perception module in each Transformer layer, so as to update each weight matrix in the attention module and the multilayer perception module in each Transformer layer.

[0048] To achieve the above and other related objectives, this application provides an electronic device, the electronic device comprising:

[0049] One or more processors;

[0050] A memory for storing one or more programs that, when executed by one or more processors, enable the electronic device to implement the aforementioned neural network sparse method.

[0051] To achieve the above and other related objectives, this application provides a computer-readable storage medium storing a computer program that, when executed by a computer's processor, causes the computer to perform the aforementioned neural network sparse method.

[0052] As described above, the neural network sparse method, system, device, and medium of this application have the following beneficial effects:

[0053] By injecting calibration data into the neural network, input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer are obtained; based on the input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer, the input-output similarity of the attention module and the multilayer perception module in each Transformer layer is obtained; based on the model target sparsity, the module preset quota ratio, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer, the module target sparsity of the attention module and the multilayer perception module in each Transformer layer is obtained; based on each Based on the module target sparsity of the attention module and the multilayer perception module in each Transformer layer, a global pruning template is generated for each weight matrix of the attention module and the multilayer perception module in each Transformer layer. According to the global pruning template, the weight matrices of the attention module and the multilayer perception module in each Transformer layer are pruned and reconstructed, thereby compressing a large language model to a highly sparse state through a single weight pruning, without retraining, while maintaining excellent performance. Compared with the sparseGPT method, it exhibits superior accuracy in benchmark tests with high sparsity. Attached Figure Description

[0054] Figure 1 The diagram shown is a flowchart illustrating the neural network sparse method provided in an embodiment of this application.

[0055] Figure 2 The diagram shows the similarity signal location calculated by a Transformer layer, including the attention module (points A and B) and the multilayer perception module (points B and C).

[0056] Figure 3 The diagram shows the similarity between the attention modules and multilayer perceptron modules of each Transformer layer in LLaMA 3.1 8B.

[0057] Figure 4 The diagram shown is a structural block diagram of a neural network sparse system provided in an embodiment of this application.

[0058] Figure 5 The diagram shown is a structural schematic of an electronic device according to an embodiment of this application. Detailed Implementation

[0059] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0060] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0061] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0062] This application provides a neural network sparse method called shrinkGPT, which is a post-training pruning method specifically tailored for large language models with the Transformer architecture. The shrinkGPT method of this application can compress to a highly sparse state through a single weight pruning, without retraining, and can maintain excellent performance. Compared with the sparseGPT method, it has better accuracy performance in benchmark tests with high sparsity.

[0063] Figure 1 A flowchart illustrating a neural network sparse method in an exemplary embodiment of this application is shown, including steps S10-S50. The following will be combined with... Figure 1 The technical solution of this application will be described in detail below.

[0064] First, step S10 is executed to inject calibration data into the neural network to obtain the input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer.

[0065] In this application, the neural network is a large language model (LLM) built on the Transformer architecture, which includes several stacked Transformer layers. Each Transformer layer includes an attention module with residual connections and a multilayer perception module. The large language model may include, for example, large models such as LLaMA, Palm, GLM, BLOOM, and GPT.

[0066] In this application, the calibration data is batch calibration data, including multiple calibration data. When the calibration data is input into the neural network model, the model first tokenizes each calibration data into consecutive tokens through a word segmenter, and then performs word embedding operation on each token to map each token into a high-dimensional word vector in a high-dimensional vector space before inputting it into the model for processing. During the processing, the input feature data and output feature data corresponding to each token of each attention module and each multilayer perception module can be obtained.

[0067] Next, step S20 is executed, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer is obtained based on the input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer.

[0068] In one specific embodiment, when obtaining the input-output similarity of the attention module and the multilayer perception module in each Transformer layer based on the input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer, the final output feature data of the attention module and the multilayer perception module in each Transformer layer can be obtained first based on the input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer; then, the input-output similarity of the attention module and the multilayer perception module in each Transformer layer can be obtained based on the input feature data and the final output feature data of the attention module and the multilayer perception module in each Transformer layer.

[0069] The following explanation uses the input-output similarity calculation process of the attention module as an example:

[0070] When calculating the input-output similarity of the attention module, the input feature vector data and output feature vector data for each token of the attention module can be obtained first. The input feature vector data for each token is then superimposed onto the output feature vector data to obtain the final output feature vector data. A preset similarity algorithm is then used to calculate the similarity between the input feature vector data and the final output feature vector data, which is taken as the similarity of the attention module for each token. The average similarity of the attention module for each token of each calibration data is then calculated to obtain the similarity of the attention module for each calibration data. Finally, the average similarity of the attention module for each calibration data is calculated to obtain the input-output similarity of the attention module. The preset similarity algorithm includes cosine similarity algorithm, Euclidean similarity algorithm, Manhattan similarity algorithm, and Pearson correlation coefficient algorithm. It should be noted that since the input-output similarity calculation of the attention module and the multi-layer attention module is similar, it will not be elaborated upon here.

[0071] Taking the cosine similarity algorithm as an example, the similarity calculation formula for each token by the attention module or multilayer perception module is as follows:

[0072]

[0073] In the formula, S represents the cosine similarity, and P1 and P2 are the input feature vector data and the final output feature vector data for each token, respectively. Figure 2 As shown, when calculating the similarity of the attention module for each token, P1 and P2 are taken from the signals at points A and B; when calculating the similarity of the multilayer perceptron module for each token, P1 and P2 are taken from the signals at points B and C. |P1| and |P2| represent the Euclidean norm (i.e., the length of the vectors) of vectors P1 and P2, respectively.

[0074] Figure 3 The image shows similarity plots between the attention module (attn curves in the figure) and the multilayer perception module (mlp curves in the figure) of the 32 Transformer layers of the LLaMA 3.1 8B model. Figure 3 As can be seen, in the LLaMA 3.1 8B model, the input-output similarity between the attention modules and the multilayer perception modules of the Transformer layers at both ends is low, while the input-output similarity between the attention modules and the multilayer perception modules of the middle Transformer layers is high. Furthermore, the closer to the tail end, the higher the similarity between the attention modules and the multilayer perception modules of the middle Transformer layers.

[0075] Next, step S30 is executed to obtain the module target sparsity of the attention module and the multilayer perception module in each Transformer layer based on the model target sparsity, the module preset quota ratio, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer.

[0076] Specifically, the sparsity quotas of the attention module and the multilayer perception module can be calculated and obtained first based on the model target sparsity and the module preset quota ratio; then, the module target sparsity of the attention module and the multilayer perception module in each Transformer layer can be obtained based on the sparsity quotas of the attention module and the multilayer perception module, as well as the input-output similarity of the attention module and the multilayer perception module in each Transformer layer.

[0077] In this application, the preset quota ratios for the modules are alpha for the attention module and 1-alpha for the multilayer perception module. By introducing alpha, the sparsity quotas for the attention module and the multilayer perception module can be adjusted, thus providing a flexible resource allocation strategy for model sparsity. This allows for targeted pruning based on module redundancy, helping to achieve more efficient model compression and optimization while maintaining model performance. It should be noted that since the attention module has higher redundancy than the multilayer perception module, alpha is typically between 0.6 and 0.7, such as 0.6, 0.61, 0.62, 0.63, 0.64, 0.65, 0.66, 0.67, 0.68, 0.69, 0.7, or other values.

[0078] The sparse quotas of the attention module and the multilayer perception module are calculated using the following formula:

[0079]

[0080]

[0081] In the formula, This represents the sparse quota of the attention module. This represents the sparse quota of the multilayer sensing module. This indicates the sparsity of the model objective.

[0082] To further refine the pruning process and achieve more effective model compression while maintaining model performance, a preset amplification parameter can be introduced. Based on the preset amplification parameter, the sparsity quota of the attention module and the multilayer perception module, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer, the target module sparsity of the attention module and the multilayer perception module in each Transformer layer can be obtained.

[0083] Specifically, the sparsity T of the target module in each of the attention modules and the multilayer perception modules in each Transformer layer can be calculated using the following formula:

[0084]

[0085] In the formula, S represents the input-output similarity of the current attention module or multilayer perception module; turbo represents the preset amplification parameter, which is between 1.0 and 1.5, such as 1.0, 1.1, 1.2, 1.3, 1.4, and 1.5; Mean(S) is the average input-output similarity of the attention module or multilayer perception module in all Transformer layers of the model; and base is the sparse quota of the attention module or multilayer perception module.

[0086] By introducing a preset amplification parameter, the weight matrices of attention modules and multilayer perception modules with high input-output similarity can be more specifically pruned, as they may contain more redundant information. Conversely, the weight matrices of attention modules and multilayer perception modules with low input-output similarity can be pruned less, as they may contain more unique or important information. Excessive pruning may have a greater impact on model performance. This application can amplify the difference between the input-output similarity and the mean similarity of attention modules and multilayer perception modules by introducing a preset amplification parameter, making this difference more obvious. This allows for a clearer distinction between which modules' weight matrices should be pruned more or less, making the pruning more targeted and helping to achieve more effective model compression while ensuring model performance.

[0087] Next, step S40 is executed, generating a global pruning template for each weight matrix of the attention module and the multilayer perception module in each Transformer layer based on the module target sparsity. It should be noted that the global pruning template remains unchanged for each weight matrix in subsequent weight pruning processes.

[0088] In one specific embodiment of this application, a global pruning template for each weight matrix of the attention module and the multilayer perception module in each Transformer layer is generated based on the module target sparsity of the attention module and the multilayer perception module in each Transformer layer, which may further include steps S41-S44.

[0089] In step S41, the input correlation Hessian matrix H (also known as the Hessian matrix or Hessian matrix) of the target weight matrix W is obtained. The target weight matrix W is any weight matrix of the attention module or the multilayer perception module.

[0090] It should be noted that each attention module includes four weight matrices: the Q weight matrix of the Q processing unit, the K weight matrix of the K processing unit, the V weight matrix of the V processing unit, and the O weight matrix of the O processing unit. These correspond to each other, where the Q weight matrix is ​​also called the query weight matrix, the K weight matrix is ​​also called the key weight matrix, the V weight matrix is ​​also called the output weight matrix, and the O weight matrix is ​​also called the output weight matrix. The multilayer perception module includes three weight matrices: the up weight matrix of the up processing unit, the gate weight matrix of the gate processing unit, and the down weight matrix of the down processing unit. The up weight matrix is ​​also called the upsampling weight matrix or the up-dimensional weight matrix, the gate weight matrix is ​​also called the gate weight matrix or the control weight matrix, and the down weight matrix is ​​the downsampling weight matrix or the down-dimensional weight matrix.

[0091] When obtaining the input correlation Hessian matrix H of the target weight matrix W, after injecting calibration data into the neural network in step S10, the input feature data X of the processing unit corresponding to the target weight matrix W can be obtained, and the input correlation Hessian matrix H of the target weight matrix W can be constructed according to the following formula based on the input feature data:

[0092] H = (XX T + λI)

[0093] In the formula, X represents the input feature data, X T λI represents the transpose of the input feature data, I represents the identity matrix, λ is a scalar, and λI is a diagonal matrix whose diagonal elements are λ and the rest are 0. Adding λI is a regularization technique that can prevent the matrix from being singular or close to singular during optimization.

[0094] In step S42, the Hessian inverse matrix H is generated based on the input correlation Hessian matrix H of the target weight matrix W. -1 Wherein, the Hessian inverse matrix H -1A matrix whose lower half is zero. Specifically, the Hessian inverse matrix H of the Hessian matrix H can be generated through Cholesky decomposition. -1 And ensure that the Hessian inverse matrix H -1 The lower half is 0, so that the weights contained in the already clipped columns are frozen during the subsequent reconstruction process after weight clipping, ensuring that the structure of the weight matrix is ​​not destroyed.

[0095] In step S43, according to the Hessian inverse matrix H -1 The error introduced by setting each weight in the target weight matrix W to zero is calculated using the following formula:

[0096]

[0097] In the formula, This represents the error introduced by resetting the q-th weight of the weight matrix to zero; Indicates a weighted index. This represents the q-th weight in the weight matrix; This represents the Hessian inverse of the input-correlated Hessian matrix of the weight matrix.

[0098] In step S44, a global pruning template M for the target weight matrix W is generated based on the error introduced by resetting each weight to zero in the target weight matrix W and the module target sparsity corresponding to the target weight matrix W. It should be noted that the module target sparsity corresponding to the target weight matrix W refers to the module target sparsity of the attention module or multilayer perception module to which the target weight matrix W belongs; in other words, the weight matrix of each attention module or multilayer perception module shares the module target sparsity.

[0099] In an optional embodiment of this application, when generating a global pruning template for the target weight matrix based on the error introduced by resetting each weight to zero in the target weight matrix and the target sparsity of the corresponding module in the target weight matrix, the weights of the target weight matrix can be sorted first according to the error introduced by resetting each weight to zero in the target weight matrix; then, the number of weights to be pruned is determined according to the target sparsity of the corresponding module in the target weight matrix and the total number of weights in the target weight matrix; next, the weights to be pruned in the target weight matrix are determined according to the sorting of the weights in the target weight matrix and the number of weights to be pruned; finally, a global pruning template for the target weight matrix is ​​generated according to the weights to be pruned in the target weight matrix. The generated global pruning template can be used to guide the pruning of the target weight matrix. After resetting a large number of unimportant weights to zero, only non-zero weights and their position information need to be saved when storing the model, which greatly reduces the storage space of the model.

[0100] As an example, if the weights of the target weight matrix are sorted in ascending order of error, the weights at the top of the sort that are equal to the number of weights to be clipped can be selected as the weights to be clipped in the target weight matrix to generate a matrix as a global clipping template. This matrix has the same size as the target weight matrix and can be formed by setting the weights to be clipped to 0 and the weights not to be clipped to 1.

[0101] It should be noted that by considering the error introduced by setting each weight to zero, we can accurately determine which weights have a smaller impact on the model's performance. Setting the positions corresponding to these weights to zero in the global pruning template can remove redundant information in the model without significantly reducing its accuracy, making the model more lightweight and thus improving its running speed and efficiency.

[0102] It should be noted that since the module objective sparsity reflects the importance of the module in the whole model and the degree of influence on the output, the global pruning template generated by combining the module objective sparsity can ensure that modules that are important to the model performance are not over-pruned during the pruning process. This can optimize the model structure while maintaining the model's accuracy and generalization ability to the greatest extent.

[0103] Finally, step S50 is executed, in which the weight matrices of the attention module and the multilayer perception module in each Transformer layer are pruned and reconstructed according to the global pruning template of each weight matrix of the attention module and the multilayer perception module in each Transformer layer, so as to update each weight matrix of the attention module and the multilayer perception module in each Transformer layer.

[0104] In one specific embodiment of this application, the weight matrices of the attention module and the multilayer perception module in each Transformer layer can be pruned column by column according to the global pruning template and Hesse inverse matrix of each weight matrix of each weight matrix of the attention module and the multilayer perception module in each Transformer layer. The weights of the remaining unpruned columns are then reconstructed using the classic OBS method to update each weight matrix of the attention module and the multilayer perception module in each Transformer layer.

[0105] The weights of the remaining pruned columns are reconstructed and updated using the classic OBS (Optimal Brain Surgeon) method, with the default formula being:

[0106]

[0107] In the formula, This represents the compensation value for the weights; Indicates a weighted index. This represents the q-th weight in the weight matrix; This represents the Hess inverse of the input-related Hessian matrix of the weight matrix; express The corresponding unit vector.

[0108] In one specific embodiment of this application, the pseudocode for the shrinkGPT algorithm is as follows:

[0109] / / Inject calibration data, and calculate S layer by layer according to attention and multi-layer perception modules (based on formula 4).

[0110] S<-- (A_batch, B_batch, C_batch )

[0111] / / Calculate T based on alpha, S, turbo, and t.

[0112] T <-- (alpha, S, turbo, t)

[0113] / / Begin cropping layer by layer

[0114] for i = 0, 1, 2, ... do

[0115] / / Perform Cholesky decomposition on the inverse matrix of H, with the lower half consisting entirely of 0s.

[0116] H_chole<-- Cholesky(H -1 )

[0117] / / Calculate the global clipping template M

[0118] M <-- (W, T, H_chole )

[0119] / / Perform column-by-column trimming and reconstruction on matrix W.

[0120] for j = 0, 1, 2, ... do

[0121] / / Get column vector

[0122] w<-- W[:, j]

[0123] / / Press M to crop

[0124] q<-- (M, w)

[0125] / / Restructure the columns that follow.

[0126] W<-- (w, q, H_chole)

[0127] End for

[0128] End for

[0129] The pruning performance tests of the shrinkGPT method of this application and the existing sparseGPT method are shown in Tables 1 and 2. Tables 1 and 2 are the benchmark test results of shrinkGPT and sparseGPT on LLaMA3.1 8B with 40% and 50% model objective sparsity, respectively, using GSM8K and TruthfulQA benchmarks.

[0130] Table 1.1 llama3.1 8B 40% pruning baseline test

[0131]

[0132] Table 2.1 8B 50% Pruning Benchmark Test

[0133]

[0134] Note: 1. sparseGPT uses the C4 dataset, with 64 samples and a block size of 1024;

[0135] 2. shrinkGPT uses alpha = 0.65, C4 dataset, and turbo = 1.0;

[0136] 3. GSM8K uses 60 questions for testing, with CoT enabled, and n_shots=3;

[0137] 4. TruthfulQA health: 55 multiple choice questions; TruthfulQA history: 24 multiple choice questions.

[0138] 5. The values ​​in column Avg are calculated using a weighted average.

[0139] In benchmark tests, the shrinkGPT method proposed in this application exhibits superior performance at high sparsity compared to the sparseGPT method. Moreover, the benchmark performance of the shrinkGPT method becomes increasingly outstanding as the sparsity continues to increase.

[0140] Based on the same concept, such as Figure 4 As shown, this application also provides a neural network sparse system 11, which includes a data acquisition module 111, a similarity calculation module 112, a sparsity calculation module 113, a template acquisition module 114, and a pruning and reconstruction module 115.

[0141] The data acquisition module 111 is used to inject calibration data into the neural network to obtain the input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer.

[0142] The similarity calculation module 112 is used to obtain the input-output similarity of the attention module and the multilayer perception module in each Transformer layer based on the input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer;

[0143] The sparsity calculation module 113 is used to obtain the module target sparsity of the attention module and the multilayer perception module in each Transformer layer based on the model target sparsity, the module preset quota ratio, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer.

[0144] Template acquisition module 114 is used to generate a global pruning template for each weight matrix of the attention module and the multilayer perception module in each Transformer layer based on the module target sparsity of the attention module and the multilayer perception module in each Transformer layer.

[0145] The pruning and reconstruction module 115 is used to prune and reconstruct the weight matrices of the attention module and the multilayer perception module in each Transformer layer according to the global pruning template of each weight matrix of the attention module and the multilayer perception module in each Transformer layer, so as to update each weight matrix in the attention module and the multilayer perception module in each Transformer layer.

[0146] It should be noted that the neural network sparse system 11 provided in the above embodiments and the neural network sparse method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the neural network sparse system 11 provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation.

[0147] like Figure 5 The diagram shown is a schematic representation of an electronic device that implements the sparse neural network method of this application.

[0148] The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program, such as a neural network sparse program, stored in the memory 12 and executable on the processor 13.

[0149] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 12 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 1. Furthermore, the memory 12 can include both internal and external storage units of the electronic device 1. The memory 12 can be used not only to store application software and various types of data installed on the electronic device 1, such as sparse neural network code, but also to temporarily store data that has been output or will be output.

[0150] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the electronic device 1, connecting various components of the electronic device 1 through various interfaces and lines. It executes programs or modules (such as neural network sparse programs) stored in the memory 12 and calls data stored in the memory 12 to perform various functions of the electronic device 1 and process data.

[0151] The processor 13 executes the operating system of the electronic device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the above-described neural network sparse method.

[0152] For example, the computer program may be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a data acquisition module 111, a similarity calculation module 112, a sparsity calculation module 113, a template acquisition module 114, and a pruning and reconstruction module 115.

[0153] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium, which can be non-volatile or volatile. The software functional module stored in the storage medium includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute some functions of the neural network sparse method described in the various embodiments of this application.

[0154] In summary, the neural network sparse method, system, device, and medium disclosed in this application obtains the input and output feature data of the attention module and the multilayer perception module in each Transformer layer by injecting calibration data into the neural network; obtains the input-output similarity of the attention module and the multilayer perception module in each Transformer layer based on the input and output feature data of the attention module and the multilayer perception module in each Transformer layer; and obtains the input and output feature data of the attention module and the multilayer perception module in each Transformer layer based on the model target sparsity, the module preset quota ratio, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer. The module target sparsity of the perception module is determined. Based on the module target sparsity of the attention module and the multi-layer perception module in each Transformer layer, a global pruning template for each weight matrix of the attention module and the multi-layer perception module in each Transformer layer is generated. Based on the global pruning template for each weight matrix of the attention module and the multi-layer perception module in each Transformer layer, the weight matrices of the attention module and the multi-layer perception module in each Transformer layer are pruned and reconstructed. This achieves the compression of a large language model to a highly sparse state through a single weight pruning, without retraining, while maintaining excellent performance. Compared with the sparseGPT method, it exhibits superior accuracy in benchmark tests with high sparsity. Therefore, this application effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0155] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A sparse neural network method, characterized in that, The neural network is a large language model trained in the fields of text generation, machine translation, question answering system, text summarization, and sentiment analysis. It includes several stacked Transformer layers, each of which includes an attention module and a multilayer perception module. The method includes: Calibration data is injected into the neural network to obtain input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer, wherein the calibration data includes a natural language dataset; Based on the input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer, obtain the input-output similarity of the attention module and the multilayer perception module in each Transformer layer; Based on the model target sparsity, the module preset quota ratio, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer, the module target sparsity of the attention module and the multilayer perception module in each Transformer layer is obtained. Based on the module target sparsity of the attention module and the multilayer perception module in each Transformer layer, a global pruning template for each weight matrix of the attention module and the multilayer perception module in each Transformer layer is generated. Based on the global pruning template of each weight matrix of the attention module and the multilayer perception module in each Transformer layer, the weight matrices of the attention module and the multilayer perception module in each Transformer layer are pruned and reconstructed to obtain a lightweight big oracle model that is easy to deploy.

2. The neural network sparse method according to claim 1, characterized in that, Based on the input and output feature data of the attention module and the multilayer perception module in each Transformer layer, the input-output similarity of the attention module and the multilayer perception module in each Transformer layer is obtained, including: Based on the input and output feature data of the attention module and the multilayer perception module in each Transformer layer, obtain the final output feature data of the attention module and the multilayer perception module in each Transformer layer; Based on the input feature data and final output feature data of the attention module and the multilayer perception module in each Transformer layer, the input-output similarity of the attention module and the multilayer perception module in each Transformer layer is obtained.

3. The neural network sparse method according to claim 2, characterized in that, Based on the input feature data and final output feature data of the attention module and the multilayer perception module in each Transformer layer, the input-output similarity of the attention module and the multilayer perception module in each Transformer layer is obtained, including: Based on the input feature data and final output feature data of the attention module and the multilayer perception module in each Transformer layer, a preset similarity algorithm is used to obtain the input-output similarity of the attention module and the multilayer perception module in each Transformer layer; The preset similarity algorithms include cosine similarity algorithm, Euclidean similarity algorithm, Manhattan similarity algorithm, and Pearson correlation coefficient algorithm.

4. The neural network sparse method according to claim 1, characterized in that, Based on the model's target sparsity, the module's preset quota ratio, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer, the module's target sparsity is obtained, including: Based on the model target sparsity and the module preset quota ratio, the sparsity quota of the attention module and the multilayer perception module is calculated and obtained. Based on the sparse quotas of the attention module and the multilayer perception module, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer, the module target sparsity of the attention module and the multilayer perception module in each Transformer layer is obtained.

5. The neural network sparse method according to claim 4, characterized in that, Based on the sparse quotas of the attention module and the multilayer perception module, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer, the module target sparsity of the attention module and the multilayer perception module in each Transformer layer is obtained, including: Based on the preset amplification parameters, the sparsity quotas of the attention module and the multilayer perception module, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer, the target module sparsity of the attention module and the multilayer perception module in each Transformer layer is obtained.

6. The neural network sparse method according to claim 5, characterized in that, The sparsity T of the target module in each Transformer layer's attention module and multilayer perception module is calculated using the following formula: In the formula, S represents the input-output similarity of the current attention module or multilayer perception module; turbo represents the preset amplification parameter; Mean(S) is the average input-output similarity of the attention module or multilayer perception module in all Transformer layers; and base is the sparse quota of the attention module or multilayer perception module.

7. The neural network sparse method according to claim 1, characterized in that, Based on the module target sparsity of the attention module and the multilayer perception module in each Transformer layer, a global pruning template for each weight matrix of the attention module and the multilayer perception module in each Transformer layer is generated, including: Obtain the input correlation Hesse matrix of the target weight matrix, wherein the target weight matrix is ​​any weight matrix of the attention module or the multilayer perception module; Based on the input related Hessian matrix of the target weight matrix, a Hessian inverse matrix is ​​generated, wherein the lower half of the Hessian inverse matrix is ​​a matrix with zeros. The error introduced by setting each weight in the target weight matrix to zero is calculated based on the Hesse inverse matrix. Based on the error introduced by resetting each weight to zero in the target weight matrix, and the module target sparsity corresponding to the target weight matrix, a global pruning template for the target weight matrix is ​​generated.

8. The neural network sparse method according to claim 7, characterized in that, Based on the error introduced by resetting each weight to zero in the target weight matrix, and the module target sparsity corresponding to the target weight matrix, a global pruning template for the target weight matrix is ​​generated, including: The weights of the target weight matrix are sorted according to the error introduced by resetting each weight to zero in the target weight matrix; The number of weights to be pruned is determined based on the module target sparsity corresponding to the target weight matrix and the total number of weights in the target weight matrix; The weights to be clipped in the target weight matrix are determined based on the order of the weights in the target weight matrix and the number of weights to be clipped. Generate a global clipping template for the target weight matrix based on the weights that need to be clipped in the target weight matrix.

9. The neural network sparse method according to claim 7, characterized in that, Based on the global pruning template of each weight matrix of the attention module and the multilayer perception module in each Transformer layer, the weight matrices of the attention module and the multilayer perception module in each Transformer layer are pruned and reconstructed, including: Based on the global pruning template and Hesse inverse matrix of each weight matrix of the attention module and the multilayer perception module in each Transformer layer, the weight matrices of the attention module and the multilayer perception module in each Transformer layer are pruned column by column, and the weights of the remaining pruned columns are reconstructed using the OBS method.

10. A sparse neural network system, characterized in that, The neural network is a large language model trained in the fields of text generation, machine translation, question answering system, text summarization, and sentiment analysis. It includes several stacked Transformer layers, each of which includes an attention module and a multilayer perception module. The system includes: The data acquisition module is used to inject calibration data into the neural network to obtain the input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer. The calibration data includes a natural language dataset. The similarity calculation module is used to obtain the input-output similarity of the attention module and the multilayer perception module in each Transformer layer based on the input feature data and output feature data of the attention module and the multilayer perception module in each Transformer layer. The sparsity calculation module is used to obtain the module target sparsity of the attention module and the multilayer perception module in each Transformer layer based on the model target sparsity, the module preset quota ratio, and the input-output similarity of the attention module and the multilayer perception module in each Transformer layer. The template acquisition module is used to generate a global pruning template for each weight matrix of the attention module and the multilayer perception module in each Transformer layer based on the module target sparsity of the attention module and the multilayer perception module in each Transformer layer. The pruning and reconstruction module is used to prune and reconstruct the weight matrices of the attention module and the multilayer perception module in each Transformer layer according to the global pruning template of each weight matrix of the attention module and the multilayer perception module in each Transformer layer, so as to obtain a lightweight big oracle model that is easy to deploy.

11. An electronic device, characterized in that: The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the neural network sparse method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the neural network sparse method according to any one of claims 1 to 9.

Images