Single network pruning method, device, equipment, medium and product

By pruning the weight matrix of the network layer column by column and updating the Heisen matrix, the problem of inefficiency of the existing single-shot network pruning method is solved, and efficient pruning and optimization of large-scale models is achieved.

CN119026663APending Publication Date: 2024-11-26CHINA MOBILE GROUP ZHEJIANG +3
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411107391.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing single-shot network pruning method is low in efficiency in super-large-scale models, has a large amount of calculation and is difficult to apply, resulting in inefficiency.

Method used

By obtaining the pruning mask matrix corresponding to the weight matrix of the network layer, pruning the columns in the weight matrix column by column, and using the inverse matrix of the Heisen matrix to update the remaining weight information that has not been pruned, reducing the computational complexity of the algorithm.

Benefits of technology

It effectively improves the efficiency of single-shot network pruning, reduces the cost of computing resources and time, and is suitable for large-scale model optimization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119026663A_ABST
    Figure CN119026663A_ABST
Patent Text Reader

Abstract

The invention provides a single network pruning method and device, equipment, a medium and a product. The method comprises the following steps: acquiring a pruning mask matrix corresponding to a weight matrix in a single network layer; determining a first column in the weight matrix as a current pruning column; pruning the weight of the current pruning column based on the pruning mask matrix; based on the trimmed weight in the current pruning column, determining the weight to be updated of the column to be updated in the weight matrix; updating the weight to be updated based on the inverse matrix of the Hessian matrix; determining the next column of the current pruning column as the current pruning column, and iteratively executing the steps of weight pruning and weight updating until the current pruning column is the penultimate column; and based on the pruning mask matrix, pruning the weight of the last column in the weight matrix to obtain the weight matrix after the network layer is pruned. According to the single-time network pruning method provided by the invention, the calculation complexity of an algorithm is reduced, so that the cost of calculation resources and time is reduced, and the efficiency of single-time network pruning is effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a single network pruning method, device, equipment, medium and product. Background Art

[0002] Pruning is a popular model compression technology that can reduce the size or complexity of the model by removing unnecessary or redundant components, thus achieving model lightweighting. Existing pruning methods can be mainly divided into structured pruning and unstructured pruning. The main difference between the two lies in the pruning target and the resulting network structure.

[0003] Structured pruning removes connections or hierarchical structures based on specific rules while retaining the overall network structure; unstructured pruning replaces weight values ​​that meet certain pruning conditions with zero without reconstructing the entire neural network. Both methods are effective in vision and smaller-scale models, but they can lead to a loss of accuracy, requiring a lot of retraining of the model to restore accuracy. Although there are some single-shot network pruning algorithms that can compress models without retraining, these single-shot network pruning algorithms are too computationally intensive to be applied to very large-scale models, resulting in low efficiency of single-shot network pruning. Summary of the invention

[0004] The present invention provides a single network pruning method, device, equipment, medium and product, which are used to solve the problem of low efficiency of the single network pruning method in the prior art.

[0005] In a first aspect, the present invention provides a single network pruning method, comprising: Get the pruning mask matrix corresponding to the weight matrix in a single network layer; Determine the first column in the weight matrix as the current pruning column; Based on the pruning mask matrix, pruning the weight of the current pruning column; Based on the pruned weights in the current pruned column, determine the weights to be updated of the columns to be updated in the weight matrix; the columns to be updated refer to the columns after the current pruned column in the weight matrix; the weights to be updated refer to the weights belonging to the same row as the pruned weights in the current pruned column; Based on the inverse matrix of the Hessian matrix, the weights to be updated are updated; Determine the next column of the current pruned column as the current pruned column, and iteratively perform the step of pruning the weight of the current pruned column based on the pruning mask matrix until the current pruned column is the second to last column; Based on the pruning mask matrix, the weights in the last column of the weight matrix are pruned to obtain the pruned weight matrix of the network layer.

[0006] In one embodiment, the updating of the weights to be updated based on the inverse matrix of the Hessian matrix includes: Generate a Hessian matrix based on the input vector of the network layer; Calculate the inverse matrix of the Hessian matrix; Determining a weight vector of the current pruned column; Based on the inverse matrix of the Hessian matrix and the weight vector of the current pruned column, the weights to be updated are updated to obtain a current updated weight matrix.

[0007] In one embodiment, the weight to be updated is updated based on the inverse matrix of the Hessian matrix and the weight vector of the current pruned column to obtain the current updated weight matrix. The specific calculation formula is as follows:

[0008] in, Represents the currently updated weight matrix; A weight vector representing the current pruned column; The element at the first row and first column of the inverse matrix of the Hessian matrix; The vector representing the first row of the inverse of the Hessian matrix.

[0009] In one embodiment, obtaining a pruning mask matrix corresponding to a weight matrix in a single network layer includes: Calculate the importance of each weight in the weight matrix to obtain multiple weight importances; Generating a weight importance matrix according to the plurality of weight importances; Select the weight importances of the top N largest values ​​in each column of the weight importance matrix; The mask of the weight importance of the first N maximum values ​​in each column is determined to be 1, and the mask of the remaining weight importance in the column is determined to be 0; the remaining weight importance refers to the remaining weight importance in the column except the weight importance of the first N maximum values; A pruning mask matrix is ​​generated according to the mask corresponding to each weight importance in the weight importance matrix.

[0010] In one embodiment, the importance of each weight in the weight matrix is ​​calculated to obtain multiple weight importances. The specific calculation formula is as follows:

[0011] in, Characterize the weight matrix i Linej The weight importance corresponding to the column weight; Represents the absolute value operator; Characterize the weight matrix i Line j Column weights; The first step for evaluating the input vector X to the network layer j characteristic The norm is separated on N×L different tokens, where N is the batch dimension of the input vector X and L is the sequence dimension of the input vector X.

[0012] In one embodiment, pruning the weight of the current pruning column based on the pruning mask matrix includes: Determining the first column of the pruning mask matrix as a current mask column; Resetting the weight of the current pruning column corresponding to the mask of 0 in the current mask column to 0; The weight of the current pruning column corresponding to the mask being 1 in the current mask column remains unchanged.

[0013] In one embodiment, after obtaining the weight matrix after the network layer pruning, the method includes: Based on the pruned weight matrix of the network layer, the model to be optimized where the network layer is located is compressed.

[0014] In a second aspect, the present invention further provides a single network pruning device, comprising: An acquisition module is used to obtain the pruning mask matrix corresponding to the weight matrix in a single network layer; A first determination module, configured to determine the first column in the weight matrix as a current pruning column; A first pruning module, configured to prune the weight of the current pruning column based on the pruning mask matrix; A second determination module is used to determine the weights to be updated of the columns to be updated in the weight matrix based on the pruned weights in the current pruned column; the columns to be updated refer to the columns after the current pruned column in the weight matrix; the weights to be updated refer to the weights belonging to the same row as the pruned weights in the current pruned column; An updating module, used for updating the weights to be updated based on the inverse matrix of the Hessian matrix; an iteration module, configured to determine the next column of the current pruned column as the current pruned column, and iteratively perform a step of pruning the weight of the current pruned column based on the pruning mask matrix until the current pruned column is the second to last column; The second pruning module is used to prune the weights of the last column in the weight matrix based on the pruning mask matrix to obtain the pruned weight matrix of the network layer.

[0015] In a third aspect, the present invention provides a device, comprising an electronic device, wherein the electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the single network pruning method as described in any one of the above are implemented.

[0016] In a fourth aspect, the present invention further provides a medium, comprising a non-transitory computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the single network pruning methods described above are implemented.

[0017] In a fifth aspect, the present invention also provides a product, comprising a computer program product, wherein the computer program product comprises a computer program, wherein the computer program can be stored on a non-transitory computer-readable storage medium, and wherein when the computer program is executed by the processor, the steps of the single network pruning method as described above are implemented.

[0018] The single network pruning method, device, equipment, medium and product provided by the present invention obtain a pruning mask matrix corresponding to a weight matrix in a single network layer, and selectively prune the columns in the weight matrix column by column through the pruning mask matrix, thereby reducing network parameters, and when weight pruning is performed on each column in the weight matrix, the remaining weight information that has not been pruned is updated by reusing the inverse matrix of the Hessian matrix, that is, the inverse matrix of the Hessian matrix is ​​reused to update the weights to be updated in the same row as the pruned weights in the subsequent columns, and finally the weight matrix after network layer pruning is obtained, which can be used for subsequent model optimization. This pruning method reduces the computational complexity of the algorithm, thereby reducing the cost of computing resources and time, and effectively improving the efficiency of single network pruning. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 This is one of the flow charts of the single network pruning method provided by the present invention.

[0021] Figure 2 This is the second flow chart of the single network pruning method provided by the present invention.

[0022] Figure 3 It is a visualized schematic diagram of the weight matrix pruning provided by the present invention.

[0023] Figure 4 It is a schematic diagram of the structural coefficient matrix and compressed representation provided by the present invention.

[0024] Figure 5 It is a schematic diagram of mask generation based on weight importance provided by the present invention.

[0025] Figure 6 It is a structural schematic diagram of a single network pruning device provided by the present invention.

[0026] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein.

[0029] Combine the following Figure 1-Figure 7 The single network pruning method, device, equipment, medium and product provided by the present invention are described.

[0030] Combination Figure 1 and Figure 2 , Figure 1 This is one of the flow charts of the single network pruning method provided by the present invention. Figure 2 This is the second flow chart of the single network pruning method provided by the present invention.

[0031] like Figure 1 As shown, the method includes the following: Step 101, obtaining a pruning mask matrix corresponding to a weight matrix in a single network layer; Step 102: determine the first column in the weight matrix as the current pruning column; Step 103: Pruning the weight of the current pruned column based on the pruning mask matrix; Step 104: Determine the weights to be updated of the columns to be updated in the weight matrix based on the pruned weights in the currently pruned columns; Step 105: updating the weights to be updated based on the inverse matrix of the Hessian matrix; Step 106: determine the next column of the current pruned column as the current pruned column, and iteratively perform the step of pruning the weight of the current pruned column based on the pruning mask matrix until the current pruned column is the second to last column; Step 107: Prune the weights in the last column of the weight matrix based on the pruning mask matrix to obtain the pruned weight matrix of the network layer.

[0032] It should be noted that the single network pruning method provided in the embodiment of the present invention is implemented based on a single network pruning device. Figure 2 As shown, the single network pruning method provided by the embodiment of the present invention first evaluates the importance of a single weight in a single network layer by multiplying the size of the weight by the corresponding input feature norm, then generates a weight importance matrix according to the importance of all weights, and then generates a pruning mask matrix through the weight importance matrix; then, the pruning mask matrix and the inverse matrix of the reused Hession matrix are used to delete the weights column by column and update the remaining weights, which is a fast pruning process. The embodiment of the present invention describes the single network pruning method by taking a single network pruning device as the execution subject as an example.

[0033] Specifically, the single network pruning device determines the network layer that needs to be pruned in the model to be optimized. Multiple network layers in the model to be optimized can be pruned separately, or a single network layer in the model to be optimized can be pruned. The embodiment of the present invention describes the pruning of a single network layer.

[0034] Furthermore, the single network pruning device determines the weight matrix of a single network layer that needs to be pruned in the model to be optimized, wherein each row of the weight matrix corresponds to a neuron in the network layer, and each column of the weight matrix corresponds to the connection weights of all neurons in the previous layer to a neuron in the network layer.

[0035] Furthermore, the single network pruning device evaluates the importance of each weight in the weight matrix to obtain the weight importance of each weight in the weight matrix.

[0036] Furthermore, the single network pruning device determines whether to retain the corresponding weight based on the weight importance of each weight, according to the size of the weight importance, so as to generate a mask corresponding to each weight, and determines that the mask corresponding to the weight is the optimal pruning mask according to the size of the weight importance.

[0037] It should be noted that during the pruning process, the role of the mask (Mask, M) is to record and represent the decisions made during the pruning process. Specifically, the mask M is a binary vector, in which each bit corresponds to a feature or node in the original model, and a feature or node corresponds to a weight. During the pruning process, unimportant features or nodes can be removed by deleting their corresponding weights. By removing some unimportant features or nodes, the complexity of the model can be reduced to simplify the model, thereby improving the efficiency and performance of the model.

[0038] In order to record these pruning decisions, a mask M is used to represent that when a feature or node is removed, the bit of the mask M corresponding to the feature or node will be set to 0; and when a feature or node is retained, the bit of the mask M corresponding to the feature or node will be set to 1.

[0039] Furthermore, the single network pruning device generates a pruning mask matrix corresponding to the weight matrix in the network layer according to the mask corresponding to each weight. Therefore, the pruning mask matrix can be used to perform weight pruning and update the remaining weights of the weight matrix column by column.

[0040] Furthermore, the single network pruning device determines the first column in the weight matrix as the current pruning column.

[0041] Furthermore, the single network pruning device prunes the weight of the current pruned column based on the mask column at the same position as the current pruned column in the pruning mask matrix.

[0042] Furthermore, the single network pruning device determines the weights to be updated of the columns to be updated in the weight matrix based on the pruned weights in the current pruned column, wherein the columns to be updated refer to the columns after the current pruned column in the weight matrix. For example, there are N columns in the weight matrix, and the current pruned column is the first column. Then the columns after the current pruned column, i.e., the columns to be updated, are the second column, the third column, ..., the Nth column; the weights to be updated refer to the weights belonging to the same row as the pruned weights in the current pruned column. There may be multiple pruned weights, and the update of the weights to be updated in the same row for each pruned weight is the same. Therefore, the update of the weights to be updated in the same row for a pruned weight is described.

[0043] Furthermore, the single network pruning device generates an inverse matrix of the Hessian matrix.

[0044] Furthermore, the single network pruning device updates the weights to be updated based on the inverse matrix of the generated Hessian matrix.

[0045] Furthermore, the single network pruning device determines the next column of the current pruned column as the current pruned column, and returns to the step of pruning the weights of the current pruned column based on the pruning mask matrix, and iteratively executes the process of weight pruning and updating the weights to be updated until the current pruned column is the second-to-last column, completing the weight pruning of the second-to-last column and the updating of the weights to be updated.

[0046] Furthermore, since there are no columns to be updated after the last column of the weight matrix, and there are no weights to be updated, only the weights of the last column need to be pruned. Therefore, the single network pruning device prunes the weights of the last column in the weight matrix based on the pruning mask matrix.

[0047] In one embodiment, referring to Figure 3 , Figure 3 It is a visualization diagram of the weight matrix pruning provided by the present invention. Figure 3 As shown in the figure, the pruning mask matrix and the weight matrix (denoted as W in the figure) are both 6×4 in size, and each mask has a one-to-one correspondence with the weight. For ease of description, the weight of the i-th row and j-th column is called W ij For the pruning process of the first column, the gray grid in the first column of the pruning mask matrix represents a mask of 0, then correspondingly, the grids at the same position in the first column of the weight matrix need to be pruned, that is, pruning W 21 , W 31 , W 51 , which is shown as a cross in the figure. Further, we need to 21 The weight to be updated in the same row W 22 , W 23 , W 24 , and W 31 The weight to be updated in the same row W 32 , W 33 , W 34 , and W 51 The weight to be updated in the same row W 52 , W 53 , W 54 The weights to be updated are updated at the same time. The weights to be updated are represented by dark green grids in the figure. The weights in the rows where the unpruned weights are located do not need to be updated. The weights that do not need to be updated are represented by light green grids in the figure. When updating the weights to be updated, the inverse matrix of the Hessian matrix is ​​used The first row vector in is used to update the weights to be updated. The row where the inverse matrix of the Hessian matrix is ​​applied is represented by a dark blue grid. Through the above steps, the first column of weights is pruned and the weights to be updated are updated, and the weight matrix after the first update can be obtained. The weight pruning and the update of the weights to be updated are further implemented column by column in the above manner until the penultimate column is reached. The weight matrix after the second update and the weight matrix after the third update can be obtained respectively. Since the last column does not need to update the weights to be updated, only the last column needs to be pruned. Finally, the weight matrix after the fourth update is the pruned weight matrix, and the pruned weight matrix can be applied to the model optimization process.

[0048] It should be noted that represents the inverse matrix applied during the first update, represents the inverse matrix applied during the second update, represents the inverse matrix applied during the third update, Represents the inverse matrix applied during the fourth update, and each update applies the inverse matrix of the same Hessian matrix, but uses different row vectors in the inverse matrix for update.

[0049] In summary, it should be noted that the process of weight matrix pruning can be divided into three steps: (a) Calculation of the initial Hessian matrix H, with a time complexity of , where n represents the number of samples of the Hessian matrix H, Represents the square of the length of the column vector in the Hessian matrix H; (b) The calculation of the inverse matrix of the Hessian matrix has a time complexity of ,in, The cube of the length of the column vector in the Hessian matrix H represents; (c) The iterative weight update process has a total time complexity of ,in, Characterizes the length of the column vector in the Hessian matrix H, Represents the length of the row vector in the Hessian matrix H. For a model with a Transformer structure, its overall time complexity can be simplified to ,in, The cube of the number of samples in the hidden layer is represented. The overall time complexity is reduced by an order of magnitude compared to existing methods, which makes it practical even on large models such as GPT.

[0050] Optionally, the single-shot network pruning method provided by the present invention can also be easily adapted to heterogeneous networks, such as the popular n:m sparse format, which can provide computational acceleration on the NVIDIA GPU (NVIDIA Graphics Processing Unit) of the Ampere architecture. Specifically, each m consecutive weights should contain n exact zeros. Therefore, this method can simply select the size of each mask block to be m, and then enforce zero constraints in the mask selection of each row by selecting n weights that result in the minimum error based on the weight importance. Similar strategies can also be applied to other semi-structured pruning modes.

[0051] Reference Figure 4 , Figure 4 NVIDIA-A100 GPU adds support for fine-grained structure sparsity to its tensor core. The sparse tensor core accelerates the 2:4 sparsity mode. In each continuous block of four values, two values ​​must be zero, which naturally leads to 50% sparsity. Figure 4 As shown, no vector or block structures are pruned together. Under this rule, the model is easy to compress and has low metadata overhead. Sparse Tensor Cores accelerate this format by operating only on non-zero values ​​in the compressed matrix. They use metadata stored with non-zeros to extract only necessary values ​​from another uncompressed operand. Therefore, for 2 times the sparsity, the same effective calculation can be completed in half the time.

[0052] Therefore, through the above-mentioned one-time pruning strategy, a model with more than 100 billion parameters can be run on a single microprocessor within a few hours without retraining, and it is accurate enough to prune the model to a sparsity level of 50%-60% without significantly reducing performance. Moreover, this method can be extended to semi-structured (2:4 and 4:8) modes, using hardware for inference acceleration, and can be compatible with existing weight quantization methods.

[0053] The single network pruning method provided by the present invention obtains the pruning mask matrix corresponding to the weight matrix in a single network layer, and selectively prunes the columns in the weight matrix column by column through the pruning mask matrix, thereby reducing the network parameters, and when weight pruning is performed on each column in the weight matrix, the remaining weight information that has not been pruned is updated by reusing the inverse matrix of the Hessian matrix, that is, the inverse matrix of the Hessian matrix is ​​reused to update the weights to be updated in the same row as the pruned weights in the subsequent columns, and finally the weight matrix after network layer pruning is obtained, which can be used for subsequent model optimization. This pruning method reduces the computational complexity of the algorithm, thereby reducing the cost of computing resources and time, and effectively improving the efficiency of single network pruning.

[0054] Further, based on step 101, obtaining a pruning mask matrix corresponding to a weight matrix in a single network layer includes: Calculate the importance of each weight in the weight matrix to obtain multiple weight importances; Generating a weight importance matrix according to the plurality of weight importances; Select the weight importances of the top N maximum values ​​in each column of the weight importance matrix; The mask of the weight importance of the first N maximum values ​​in each column is determined to be 1, and the mask of the remaining weight importance in the column is determined to be 0; the remaining weight importance refers to the remaining weight importance in the column except the weight importance of the first N maximum values; A pruning mask matrix is ​​generated according to the mask corresponding to each weight importance in the weight importance matrix.

[0055] It should be noted that updating weights during pruning can lead to significant changes in importance, and more accurate results will be obtained if this effect can be taken into account during the mask selection stage. For example, consider a neuron with two inputs and corresponding weights , where the weight , the standard pruning method generally chooses to delete the weight If the input and With the same magnitude, this pruning strategy is fine. However, for large models, the two inputs may differ significantly in size, e.g. ,at this time , in this case, it is obvious to remove the weight It will be better than removing weight The impact on the output of the entire model is smaller.

[0056] The above examples illustrate that general pruning methods or pruning methods suitable for small models do not take into account the input activation values, which may play an equally important role as the weight size in determining the output of neurons. For example, it is particularly important to prune large language models (LLMs) considering the large amplitude features that appear inside them. Therefore, as the first part of this method, we propose to design a pruning metric for models with large amplitude features and form a pruning mask to deal with this limitation while maintaining the simplicity of amplitude pruning.

[0057] Specifically, the single network pruning device calculates the importance of each weight in the weight matrix to obtain multiple weight importances.

[0058] In one embodiment, consider a shape The fully connected layer with weight W. For the LLM model, this fully connected layer accepts an input vector X with a shape of where N and L are the batch dimension and the sequence dimension respectively. For each individual weight, its importance degree is evaluated by the product of its magnitude and the norm of the corresponding input feature. The specific calculation formula is as follows:

[0059] where, represents the weight importance corresponding to the weight in the i th row and j th column of the weight matrix; represents the absolute value operator; represents the weight in the i th row and j th column of the weight matrix; is used to evaluate the j th feature of the input vector X input to the network layer, and the norm is separated over N×L different tokens, where N is the batch dimension of the input vector X and L is the sequence dimension of the input vector X.

[0060] Furthermore, the single-shot network pruning device generates a weight importance matrix based on multiple weight importances.

[0061] Furthermore, the single-shot network pruning device compares the weight importances in each column of the weight importance matrix to obtain the comparison result for each column in the weight importance matrix.

[0062] Furthermore, the single-shot network pruning device selects the top N maximum weight importances in each column of the weight importance matrix according to the comparison result for each column in the weight importance matrix, where N can be set according to the actual situation.

[0063] Furthermore, the single-shot network pruning device sets the mask of the top N maximum weight importances in each column to 1 and sets the mask of the remaining weight importances in each column to 0.

[0064] Furthermore, the single-shot network pruning device generates a pruning mask matrix according to the mask corresponding to each weight importance in the weight importance matrix.

[0065] Specifically, if there are K weight importances greater than 0 in a certain column and K < N, then only the masks of these K weight importances are set to 1 in this column, and the remaining weight importances in this column are all 0, and the corresponding masks are all 0.

[0066] In one embodiment, referring to Figure 5 , Figure 5The weight matrix W has the weight values ​​of 3, 4, -4, -2 in the first column, and the absolute values ​​are 3, 4, 4, 2. The element of the first column of is 1, and the two are multiplied to obtain the weight importance of the first column of the weight importance matrix, which is 3, 4, 4, and 2. The weight importances of the first two maximum values, i.e., 4 and 4, are selected, and the masks of these two weight importances are determined to be 1, and the masks of the other two weight importances are determined to be 0. According to the above method, the weight importance of each column can be calculated, and its corresponding mask can be determined, and finally the pruning mask matrix corresponding to the weight matrix can be obtained.

[0067] It should be noted that when the corresponding input of the weight under consideration has a relatively large amplitude feature, the weight itself tends to be assigned a larger weight importance, rather than potentially having a low amplitude, which solves the problem that large models usually have relatively large amplitude input features. Figure 5 As shown in the figure, the weights corresponding to the large-amplitude features will not be set to 0 in the pruning mask and will be retained. Secondly, the calculation of this method is relatively simple. Once the norm vector of the input feature activation is obtained, the element-by-element dot product can be used to calculate the weight importance.

[0068] The embodiment of the present invention takes into account that the model may have a large-amplitude feature and its corresponding weight value is small, which may cause the weight to be deleted. The importance of the weight is evaluated by the product of the numerical size of the individual weight and the norm of the corresponding input feature, and whether to delete it is judged according to its importance to generate a corresponding optimal mask, thereby improving the reliability and accuracy of the generation of the pruning mask matrix, and then the subsequent pruning based on the pruning mask matrix can be more accurate, so that the model optimized according to the pruned weight matrix can ensure the accuracy of model detection when the model sparsity is 50%.

[0069] Further, based on step 103, pruning the weight of the current pruning column based on the pruning mask matrix includes: Determining the first column of the pruning mask matrix as a current mask column; Resetting the weight of the current pruning column corresponding to the mask of 0 in the current mask column to 0; The weight of the current pruning column corresponding to the mask being 1 in the current mask column remains unchanged.

[0070] Specifically, the single network pruning device determines the first column of the pruning mask matrix as the current mask column.

[0071] Furthermore, the single network pruning device resets the weight of the current pruning column corresponding to the mask of 0 in the current mask column to 0, which can also be considered as deleting the weight corresponding to the mask of 0.

[0072] Furthermore, the single network pruning device keeps the weight of the current pruning column corresponding to the mask of 1 in the current mask column unchanged.

[0073] Therefore, through the above steps, each column of weights in the weight matrix can be pruned according to each column mask in the pruning mask matrix.

[0074] The embodiment of the present invention achieves pruning of some weights in the neural network by resetting the weights corresponding to the positions with 0 in the mask column to 0, thereby reducing the complexity and computational complexity of the model while maintaining the performance and accuracy of the network.

[0075] Further, based on step 105, the weights to be updated are updated based on the inverse matrix of the Hessian matrix, including: Generate a Hessian matrix based on the input vector of the network layer; Calculate the inverse matrix of the Hessian matrix; Determining a weight vector of the current pruned column; Based on the inverse matrix of the Hessian matrix and the weight vector of the current pruned column, the weights to be updated are updated to obtain a current updated weight matrix.

[0076] Specifically, the single network pruning device determines an input vector input to the network layer, and generates a Hessian matrix based on the input vector.

[0077] The calculation formula of the Hessian matrix is ​​as follows:

[0078] Where H represents the Hessian matrix; X represents the input vector; A vector representing the transposed input vector.

[0079] Furthermore, the single network pruning device inverts the Hessian matrix to obtain an inverse matrix of the Hessian matrix.

[0080] Furthermore, the single network pruning device determines the weight vector of the current pruned column.

[0081] Furthermore, the single network pruning device updates the weights to be updated based on the inverse matrix of the Hessian matrix and the weight vector of the current pruned column to obtain the current updated weight matrix.

[0082] Among them, based on the inverse matrix of the Hessian matrix and the weight vector of the current pruned column, the weight to be updated is updated to obtain the current updated weight matrix. The specific calculation formula is as follows:

[0083] in, Represents the currently updated weight matrix, that is, the weight matrix after the first update; A weight vector representing the current pruned column; The element at the first row and first column of the inverse matrix of the Hessian matrix; The vector representing the first row of the inverse of the Hessian matrix.

[0084] It can be understood that when each column is pruned, there will be a corresponding update of the weight to be updated, and the updated weight matrix will be obtained. Therefore, each time the weight to be updated is updated, the updated weight matrix is ​​obtained. The specific calculation formula can be extended to:

[0085] Among them, m is used to represent the mth weight update when performing the mth column weight pruning. Represents the weight matrix after the mth update, providing the optimal adjustment of the weights to be updated to compensate for the error caused by pruning at m; A weight vector representing the mth column in the weight matrix; The element at the mth row and mth column position of the inverse matrix of the Hessian matrix; The vector representing the mth row of the inverse matrix of the Hessian matrix.

[0086] The embodiment of the present invention updates the weights to be updated in each pruning process by reusing the inverse matrix of the Hessian matrix, which can effectively update the weight matrix and reduce the consumption of computing resources. The final weight matrix after network layer pruning can be used for subsequent model optimization to improve the performance and efficiency of the model.

[0087] Further, after obtaining the weight matrix after the network layer pruning, it includes: Based on the pruned weight matrix of the network layer, the model to be optimized where the network layer is located is compressed.

[0088] Specifically, the single network pruning device compresses the model to be optimized where the network layer is located based on the weight matrix after the network layer is pruned.

[0089] Optionally, the single network pruning device obtains weight matrices of multiple network layers after being pruned respectively, and further, based on the weight matrices of multiple network layers after being pruned respectively, compresses the models to be optimized where the multiple network layers are located.

[0090] It should be noted that the single network pruning process of the present invention is preferably used for the optimization and improvement of large models with large-scale features, especially LLM models. Model compression technology can meet the current growing demand for efficient and convenient models, because model compression technology can reduce the size of the model, improve the efficiency of the model, while maintaining or even improving the performance of the model. In various fields such as cloud computing, edge computing, and mobile devices, there is a strong demand for model compression. In the field of cloud computing, compression can reduce storage and computing costs and improve service efficiency. In the field of edge computing, due to limited device resources, model compression can help achieve more efficient computing and reasoning. In the field of mobile devices, model compression can make artificial intelligence applications more lightweight and improve user experience. In addition, autonomous driving, medical image analysis, recommendation systems, etc. also need to use model compression technology to improve performance.

[0091] The embodiment of the present invention compresses the model to be optimized in the network layer through the weight matrix after network layer pruning, which can save the storage and computing resources required for the model and reduce costs. By optimizing the model, the performance of the product or service can be improved and the user experience can be enhanced.

[0092] The single network pruning device provided by the present invention is described below. The single network pruning device described below and the single network pruning method described above can be referred to each other.

[0093] Reference Figure 6 , Figure 6 It is a structural schematic diagram of a single network pruning device provided by the present invention.

[0094] The single network pruning device comprises: The acquisition module 610 is used to obtain the pruning mask matrix corresponding to the weight matrix in a single network layer.

[0095] The first determining module 620 is configured to determine the first column in the weight matrix as a current pruning column.

[0096] The first pruning module 630 is configured to prune the weight of the current pruning column based on the pruning mask matrix.

[0097] The second determination module 640 is used to determine the weights to be updated of the columns to be updated in the weight matrix based on the pruned weights in the current pruned column; the columns to be updated refer to the columns after the current pruned column in the weight matrix; the weights to be updated refer to the weights belonging to the same row as the pruned weights in the current pruned column.

[0098] The updating module 650 is used to update the weights to be updated based on the inverse matrix of the Hessian matrix.

[0099] The iteration module 660 is used to determine the next column of the current pruned column as the current pruned column, and iteratively perform the step of pruning the weight of the current pruned column based on the pruning mask matrix until the current pruned column is the second to last column.

[0100] The second pruning module 670 is used to prune the weights of the last column in the weight matrix based on the pruning mask matrix to obtain the pruned weight matrix of the network layer.

[0101] The single network pruning device provided by the present invention obtains the pruning mask matrix corresponding to the weight matrix in a single network layer, and selectively prunes the columns in the weight matrix column by column through the pruning mask matrix, thereby reducing the network parameters, and when weight pruning is performed on each column in the weight matrix, the remaining weight information that has not been pruned is updated by reusing the inverse matrix of the Hessian matrix, that is, the inverse matrix of the Hessian matrix is ​​reused to update the weights to be updated in the same row as the pruned weights in the subsequent columns, and finally the weight matrix after network layer pruning is obtained, which can be used for subsequent model optimization. This pruning method reduces the computational complexity of the algorithm, thereby reducing the cost of computing resources and time, and effectively improving the efficiency of single network pruning.

[0102] Furthermore, the acquisition module 610 is also used for: Calculate the importance of each weight in the weight matrix to obtain multiple weight importances; Generating a weight importance matrix according to the plurality of weight importances; Select the weight importances of the top N maximum values ​​in each column of the weight importance matrix; The mask of the weight importance of the first N maximum values ​​in each column is determined to be 1, and the mask of the remaining weight importance in the column is determined to be 0; the remaining weight importance refers to the remaining weight importance in the column except the weight importance of the first N maximum values; A pruning mask matrix is ​​generated according to the mask corresponding to each weight importance in the weight importance matrix.

[0103] Furthermore, the first pruning module 630 is further configured to: Determining the first column of the pruning mask matrix as a current mask column; Resetting the weight of the current pruning column corresponding to the mask of 0 in the current mask column to 0; The weight of the current pruning column corresponding to the mask being 1 in the current mask column remains unchanged.

[0104] Furthermore, the updating module 650 is further configured to: Generate a Hessian matrix based on the input vector of the network layer; Calculate the inverse matrix of the Hessian matrix; Determining a weight vector of the current pruned column; Based on the inverse matrix of the Hessian matrix and the weight vector of the current pruned column, the weights to be updated are updated to obtain a current updated weight matrix.

[0105] Furthermore, the single network pruning device is also used for: Based on the pruned weight matrix of the network layer, the model to be optimized where the network layer is located is compressed.

[0106] It should be noted that the single network pruning device provided by the present invention can execute the single network pruning method described in any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0107] Figure 7 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 7 As shown, the electronic device may include: a processor (processor) 710 , a communication interface (Communications Interface) 720 , a memory (memory) 730 and a communication bus 740 , wherein the processor 710 , the communication interface 720 , and the memory 730 communicate with each other through the communication bus 740 . The processor 710 can call the logic instructions in the memory 730 to execute a single network pruning method, which includes: obtaining a pruning mask matrix corresponding to a weight matrix in a single network layer; determining the first column in the weight matrix as the current pruning column; pruning the weight of the current pruning column based on the pruning mask matrix; determining the weight to be updated of the column to be updated in the weight matrix based on the pruned weight in the current pruning column; the column to be updated refers to the column after the current pruning column in the weight matrix; the weight to be updated refers to the weight belonging to the same row as the pruned weight in the current pruning column; updating the weight to be updated based on the inverse matrix of the Hessian matrix; determining the next column of the current pruning column as the current pruning column, and iteratively executing the step of pruning the weight of the current pruning column based on the pruning mask matrix until the current pruning column is the second to last column; pruning the weight of the last column in the weight matrix based on the pruning mask matrix to obtain the weight matrix after the network layer is pruned.

[0108] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0109] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the single network pruning method provided by the above embodiments, and the method includes: obtaining a pruning mask matrix corresponding to a weight matrix in a single network layer; determining the first column in the weight matrix as a current pruning column; based on the pruning mask matrix, pruning the weight of the current pruning column; based on the pruned weights in the current pruning column, determining the columns to be updated in the weight matrix The weights to be updated are: the column to be updated refers to the column after the current pruned column in the weight matrix; the weights to be updated refer to the weights belonging to the same row as the pruned weights in the current pruned column; based on the inverse matrix of the Hessian matrix, the weights to be updated are updated; the next column of the current pruned column is determined as the current pruned column, and the step of pruning the weights of the current pruned column is iteratively performed based on the pruning mask matrix until the current pruned column is the second to last column; based on the pruning mask matrix, the weights of the last column in the weight matrix are pruned to obtain the weight matrix after the network layer is pruned.

[0110] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the single network pruning method provided in the above embodiments, the method comprising: obtaining a pruning mask matrix corresponding to a weight matrix in a single network layer; determining the first column in the weight matrix as a current pruning column; based on the pruning mask matrix, pruning the weight of the current pruning column; determining the weight to be updated of the column to be updated in the weight matrix based on the pruned weight in the current pruning column; the column to be updated refers to the weight matrix. The columns after the current pruned column in the weight matrix; the weights to be updated refer to the weights belonging to the same row as the pruned weights in the current pruned column; based on the inverse matrix of the Hessian matrix, the weights to be updated are updated; the next column of the current pruned column is determined as the current pruned column, and the step of pruning the weights of the current pruned column is iteratively performed based on the pruning mask matrix until the current pruned column is the second to last column; based on the pruning mask matrix, the weights of the last column in the weight matrix are pruned to obtain the weight matrix after the network layer is pruned.

[0111] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A single network pruning method, It is characterized in that include: Get the pruning mask matrix corresponding to the weight matrix in a single network layer; Determine the first column in the weight matrix as the current pruning column; Based on the pruning mask matrix, pruning the weight of the current pruning column; Based on the pruned weights in the current pruned column, determine the weights to be updated of the columns to be updated in the weight matrix; the columns to be updated refer to the columns after the current pruned column in the weight matrix; the weights to be updated refer to the weights belonging to the same row as the pruned weights in the current pruned column; Based on the inverse matrix of the Hessian matrix, the weights to be updated are updated; Determine the next column of the current pruned column as the current pruned column, and iteratively perform the step of pruning the weight of the current pruned column based on the pruning mask matrix until the current pruned column is the second to last column; Based on the pruning mask matrix, the weights in the last column of the weight matrix are pruned to obtain the pruned weight matrix of the network layer.

2. The single network pruning method according to claim 1, It is characterized in that The updating of the weights to be updated based on the inverse matrix of the Hessian matrix comprises: Generate a Hessian matrix based on the input vector of the network layer; Calculate the inverse matrix of the Hessian matrix; Determining a weight vector of the current pruned column; Based on the inverse matrix of the Hessian matrix and the weight vector of the current pruned column, the weights to be updated are updated to obtain a current updated weight matrix.

3. The single network pruning method according to claim 2, It is characterized in that Based on the inverse matrix of the Hessian matrix and the weight vector of the current pruned column, the weight to be updated is updated to obtain the current updated weight matrix. The specific calculation formula is as follows: in, Represents the currently updated weight matrix; A weight vector representing the current pruned column; The element at the first row and first column of the inverse matrix of the Hessian matrix; The vector representing the first row of the inverse of the Hessian matrix.

4. The single network pruning method according to claim 1, It is characterized in that The obtaining of the pruning mask matrix corresponding to the weight matrix in a single network layer includes: Calculate the importance of each weight in the weight matrix to obtain multiple weight importances; Generating a weight importance matrix according to the plurality of weight importances; Select the weight importances of the top N maximum values ​​in each column of the weight importance matrix; The mask of the weight importance of the first N maximum values ​​in each column is determined to be 1, and the mask of the remaining weight importance in the column is determined to be 0; the remaining weight importance refers to the remaining weight importance in the column except the weight importance of the first N maximum values; A pruning mask matrix is ​​generated according to the mask corresponding to each weight importance in the weight importance matrix.

5. The single network pruning method according to claim 4, It is characterized in that The importance of each weight in the weight matrix is ​​calculated to obtain multiple weight importances. The specific calculation formula is as follows: in, Characterize the weight matrix i Line j The weight importance corresponding to the column weight; Represents the absolute value operator; Characterize the weight matrix i Line j Column weights; The first step for evaluating the input vector X to the network layer j characteristic The norm is separated on N×L different tokens, where N is the batch dimension of the input vector X and L is the sequence dimension of the input vector X.

6. The single network pruning method according to claim 1, It is characterized in that The step of pruning the weight of the current pruning column based on the pruning mask matrix includes: Determining the first column of the pruning mask matrix as a current mask column; Resetting the weight of the current pruning column corresponding to the mask of 0 in the current mask column to 0; The weight of the current pruning column corresponding to the mask being 1 in the current mask column remains unchanged.

7. The single network pruning method according to any one of claims 1 to 6, It is characterized in that After obtaining the weight matrix of the pruned network layer, it includes: Based on the pruned weight matrix of the network layer, the model to be optimized where the network layer is located is compressed.

8. A single network pruning device, It is characterized in that include: An acquisition module is used to obtain the pruning mask matrix corresponding to the weight matrix in a single network layer; A first determination module, configured to determine the first column in the weight matrix as a current pruning column; A first pruning module, configured to prune the weight of the current pruning column based on the pruning mask matrix; A second determination module is used to determine the weights to be updated of the columns to be updated in the weight matrix based on the pruned weights in the current pruned column; the columns to be updated refer to the columns after the current pruned column in the weight matrix; the weights to be updated refer to the weights belonging to the same row as the pruned weights in the current pruned column; An updating module, used for updating the weights to be updated based on the inverse matrix of the Hessian matrix; an iteration module, configured to determine the next column of the current pruned column as the current pruned column, and iteratively perform a step of pruning the weight of the current pruned column based on the pruning mask matrix until the current pruned column is the second to last column; The second pruning module is used to prune the weights of the last column in the weight matrix based on the pruning mask matrix to obtain the pruned weight matrix of the network layer.

9. A device, comprising an electronic device, the electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the single network pruning method according to any one of claims 1 to 7 are implemented.

10. A medium comprising a non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the single network pruning method according to any one of claims 1 to 7 are implemented.

11. A product, the product comprising a computer program product, the computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the single network pruning method according to any one of claims 1 to 7 are implemented.