Hybrid pruning method and device based on Bert model and storage medium

Abstract

The invention discloses a hybrid pruning method and device based on a Bert model and a storage medium, and relates to the technical field of neural networks, and the hybrid pruning method based on the Bert model comprises the following steps: obtaining a pruning amount of current iteration, the pruning amount being calculated through a remainder compensation strategy according to a preset target pruning parameter; loading a mask matrix generated by (n-1) th iteration, identifying and removing all parameter positions marked as zero parameters, and generating an intermediate matrix only containing non-zero parameters; rearranging the non-zero parameters of the intermediate matrix into a continuously stored dense matrix according to a row priority sequence; the dimension of the dense matrix is adjusted, the row number of the dense matrix is kept unchanged, the column number of the dense matrix is reduced to the target dimension, the dense matrix is remolded into a regularized matrix, and the pruned Bert model is output. According to the method, the technical effect of improving the throughput of the Bert model on the basis of ensuring the model precision is achieved.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a hybrid pruning method, device, and storage medium based on a Bert model. Background Art

[0002] Currently, the Bert model is commonly used for text classification in natural language processing tasks. However, due to the large number of parameters of the Bert model, as well as the high computational complexity of the Bert model's attention mechanism and feedforward neural network, the storage and loading of the Bert model require a large amount of memory resources, resulting in low throughput when the Bert model is deployed in a resource-limited environment.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a hybrid pruning method, device and storage medium based on the Bert model, aiming to solve the technical problem of how to improve the throughput of the Bert model while ensuring the accuracy of the model.

[0005] To achieve the above objectives, the present application proposes a hybrid pruning method based on the Bert model, which is applied to the nth iteration. The hybrid pruning method based on the Bert model includes: Obtaining the pruning amount for the current iteration, where the pruning amount is calculated using a remainder compensation strategy based on preset target pruning parameters; Load the mask matrix generated by iteration n-1, identify and remove all parameter positions marked as zero-valued parameters, and generate an intermediate matrix containing only non-zero parameters; Rearranging the non-zero parameters of the intermediate matrix into a dense matrix stored continuously in row-major order; The dimension of the dense matrix is ​​adjusted so that the number of rows of the dense matrix remains unchanged and the number of columns is reduced to the target dimension to reshape it into a regularized matrix, and the pruned BERT model is output.

[0006] In one embodiment, applied to the 1st to n-1th iterations, before the step of loading the mask matrix generated by the n-1th iteration, identifying and removing all parameter positions marked as zero-valued parameters, and generating an intermediate matrix containing only non-zero parameters, the steps include: Generate an initial mask matrix of corresponding dimensions according to the number of neuron nodes in the Bert model, where all elements in the initial mask matrix are non-zero parameters; The initial mask matrix or the mask matrix generated in the previous iteration is loaded, and based on the importance score of each neuron node and the pruning amount allocated in the current iteration, the corresponding neuron node is marked as a zero-value parameter to generate the updated mask matrix.

[0007] In one embodiment, the target pruning parameters include a target number of retained heads and a total number of iterations; The step of obtaining the pruning amount of the current iteration, wherein the pruning amount is calculated according to the preset target pruning parameter by the remainder compensation strategy, includes: According to the preset target number of retained heads, the total number of self-attention heads to be pruned is calculated, i.e., the total number of heads pruned; Based on the total number of iterations, evenly distribute the total amount of head pruning to each iteration of the Bert model, and obtain the remainder of the evenly distributed number of heads; Calculating the number of head quantity remainder allocation times according to the head quantity remainder, and determining the corresponding number of iterations x based on the number of head quantity remainder allocation times; The remainder of the number of heads is distributed to the first x iterations according to the iteration order, so that an additional head pruning amount is added to each of the first x iterations, and the pruning amount of each iteration of the BERT model is determined.

[0008] In one embodiment, the step of loading the initial mask matrix or the mask matrix generated in the previous iteration, and marking the corresponding neuron nodes as zero-value parameters based on the importance score of each neuron node and the pruning amount allocated in the current iteration to generate the updated mask matrix includes: For the iterative pruning process of the attention layer, load the initial mask matrix or the mask matrix of the attention layer generated in the previous iteration, mark the corresponding number of attention heads based on the importance scores of the attention heads, and update the mask matrix of the attention layer; For the pruning iterative process of the feedforward neural network, the initial mask matrix or the mask matrix of the feedforward neural network generated by the previous iteration is loaded, and based on the back-propagated channel importance scores, the corresponding number of intermediate layer channels are marked to update the mask matrix of the feedforward neural network.

[0009] In one embodiment, the iterative pruning process for the attention layer, loading the initial mask matrix or the mask matrix of the attention layer generated in the previous iteration, marking a corresponding number of the attention heads based on the importance scores of the attention heads, and updating the mask matrix of the attention layer includes the following steps: Input the sample data into the attention layer of the Bert model, and calculate the importance score of each attention head through forward propagation and backpropagation; Based on the importance scores of each of the attention heads from low to high, prune the attention heads corresponding to the pruning amount allocated in the current iteration, and update the initial mask matrix or the mask matrix of the attention layer.

[0010] In one embodiment, the step of inputting sample data into the attention layer of the Bert model and calculating the importance score of each attention head through forward propagation and backpropagation includes: Input the sample data through the attention layer of the Bert model, perform forward propagation, obtain the weight tensor and bias vector of each attention head of the attention layer, and the loss function value of the Bert model; Performing backpropagation based on the loss function value to calculate a weight gradient tensor of the weight tensor and a bias gradient tensor of the bias vector; The importance score of each attention head is obtained by weighted combination of the weight gradient tensor and the bias gradient tensor of each attention head.

[0011] In one embodiment, after the step of inputting the sample data into the attention layer of the Bert model and calculating the importance score of each attention head through forward propagation and backpropagation, the following steps are included: Calculating the minimum absolute value of the weight tensor of each attention head in the attention layer; Based on a preset noise scaling range, randomly generating a noise amplitude scaling factor within the noise scaling range; Generate a noise matrix of the same dimension as the current layer attention score tensor according to the minimum value and the noise amplitude scaling factor; The noise matrix is ​​superimposed on the original importance score to obtain the noise-enhanced importance score.

[0012] In one embodiment, the step of pruning a number of attention heads corresponding to a pruning amount allocated in the current iteration based on the importance scores of the attention heads from low to high, and updating the initial mask matrix or the mask matrix of the attention layer includes: According to the importance scores, all the attention heads in the attention layer are fully connected and sorted from low to high to determine a sorting result; Determine the number of attention heads to be removed in the current iteration based on the pruning amount allocated for the current iteration; According to the sorting result and the number of attention heads, the attention heads to be removed in the current iteration are selected, and the corresponding attention head positions are determined; The initial mask matrix or the mask matrix generated in the previous iteration is loaded, the attention head position is binary marked, and the initial mask matrix or the mask matrix is ​​updated.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a hybrid pruning device based on the Bert model, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the hybrid pruning method based on the Bert model as described above.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the hybrid pruning method based on the Bert model as described above are implemented.

[0015] The present application provides a hybrid pruning method based on the Bert model. The present application first obtains the pruning amount for the current iteration. The pruning amount is calculated based on the preset target pruning parameters through a residual compensation strategy. The mask matrix generated by the n-1th iteration is loaded, and the parameter positions of all parameters marked as zero are identified and removed to generate an intermediate matrix containing only non-zero parameters. The non-zero parameters of the intermediate matrix are rearranged into a dense matrix stored continuously in row-first order. The dimension of the dense matrix is ​​adjusted so that the number of rows of the dense matrix remains unchanged and the number of columns is reduced to the target dimension to reshape it into a regularized matrix, and the pruned Bert model is output. The present application accurately allocates the pruning amount through a residual compensation mechanism, ensuring that the total pruning amount strictly matches the preset target and avoiding under- or over-pruning caused by rounding errors. Through structured pruning, the parameters marked as zero are removed, and the sparse mask matrix generated by unstructured pruning is reshaped into a regularized matrix, improving computational density and memory efficiency, significantly improving model throughput, and maintaining stable accuracy. The present application achieves the technical effect of improving the throughput of the Bert model while ensuring model accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A flowchart of the first embodiment of the hybrid pruning method based on the Bert model of this application is provided; Figure 2A schematic diagram of matrix reshaping provided in Example 1 of the hybrid pruning method based on the Bert model of this application; Figure 3 A flowchart of the second embodiment of the hybrid pruning method based on the Bert model of this application is provided; Figure 4 This is a schematic diagram of unstructured pruning provided in Example 2 of the hybrid pruning method based on the Bert model of this application; Figure 5 A flowchart of the third embodiment of the hybrid pruning method based on the Bert model of this application is provided; Figure 6 A flowchart of the fourth embodiment of the hybrid pruning method based on the Bert model of this application is provided; Figure 7 A flowchart of the fifth embodiment of the hybrid pruning method based on the Bert model of this application is provided; Figure 8 Schematic diagram of the device structure of the hardware operating environment involved in the hybrid pruning method based on the Bert model in the embodiment of the present application.

[0019] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0021] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0022] Currently, the Bert model is commonly used for text classification in natural language processing tasks. However, due to the large number of parameters of the Bert model, as well as the high computational complexity of the Bert model's attention mechanism and feedforward neural network, the storage and loading of the Bert model require a large amount of memory resources, resulting in low throughput when the Bert model is deployed in a resource-limited environment.

[0023] This application uses a remainder compensation mechanism to precisely allocate pruning amounts, ensuring that the total pruning amount strictly matches the preset target and avoiding under- or over-pruning due to rounding errors. Structured pruning removes parameters marked as zero and reshapes the sparse mask matrix generated by unstructured pruning into a regularized matrix, improving computational density and memory efficiency, significantly increasing model throughput while maintaining stable accuracy.

[0024] It should be noted that the execution entity of this embodiment can be a hybrid pruning device based on the Bert model, or a computing service device with data processing, network communication, and program execution capabilities, such as a tablet computer, personal computer, or mobile phone, or a control device for a hybrid pruning device based on the Bert model that can implement the above functions. This embodiment does not specifically limit this. The following describes this embodiment and the following embodiments using a hybrid pruning device based on the Bert model as the execution entity.

[0025] Example 1 Based on this, this application proposes a hybrid pruning method based on the Bert model in the first embodiment, please refer to Figure 1 , the hybrid pruning method based on the Bert model includes: Step S10: obtaining the pruning amount of the current iteration, wherein the pruning amount is calculated according to a preset target pruning parameter through a remainder compensation strategy.

[0026] The precise amount of pruning is obtained in each iteration to ensure that the predetermined pruning target is achieved within the total number of iterations.

[0027] In this embodiment, the preset target pruning parameters are pre-set quantitative indicators of pruning targets, which are used to control the degree of model compression, including the number of neurons, the target retention number, and the total number of iterations. The target retention number is the number of neurons that need to be retained after pruning, and the total amount of pruning that needs to be pruned can be calculated by the target retention number. The total number of iterations is the total number of rounds of the pruning iterative process. The remainder compensation strategy is that when calculating the pruning amount of each iteration, it may be impossible to evenly distribute the pruning amount. The remaining small amount of pruning is reasonably distributed to some iterative steps by means of remainder compensation. The remainder after dividing the total pruning amount by the total number of iterations is preferentially allocated to the first few iterations to ensure that the total pruning amount is accurately achieved.

[0028] As an optional implementation method for calculating the amount of pruning, the original total number of neurons in the Bert model is obtained, and then the number of neurons to be pruned is calculated based on the target retention number, that is, "total pruning amount = original total number of neurons - target retention number". The quotient and remainder after dividing the total pruning amount by the total number of iterations are calculated, and the quotient is used as the basic pruning amount for each iteration. The remainder is then preferentially allocated to the first few iterations to determine the pruning amount for each iteration. The total pruning amount is reasonably distributed to each iteration through the remainder compensation strategy, ensuring that the pruning process meets the expected compression target while maintaining the stability and controllability of the pruning amount, avoiding a sudden drop in model accuracy due to excessive pruning in a single time.

[0029] Step S20 , loading the mask matrix generated by the (n-1)th iteration, identifying and removing all parameter positions marked as zero-valued parameters, and generating an intermediate matrix containing only non-zero parameters.

[0030] In the last iteration of the model pruning process, the mask matrix generated in the n-1th iteration is loaded. By removing the zero-value parameters in the mask matrix, the irregular mask matrix obtained after multiple iterative pruning is converted into a regularized matrix to facilitate efficient storage and fast calculation of the model, thereby achieving the final compression and optimization of the model and improving the deployment efficiency and performance of the model.

[0031] It should be noted that the n-1th iteration process is an unstructured iterative pruning process. The mask matrix is ​​a matrix with the same shape as the model parameter matrix generated during the unstructured pruning process of the Bert model. It is used to indicate which neuron nodes or parameters in the model need to be retained and which need to be pruned. The position with a value of 1 indicates that the corresponding neuron node or parameter is retained, and the position with a value of 0 indicates that the corresponding neuron node or parameter is pruned, that is, marked as a zero-value parameter. The zero-value parameter is a parameter with a value of 0, that is, the parameter corresponding to the neuron node that needs to be pruned. The intermediate matrix is ​​a matrix generated after a bitwise filtering operation, which only contains the non-zero parameters in the original parameter matrix. The bitwise filtering operation is an operation to filter matrix elements according to a condition. The condition is whether the value in the mask matrix is ​​non-zero. The bitwise filtering operation can filter out all non-zero parameters from the original parameter matrix to form a new intermediate matrix.

[0032] For example, obtain the mask matrix generated by the n-1th iteration, traverse each element in the mask matrix, store all parameter positions with a value of 0 using a data structure such as a list, and remove the parameters marked as 0 from the original parameter matrix based on the mask matrix to filter out all non-zero parameters. The filtered non-zero parameters are combined into a new matrix, namely the intermediate matrix.

[0033] Step S30: rearrange the non-zero parameters of the intermediate matrix into a dense matrix that is stored continuously in row-priority order.

[0034] It should be noted that row-major order is a data storage and arrangement method in which elements in a matrix are arranged sequentially according to their rows. A dense matrix is ​​a contiguous matrix formed by rearranging the non-zero parameters in the intermediate matrix in row-major order.

[0035] For example, the non-zero parameters in the intermediate matrix are read sequentially in row-major order and rearranged into a one-dimensional array, and the rearranged one-dimensional array is converted into a dense matrix.

[0036] Step S40, adjusting the dimension of the dense matrix so that the number of rows of the dense matrix remains unchanged and the number of columns is reduced to the target dimension, reshaping it into the regularized matrix, and outputting the pruned Bert model.

[0037] It's important to note that the target dimension is a pre-set value representing the number of columns in the reshaped regularized matrix. The target dimension is determined based on the model's pruning goals and performance requirements. A regularized matrix is ​​a regularly shaped matrix whose dimensions conform to the pre-set target dimension, and whose parameters are arranged in a specific order.

[0038] For example, based on the model's pruning goals and performance requirements, the target dimension (i.e., the number of columns) of the reshaped matrix is ​​determined. The number of columns of the dense matrix is ​​reduced to the target dimension while maintaining the number of rows. The resized matrix is ​​used as the final regularized matrix to replace the original parameter matrix, thereby completing the regularized reshaping of the model. The reshaped regularized matrix is ​​applied to the BERT model, and the pruned model is output for subsequent use.

[0039] For example, please refer to Figure 2 The matrix reshaping diagram shown in the figure uses structured pruning in the nth iteration of iterative pruning. The structured pruning here is different from the traditional structured pruning. The neuron nodes at the corresponding positions are removed by the masked_select method according to the mask matrix, and the remaining non-zero parameters are structured and reshaped. For example, the model is pruned from 3072 dimensions to 1024 dimensions, and the feedforward neural network layer shape is (768, 3072). The feedforward neural network layer shape after pruning is (768, 1024). The masked_select method uses a mask matrix to select the neuron nodes to be retained from the original neuron node matrix. That is, according to the mask matrix, the neuron nodes at the position with a value of 1 in the weight matrix are selected and the neuron nodes at the position with a value of 0 are removed.

[0040] As an example of this embodiment, hybrid pruning is performed on the BERT model. Target pruning parameters, including the total pruning amount, total number of iterations, target number of heads to retain, and target dimension, are first determined. An initial mask matrix is ​​then generated, whose dimensions are the same as the corresponding weight matrix in the BERT model, and all elements are initialized to 1, indicating that all neurons are initially unpruned. The iterative pruning process begins. For each iteration from the 1st to the n-1th iteration, unstructured pruning is employed. The pruning amount for the current iteration is calculated based on the preset target pruning parameters and the residual compensation strategy. The mask matrix generated in the previous iteration is then loaded (the initial mask matrix is ​​loaded for the first iteration). Sample data is input into the BERT model's attention layer, and the importance score of each attention head is calculated through forward and backward propagation. Based on these scores, the attention heads are fully connected and sorted from low to high. The number of attention heads to be removed is determined based on the pruning amount for the current iteration. The corresponding attention heads are selected and their positions are determined. These positions are marked as 0 in the mask matrix, generating an updated mask matrix. In the nth iteration, structured pruning is performed. The mask matrix generated in the n-1th iteration is loaded, and all parameter positions marked as zero are identified and removed, generating an intermediate matrix containing only non-zero parameters. The non-zero parameters of the intermediate matrix are then rearranged in row-major order into a dense matrix stored contiguously. The dimensions of this dense matrix are adjusted so that the number of rows remains the same and the number of columns is reduced to the target dimension, reshaping it into a regularized matrix. Finally, the regularized matrix is ​​applied to the BERT model, replacing the original parameters, completing the final pruning iteration and outputting the pruned BERT model. By combining unstructured and structured pruning, the pruned BERT model achieves a significant reduction in parameter count and computational complexity while retaining important parameters, improving the model's efficiency and deployability.

[0041] This embodiment provides a hybrid pruning method based on the Bert model. It first uses a remainder compensation mechanism to precisely allocate pruning amounts, ensuring that the total pruning amount strictly matches the preset target and avoiding under- or over-pruning due to rounding errors. Structured pruning removes parameters marked as zero and reshapes the sparse mask matrix generated by unstructured pruning into a regularized matrix. This improves computational density and memory efficiency, significantly increasing model throughput while maintaining stable accuracy.

[0042] Based on Example 1, Example 2 of this application proposes a hybrid pruning method based on the Bert model, referring to Figure 3 , before step S20, including: Step S50: Generate an initial mask matrix of corresponding dimensions according to the number of neuron nodes in the Bert model, wherein the values ​​of all elements in the initial mask matrix are non-zero parameters.

[0043] Before pruning begins, an initial mask matrix is ​​generated, where all neurons are not pruned. This provides a foundation for subsequent pruning operations. The dimensions of the mask matrix are the same as the corresponding weight matrix in the BERT model, ensuring that each neuron has a corresponding mask value.

[0044] It should be noted that the number of neuron nodes refers to the total number of neuron nodes in each layer of the BERT model, which determines the dimension of the initial mask matrix. The initial mask matrix is ​​used to mark whether neuron nodes are retained. Initially, all elements are 1, indicating that no neuron nodes have been pruned.

[0045] As an optional implementation, determine the number of neuron nodes in each layer of the BERT model to ensure that the dimensions of the initial mask matrix match the model's weight matrix. Create a matrix with the same shape as the model's weight matrix and initialize all elements to 1 as the initial mask matrix.

[0046] Step S60, loading the initial mask matrix or the mask matrix generated in the previous iteration, marking the corresponding neuron nodes as zero-value parameters based on the importance score of each neuron node and the pruning amount allocated in the current iteration to generate the updated mask matrix.

[0047] During each pruning iteration, the neuron nodes are marked and updated based on the mask matrix generated in the previous iteration, the importance score of the neuron nodes, and the pruning amount of the current iteration, so as to gradually achieve the model pruning goals, optimize the model structure, reduce redundant parameters, and improve model efficiency.

[0048] In this embodiment, neuron nodes are the basic units that make up the BERT model network. They process input data and extract features by connecting and transmitting information. Each neuron node has a corresponding weight. For the attention layer of the BERT model, the neuron node is the attention head; for all linear layers of the feedforward neural network, the neuron node is the neuron node of all linear layers of the feedforward neural network. The importance score is used to measure the importance of the neuron node in the model. It can be evaluated by calculating the weight size, gradient size, and influence of the neuron node on the model output.

[0049] As an optional implementation, at the beginning of each iteration, the mask matrix generated by the previous iteration is loaded. If it is the first iteration, the initial mask matrix is ​​loaded, and the values ​​of all elements in the initial mask matrix are 1, indicating that all neuron nodes have not been pruned. According to the pruning amount assigned to the current iteration, the number of neuron nodes that need to be pruned in this iteration is determined. The importance score of each neuron node in the model is calculated, and all neuron nodes are sorted according to the calculated neuron node importance score. The neuron nodes that need to be pruned are selected in order from low to high scores. The element values ​​corresponding to the selected neuron nodes that need to be pruned in the mask matrix are marked as 0 to generate an updated mask matrix.

[0050] As an optional implementation method for calculating the importance score, a method based on the absolute value of the weight is adopted to calculate the absolute value of the weight of each neuron node, and the absolute value is used as the importance score.

[0051] As an alternative implementation for calculating importance scores, a weight-based and gradient-based approach is employed. During training, the gradient magnitude of each neuron node is recorded and used as the basis for calculating the importance score. The absolute value of each neuron node's weight is calculated, and then the gradient magnitude and the absolute value of the weight are combined to obtain the importance score for each neuron node.

[0052] Optionally, the updated mask matrix is ​​used for the next iteration, and the pruning operation is continued according to the new pruning amount and the importance score of the neuron node until all iterations are completed and the preset pruning target is achieved.

[0053] Optionally, step S60 includes: Step S61, for the pruning iterative process of the attention layer, load the initial mask matrix or the mask matrix of the attention layer generated in the previous iteration, mark the corresponding number of the attention heads based on the importance scores of the attention heads, and update the mask matrix of the attention layer.

[0054] Pruning is performed on the attention layer of the Bert model, and unimportant attention heads are gradually removed by updating the mask matrix, thereby achieving structural optimization and compression of the model and improving the computational efficiency and performance of the model.

[0055] It should be noted that the mask matrix of the attention layer is used to indicate which attention heads need to be retained and which need to be pruned during the BERT model's attention layer pruning process. Positions with a value of 1 indicate that the corresponding attention head is retained, and positions with a value of 0 indicate that the corresponding attention head is pruned, i.e., the corresponding parameters will be marked as zero-valued parameters.

[0056] Exemplarily, during the iterative pruning of the attention layer, the mask matrix of the attention layer generated in the previous iteration is first loaded. If it is the first iteration, the initial mask matrix is ​​loaded, and the values ​​of all elements in the initial mask matrix are 1, indicating that all attention heads have not been pruned. According to the pruning amount assigned to the current iteration, the number of attention heads that need to be pruned in this iteration is determined. The importance score of each attention head in the attention layer is calculated. According to the calculated importance score of the attention head, all attention heads are sorted, and the attention heads that need to be pruned are selected in order from low to high scores. The element value corresponding to the selected attention head that needs to be pruned in the mask matrix is ​​marked as 0 to generate the mask matrix of the updated attention layer.

[0057] Step S62, for the pruning iterative process of the feedforward neural network, load the initial mask matrix or the mask matrix of the feedforward neural network generated by the previous iteration, mark the corresponding number of intermediate layer channels based on the back-propagated channel importance scores, and update the mask matrix of the feedforward neural network.

[0058] Pruning operations are performed on the feedforward neural network of the Bert model. Unimportant intermediate layer channels are gradually removed by updating the mask matrix, thereby achieving structural optimization and compression of the model and improving the computational efficiency and performance of the model.

[0059] It should be noted that the mask matrix of the feedforward neural network is used to indicate which intermediate layer channels need to be retained and which need to be pruned during the feedforward neural network pruning process of the Bert model.

[0060] Exemplarily, during the pruning iteration process of the feedforward neural network, the mask matrix of the feedforward neural network generated by the previous iteration is first loaded. If it is the first iteration, the initial mask matrix is ​​loaded, and the values ​​of all elements in the initial mask matrix are 1, indicating that all intermediate layer channels have not been pruned. According to the pruning amount assigned to the current iteration, the number of intermediate layer channels that need to be pruned in this iteration is determined. For each intermediate layer channel in the feedforward neural network, the importance score is obtained by calculating the absolute value of the gradient of the channel during the back propagation process. According to the calculated channel importance score, all intermediate layer channels are sorted. In order from low to high scores, the intermediate layer channels that need to be pruned are selected. The element values ​​corresponding to the selected intermediate layer channels that need to be pruned in the mask matrix are marked as 0 to generate the updated mask matrix of the feedforward neural network.

[0061] For example, please refer to Figure 4The unstructured pruning diagram shown in the figure uses unstructured pruning in the first N-1 iterations of iterative pruning. The importance score is calculated using the importance score calculation formula. After the attention layer is operated by the importance score calculation formula, the weight of the attention layer is shape (768, 768), which is split into shape (768, 64, 12). The importance score of each attention head is obtained by summing this tensor in the 0th and 1st dimensions; similarly, the importance score of the feedforward neural network layer shape (768, 3072) can be obtained; the pruning mask matrix is ​​calculated based on the importance score, and the pruning matrix is ​​applied to the attention head and feedforward neural network layer parameters through matrix multiplication to complete the unstructured pruning.

[0062] This embodiment provides a hybrid pruning method based on the BERT model. This embodiment first generates an initial mask matrix of corresponding dimensions based on the number of neuron nodes in the BERT model. During the iteration process, the initial mask matrix or the mask matrix generated in the previous iteration is loaded. The corresponding neuron nodes are marked as zero-value parameters based on the importance scores of the neuron nodes and the pruning amount assigned to the current iteration, thereby achieving unstructured dynamic pruning. By pruning to remove unimportant neuron nodes, the number of model parameters is effectively reduced, the model complexity is reduced, the model reasoning efficiency is improved, the consumption of computing resources is reduced, the risk of model overfitting is reduced, and the model generalization ability is enhanced, while ensuring that the model performance meets certain requirements.

[0063] Based on Example 1, Example 3 of this application proposes a hybrid pruning method based on the Bert model, referring to Figure 5 , step S10 includes: Optionally, the target pruning parameters include a target number of retained heads and a total number of iterations.

[0064] In step S11, the total number of self-attention heads to be pruned, i.e., the total number of heads pruned, is calculated according to the preset target number of retained heads.

[0065] It should be noted that the target number of retained heads is the pre-set number of self-attention heads that need to be retained after pruning is completed. The total number of heads pruned is the total number of self-attention heads that need to be pruned during the entire pruning process in order to achieve the target number of retained heads.

[0066] For example, the total number of original self-attention heads in the BERT model is counted, and the total number of target retained heads is subtracted from the total number of original self-attention heads to obtain the total number of heads pruned.

[0067] Step S12: Based on the total number of iterations, the total amount of head pruning is evenly distributed to each iteration of the Bert model, and the remainder of the evenly distributed number of heads is obtained.

[0068] It should be noted that the total number of iterations is the total number of iterations performed during the pruning process. The evenly distributed head remainder is the remainder obtained by dividing the total number of heads pruned by the total number of iterations. It represents the amount of extra pruning that cannot be evenly distributed to each iteration.

[0069] Exemplarily, the total amount of head pruning is divided by the total number of iterations to obtain a quotient and a remainder, the quotient being the average number of heads pruned per iteration, and the remainder being the remainder of the evenly distributed number of heads.

[0070] Step S13: Calculate the number of head quantity remainder allocation times according to the number of head quantity remainder, and determine the corresponding number of iterations x based on the number of head quantity remainder allocation times.

[0071] It should be noted that the number of allocations for the remainder of the number of heads determines the number of iterations at which an additional pruning amount is required based on the remainder of the number of heads. That is, the number of iterations in which an additional self-attention head is pruned is equal to the remainder. The number of iterations is the number of iterations in which the additional pruning amount is required.

[0072] Step S14: distribute the remainder of the number of heads to the first x iterations according to the iteration order, so that an additional head pruning amount is added to each of the first x iterations, and determine the pruning amount of each iteration of the Bert model.

[0073] The remainder of the number of heads is reasonably distributed to the specified iterations, so that an additional head pruning amount is added each time in the first x iterations, thereby accurately determining the pruning amount for each iteration, ensuring that the entire pruning process proceeds as planned and achieves the predetermined pruning goal.

[0074] It should be noted that the iteration order refers to the order in which the iterations are performed during the pruning process. The head pruning amount is the number of self-attention heads that need to be pruned in each iteration, which is the base pruning amount plus the additional pruning amount allocated based on the remainder.

[0075] Exemplarily, the total amount of head pruning is divided by the total number of iterations to obtain the basic pruning amount. According to the number of iterations x, the additional pruning amount is distributed to the first x iterations. In the order of iterations, the pruning amount for each of the first x iterations is the basic pruning amount plus 1, and the pruning amount for the remaining iterations is the basic pruning amount.

[0076] As another optional implementation of this embodiment, for the iterative pruning process of a feedforward neural network, the target pruning parameters may further include a target parameter amount to be retained and a total number of iterations. Based on the preset target parameter amount to be retained, the total parameter amount to be pruned from the feedforward neural network, i.e., the total parameter pruning amount, is calculated; based on the total number of iterations, the total parameter pruning amount is evenly distributed to each iteration, and the parameter amount remainder of the even distribution is obtained; based on the parameter amount remainder, the number of parameter amount remainder allocations is calculated, and based on the number of parameter amount remainder allocations, the corresponding number of iterations M is determined; and based on the iteration order, the parameter amount remainder is distributed to the first M iterations, so that each of the first M iterations adds an additional parameter pruning amount.

[0077] This embodiment provides a hybrid pruning method based on the Bert model. This embodiment first gradually prunes self-attention heads, gradually reducing the number of attention heads that the model needs to process in each iteration, thereby reducing the model's computational complexity and improving its processing efficiency. The total amount of head pruning is evenly distributed across iterations, and the remaining pruning amount is rationally allocated using a remainder compensation strategy. This makes the pruning amount for each iteration more precise, avoids over-pruning or under-pruning, and improves the accuracy and effectiveness of pruning.

[0078] Based on the first and second embodiments, the fourth embodiment of the present application proposes a hybrid pruning method based on the Bert model, referring to Figure 6 , step S61 includes: Step A10: Input the sample data into the attention layer of the Bert model, and calculate the importance score of each attention head through forward propagation and backpropagation.

[0079] It should be noted that sample data refers to samples from the dataset used to train and evaluate the BERT model, consisting of input text and corresponding labels. Forward propagation is the process in a neural network where input data is passed through the layers to the output layer. Backward propagation is the process in a neural network where the gradient of the loss function with respect to the parameters of each layer is calculated and used to update the parameters. The attention head importance score measures the importance of the attention head in the model and is used to determine pruning priorities.

[0080] As an optional implementation, the sample data is fed into the attention layer of the BERT model. During the forward propagation, the absolute value of the output weight of each attention head is recorded; during the backward propagation, the magnitude of the gradient of each attention head is recorded. The absolute value of the output weight and the magnitude of the gradient are combined to calculate the importance score of each attention head.

[0081] For example, the output weights of the model's attention layer are obtained, the model's forward propagation process is performed to obtain loss data, the model's backpropagation is performed based on the loss data, and the importance score is calculated. By calculating the Taylor first-order expansion of the loss function, the bias characteristic is introduced, and the influence rate β (less than 5e-5) is set for the bias. Finally, the mathematical expectation of the importance score is calculated on the dataset to obtain the pruning importance score. The formula is:

[0082]

[0083] Optionally, step A10 includes: Step A11, input the sample data through the attention layer of the Bert model, perform forward propagation, obtain the weight tensor and bias vector of each attention head of the attention layer, and the loss function value of the Bert model.

[0084] It's important to note that the weight tensor is a multidimensional array of weight parameters in the attention head, used to compute the linear combination of the input data. The bias vector is a one-dimensional array of bias parameters in the attention head, used to adjust the linear combination results. The loss function value is a metric that measures the difference between the model's predictions and the true labels, and is used to guide the update of model parameters.

[0085] As an optional implementation, the sample data is input into the attention layer of the Bert model, and the weight tensor and bias vector of each attention head in the attention layer are obtained through forward propagation calculation. The value of the loss function is calculated based on the output of the model and the true label of the sample data.

[0086] Step A12, performing back propagation based on the loss function value, calculating the weight gradient tensor of the weight tensor and the bias gradient tensor of the bias vector.

[0087] It should be noted that the weight gradient tensor is the gradient of the weight tensor, reflecting the degree of influence of the weight parameter on the loss function. The bias gradient vector is the gradient of the bias vector, reflecting the degree of influence of the bias parameter on the loss function.

[0088] Step A13: Obtain the importance score of each attention head by weighted combination of the weight gradient tensor and the bias gradient tensor of each attention head.

[0089] By weightedly combining the weight gradient tensor and the bias gradient vector, we get the importance score of each attention head, which is used to determine the priority of pruning.

[0090] It should be noted that the weighted combination is a linear combination of the weight gradient tensor and the bias gradient vector through a certain weight coefficient to comprehensively evaluate the importance of the attention head.

[0091] Optionally, the weight coefficients of the weight gradient tensor and the bias gradient vector are determined empirically or experimentally.

[0092] Exemplarily, for each attention head, its weight gradient tensor and bias gradient vector are weightedly combined to calculate the importance score of the attention head.

[0093] Step A20: Based on the importance scores of the attention heads from low to high, prune the attention heads corresponding to the pruning amount allocated in the current iteration, and update the initial mask matrix or the mask matrix of the attention layer.

[0094] As an optional implementation, the number of attention heads to be pruned is determined based on the pruning amount assigned to the current iteration. All attention heads are sorted from low to high by importance score, and a specified number of attention heads with the lowest scores are selected as pruning targets. The corresponding elements in the mask matrix for the selected attention heads are marked as 0, generating an updated mask matrix.

[0095] Optionally, step A20 includes: Step A21, sort all the attention heads in the attention layer according to the importance scores from low to high, and determine the sorting result.

[0096] It should be noted that the full connection sorting is to sort all attention heads from low to high according to their importance scores to determine the priority of pruning.

[0097] For example, the importance score of each attention head is obtained, and all attention heads are sorted from low to high according to the importance score to obtain a sorting result.

[0098] Step A22: Determine the number of attention heads to be removed in the current iteration based on the pruning amount allocated in the current iteration.

[0099] It should be noted that the pruning amount is the number of attention heads that need to be pruned in this iteration.

[0100] Step A23: Select the attention head to be removed in the current iteration according to the sorting result and the number of attention heads, and determine the corresponding attention head position.

[0101] It should be noted that the attention head position is the position identifier of each attention head in the attention layer.

[0102] For example, based on the sorting results and the number of attention heads to be removed, the attention heads with the lowest scores are selected, and the positions of these attention heads in the attention layer are recorded.

[0103] Step A24, load the mask matrix generated in the previous iteration, perform binary marking on the attention head position, and update the initial mask matrix or the mask matrix.

[0104] It should be noted that the binary mark is in the mask matrix, marking the position of the attention head to be pruned as 0.

[0105] Exemplarily, the mask matrix generated in the previous iteration is obtained, and according to the determined attention head position, the corresponding position in the mask matrix is ​​marked as 0, and the updated mask matrix is ​​saved.

[0106] As an example of this embodiment, the pruning matrix for the attention heads is calculated based on the importance scores of the attention heads, head_importance_score. A pruning matrix mask with the same shape and all elements set to 1 is initialized based on the importance scores. The importance scores are a tensor of (12, 12). The first parameter represents the number of model layers, and the second parameter represents the number of attention heads. The attention scores of each layer are sorted by the number of model layers, and the elements corresponding to the smallest attention scores after sorting are set to 0. Attention heads are pruned by pruning the query, key, and value weights and biases of the attention heads according to the pruning matrix, and the output weights are pruned. The pruned attention heads are reshaped into a tensor (768, target_head_size * 64) that conforms to the custom parameters. The output weights are the weight matrix used in the attention mechanism to map the multi-head attention output back to the original feature dimensions. head_size_per is the number of attention heads to be pruned per iteration, calculated by dividing the total number of pruned attention heads by the total number of iterative pruning attempts.

[0107] Optionally, calculate the importance score (ffn_importance_score) of the feedforward neural network and obtain weight gradients through backpropagation. Calculate the importance scores of the weights using the importance score calculation formula, and use ffn_importance_score to calculate the pruning matrix. Initialize a pruning matrix (ffn_mask) with the same shape and all 1 elements based on the importance scores. The first parameter of the importance score is a tensor (12, 3072), with the number of network layers and the weight dimension of the feedforward neural network. Iterate through the model layers, sorting the weight scores of each layer. Then, set the corresponding elements of the first ffn_size_per smallest attention scores after sorting to 0 to obtain the pruning matrix. Prune the feedforward neural network by pruning the weights and biases of the intermediate dense layer (intermediate.dense) and the weights of the output dense layer (output.dense) according to the weight matrix. Pruning is completed according to the pruning matrix. ffn_size_per is the number of feedforward neural network parameters to be pruned per iteration, calculated by dividing the total number of parameters to be pruned by the total number of iterative pruning.

[0108] This example provides a hybrid pruning method based on the Bert model. This method first prunes unimportant attention heads, reducing the computational complexity of the model during training and inference. By ranking the importance scores of attention heads, it avoids the performance degradation caused by blind pruning and ensures the rationality of the pruning process.

[0109] Based on the first and fourth embodiments, the fifth embodiment of the present application proposes a hybrid pruning method based on the Bert model, referring to Figure 7 , after step A10, including: Step A30, calculating the minimum absolute value of the weight tensor of each attention head in the attention layer.

[0110] It should be noted that the minimum absolute value is the minimum absolute value of all elements in the weight tensor, which is used to determine the scale of the noise matrix.

[0111] For example, the weight tensor of each attention head is obtained from the attention layer, the absolute values ​​of all elements in each weight tensor are calculated, and the minimum value is found among the calculated absolute values.

[0112] Step A40: Based on a preset noise scaling range, randomly generate a noise amplitude scaling factor within the noise scaling range.

[0113] It should be noted that the noise scaling range is a pre-set range of noise amplitudes used to control the intensity of the noise. The noise amplitude scaling factor is a randomly generated factor within the noise scaling range used to adjust the amplitude of the noise matrix.

[0114] Optionally, a reasonable noise scaling range is set based on experience or experiments.

[0115] Step A50: Generate a noise matrix of the same dimension as the current layer attention score tensor based on the minimum value and the noise amplitude scaling factor.

[0116] It should be noted that the attention score tensor is a multidimensional array representing the importance scores of each attention head in the attention layer. The noise matrix is ​​a matrix of the same dimensions as the attention score tensor, containing randomly generated noise values ​​to enhance the robustness of the importance scores.

[0117] Exemplarily, the dimension of the attention score tensor of the current layer is obtained, and a noise matrix with the same dimension as the attention score tensor is generated according to the minimum absolute value of the weight tensor and the noise amplitude scaling factor.

[0118] Step A60: superimpose the noise matrix onto the original importance score to obtain the noise-enhanced importance score.

[0119] It should be noted that the original importance score is the importance score of the attention head calculated through forward propagation and backpropagation. The noise-enhanced score is the new score obtained by superimposing the noise matrix on the original importance score, which is used to improve the robustness of the model.

[0120] Exemplarily, the calculated original importance score is obtained, and the generated noise matrix is ​​added to the original importance score element by element to obtain the noise-enhanced importance score.

[0121] As an example of this embodiment, the importance score is first calculated, and a tensor of shape (768, 786) is obtained by multiplying the weight and gradient and adding the bias. The tensor is reshaped into a shape of (768, 12, 64) and then summed over the 0th and 2nd dimensions to obtain a 12-dimensional tensor. Each dimension corresponds to a self-attention head, and noise enhancement is added to the attention layer data. When the amount of data is limited, in order to prevent the model from overfitting, noise is added to the importance score. The noise is calculated by scaling a (0.1-1) times according to the minimum value of the current layer, and then a tensor with the same shape as the current layer and a size between (-mag_norm, mag_norm) is generated and added to the attention score tensor of the current layer to obtain the attention score tensor after noise enhancement.

[0122] This example provides a hybrid pruning method based on the Bert model. By first adding a noise matrix to the importance score, the model learns a more robust feature representation during training. This allows the model to make more stable predictions in the face of noise or perturbations in the input data, reducing performance fluctuations caused by noise. The output of the noise enhancement function is the enhancement of the importance score by the noise method, improving the generalization ability of the importance score on unknown datasets.

[0123] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the hybrid pruning method based on the Bert model of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0124] The present application provides a hybrid pruning device based on the Bert model. The hybrid pruning device based on the Bert model includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the hybrid pruning method based on the Bert model in the above-mentioned embodiment 1.

[0125] Reference below Figure 8 , which shows a schematic diagram of the structure of a Bert model-based hybrid pruning device suitable for implementing embodiments of the present application. The Bert model-based hybrid pruning device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptops, personal digital assistants (PDAs), and tablet computers (PADs), as well as fixed terminals such as desktop computers. Figure 8 The hybrid pruning device based on the Bert model shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0126] like Figure 8As shown, a hybrid pruning device based on the Bert model may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the hybrid pruning device based on the Bert model. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input device 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication device 1009. Communication device 1009 can allow the Bert model-based hybrid pruning device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a Bert model-based hybrid pruning device with various systems, it should be understood that implementation or presence of all the illustrated systems is not required. More or fewer systems may alternatively be implemented or present.

[0127] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0128] The Bert model-based hybrid pruning device provided in this application, which employs the Bert model-based hybrid pruning method of the aforementioned embodiment, can solve the technical problem of how to improve the throughput of the Bert model while ensuring model accuracy. Compared with the prior art, the beneficial effects of the Bert model-based hybrid pruning device provided in this application are the same as those of the Bert model-based hybrid pruning method provided in the aforementioned embodiment. Other technical features of the Bert model-based hybrid pruning device are the same as those disclosed in the aforementioned embodiment and are not further described here.

[0129] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0130] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0131] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, and the computer-readable program instructions are used to execute the hybrid pruning method based on the Bert model in the above embodiment.

[0132] The computer-readable storage medium provided herein may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including, but not limited to, wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0133] The computer-readable storage medium may be included in the hybrid pruning device based on the Bert model; or may exist independently without being assembled into the hybrid pruning device based on the Bert model.

[0134] The computer-readable storage medium carries one or more programs. When executed by a Bert-based hybrid pruning device, the Bert-based hybrid pruning device can write computer program code for performing the operations of the present application in one or more programming languages, or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0135] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0136] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0137] The computer-readable storage medium provided in this application is a computer-readable storage medium storing computer-readable program instructions (i.e., a computer program) for executing the aforementioned hybrid pruning method based on the Bert model. This computer-readable storage medium can address the technical problem of improving the throughput of the Bert model while ensuring model accuracy. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the hybrid pruning method based on the Bert model provided in the aforementioned embodiments, and are not further elaborated here.

[0138] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A hybrid pruning method based on the Bert model, characterized in that: Applied to the nth iteration, the hybrid pruning method based on the Bert model includes: Obtaining the pruning amount for the current iteration, where the pruning amount is calculated using a remainder compensation strategy based on preset target pruning parameters; Load the mask matrix generated by iteration n-1, identify and remove all parameter positions marked as zero-valued parameters, and generate an intermediate matrix containing only non-zero parameters; Rearranging the non-zero parameters of the intermediate matrix into a dense matrix stored continuously in row-major order; The dimension of the dense matrix is ​​adjusted so that the number of rows of the dense matrix remains unchanged and the number of columns is reduced to the target dimension to reshape it into a regularized matrix, and the pruned BERT model is output.

2. The hybrid pruning method based on the Bert model according to claim 1, characterized in that: Applied to the 1st to n-1th iterations, before the step of loading the mask matrix generated by the n-1th iteration, identifying and removing all parameter positions marked as zero-valued parameters, and generating an intermediate matrix containing only non-zero parameters, the method includes: Generate an initial mask matrix of corresponding dimensions according to the number of neuron nodes in the Bert model, where all elements in the initial mask matrix are non-zero parameters; The initial mask matrix or the mask matrix generated in the previous iteration is loaded, and based on the importance score of each neuron node and the pruning amount allocated in the current iteration, the corresponding neuron node is marked as a zero-value parameter to generate the updated mask matrix.

3. The hybrid pruning method based on the Bert model according to claim 1, characterized in that: The target pruning parameters include the target number of retained heads and the total number of iterations; The step of obtaining the pruning amount of the current iteration, wherein the pruning amount is calculated according to the preset target pruning parameter by the remainder compensation strategy, includes: According to the preset target number of retained heads, the total number of self-attention heads to be pruned is calculated, i.e., the total number of heads pruned; Based on the total number of iterations, evenly distribute the total amount of head pruning to each iteration of the Bert model, and obtain the remainder of the evenly distributed number of heads; Calculating the number of head quantity remainder allocation times according to the head quantity remainder, and determining the corresponding number of iterations x based on the number of head quantity remainder allocation times; The remainder of the number of heads is distributed to the first x iterations according to the iteration order, so that an additional head pruning amount is added to each of the first x iterations, and the pruning amount of each iteration of the BERT model is determined.

4. The hybrid pruning method based on the Bert model according to claim 2, characterized in that: The step of loading the initial mask matrix or the mask matrix generated in the previous iteration, and marking the corresponding neuron nodes as zero-value parameters based on the importance score of each neuron node and the pruning amount allocated in the current iteration to generate the updated mask matrix includes: For the iterative pruning process of the attention layer, load the initial mask matrix or the mask matrix of the attention layer generated in the previous iteration, mark the corresponding number of attention heads based on the importance scores of the attention heads, and update the mask matrix of the attention layer; For the pruning iterative process of the feedforward neural network, the initial mask matrix or the mask matrix of the feedforward neural network generated by the previous iteration is loaded, and based on the back-propagated channel importance scores, the corresponding number of intermediate layer channels are marked to update the mask matrix of the feedforward neural network.

5. The hybrid pruning method based on the Bert model according to claim 4, characterized in that: The iterative pruning process for the attention layer includes loading the initial mask matrix or the mask matrix of the attention layer generated in the previous iteration, marking a corresponding number of the attention heads based on the importance scores of the attention heads, and updating the mask matrix of the attention layer. Input the sample data into the attention layer of the Bert model, and calculate the importance score of each attention head through forward propagation and backpropagation; Based on the importance scores of each of the attention heads from low to high, prune the attention heads corresponding to the pruning amount allocated in the current iteration, and update the initial mask matrix or the mask matrix of the attention layer.

6. The hybrid pruning method based on the Bert model according to claim 5, characterized in that: The step of inputting sample data into the attention layer of the Bert model and calculating the importance score of each attention head through forward propagation and backpropagation includes: Input the sample data through the attention layer of the Bert model, perform forward propagation, obtain the weight tensor and bias vector of each attention head of the attention layer, and the loss function value of the Bert model; Performing backpropagation based on the loss function value to calculate a weight gradient tensor of the weight tensor and a bias gradient tensor of the bias vector; The importance score of each attention head is obtained by weighted combination of the weight gradient tensor and the bias gradient tensor of each attention head.

7. The hybrid pruning method based on the Bert model according to claim 5, characterized in that: After the step of inputting the sample data into the attention layer of the Bert model and calculating the importance score of each attention head through forward propagation and backpropagation, the method further includes: Calculating the minimum absolute value of the weight tensor of each attention head in the attention layer; Based on a preset noise scaling range, randomly generating a noise amplitude scaling factor within the noise scaling range; Generate a noise matrix of the same dimension as the current layer attention score tensor according to the minimum value and the noise amplitude scaling factor; The noise matrix is ​​superimposed on the original importance score to obtain the noise-enhanced importance score.

8. The hybrid pruning method based on the Bert model according to claim 5, characterized in that: The step of pruning the attention heads corresponding to the pruning amount allocated in the current iteration based on the importance scores of the attention heads from low to high, and updating the initial mask matrix or the mask matrix of the attention layer includes: According to the importance scores, all the attention heads in the attention layer are fully connected and sorted from low to high to determine a sorting result; Determine the number of attention heads to be removed in the current iteration based on the pruning amount allocated for the current iteration; According to the sorting result and the number of attention heads, the attention heads to be removed in the current iteration are selected, and the corresponding attention head positions are determined; The initial mask matrix or the mask matrix generated in the previous iteration is loaded, the attention head position is binary marked, and the initial mask matrix or the mask matrix is ​​updated.

9. A hybrid pruning device based on the Bert model, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the hybrid pruning method based on the Bert model according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the hybrid pruning method based on the Bert model according to any one of claims 1 to 8 are implemented.

Images