Neural network global one-time structured pruning method, system and device and medium

Abstract

The invention relates to a neural network global one-time structured pruning method, system and device and a medium, and the method comprises the steps: carrying out the parallel capturing of the input activation tensors of all target layers in a to-be-pruned neural network through a calibration data set through single-time forward propagation; according to an input activation tensor, synchronously calculating a difference entropy index and amplitude response intensity for an intermediate neuron weight group of each target layer, and performing normalization and fusion to form a static global importance map; determining an importance threshold according to a preset pruning rate, and generating a global to-be-pruned index set of which the mixed importance score is lower than the importance threshold at one time based on the global importance map; and on the basis of the index set, performing one-time physical structured pruning on the weight matrix of each target layer. Therefore, the static global importance map is generated through single forward propagation and parallel capture of the activation tensor, and maskless one-time pruning is completed through physical structured pruning.

Description

Technical Field

[0001] The present invention relates to the field of data model processing technology, and in particular to a method, system, device and medium for global one-time structured pruning of a neural network. Background Art

[0002] In the field of neural network model compression, existing methods mostly focus on unstructured pruning. The randomly distributed sparse pattern leads to low compression efficiency of model resource usage (it cannot effectively reduce structured storage requirements), and actual inference acceleration is heavily dependent on the sparse computing unit support provided by dedicated hardware, which greatly limits the universality of engineering deployment.

[0003] For example, existing unstructured neural network pruning methods (such as Wanda) use a layer-dependent processing flow: first, the importance scores of the first-layer neurons are calculated based on the original input data. After the pruning operation is completed for this layer, forward propagation is immediately performed, and the perturbed intermediate output is used as the input reference for subsequent layers. However, this layer-by-layer processing mechanism leads to a series of technical drawbacks.

[0004] Because pruning is inherently an irreversible information loss process, the approximation errors caused by pruning the first layer are directly injected into subsequent layers through the forward propagation path. When calculating the importance of secondary neurons, the system has to base its assessment on distorted input activation values, which leads to systematic deviations in the reliability of importance scores. As the number of processing layers increases, the error accumulation effect between layers through this forward dependency is formed. The input activation data of the deeper layers will accumulate the pruning errors of all previous layers, resulting in a trend of decreasing accuracy of importance assessments at each layer.

[0005] This method's implementation also introduces significant computational resource consumption. Since each layer of pruning requires re-running the full forward propagation of samples to update the inputs of subsequent layers, the computational effort required grows linearly with network depth when dealing with modern neural networks with numerous layers and complex structures.

[0006] In addition, the calibration datasets that existing methods rely on when pruning general pre-trained models are usually unable to align with the actual input data distribution of the network to be pruned. For models that have undergone professional fine-tuning, they face the dilemma of high cost of constructing calibration data and difficulty in ensuring its effectiveness. In addition, they rely on static activation data obtained in the preprocessing stage as the basis for importance assessment, which cannot effectively represent the actual activation pattern of the model during dynamic reasoning. This makes the pruning decisions based on static data analysis have essential deviations from the actual operation scenarios.

[0007] Therefore, when dealing with multi-layer neural network pruning tasks, especially when dealing with models that have been professionally fine-tuned (non-pretrained models), existing technologies face multiple technical bottlenecks such as error conduction amplification, low computational efficiency, and distorted importance assessment. It is urgent to develop new solutions to break through the core constraints of the current method system. Summary of the Invention

[0008] (1) Technical issues to be resolved In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a global one-time structured pruning method, system, device and medium for a neural network, which solves the technical problems of the existing pruning technology, which relies on the combined effect of unstructured sparse patterns and serialized layer-by-layer processing mechanisms, resulting in low model storage compression efficiency and limited hardware acceleration benefits, while inevitably causing error propagation, low computational efficiency and distorted importance assessment.

[0009] (2) Technical solution In order to achieve the above objectives, the main technical solutions adopted by the present invention include: In a first aspect, an embodiment of the present invention provides a global one-time structured pruning method for a neural network based on a hybrid amplitude entropy importance map, comprising: Through a single forward propagation, the calibration dataset is used to capture the input activation tensors of all target layers in parallel in the neural network to be pruned; According to the input activation tensor, the correlation weights of the intermediate neurons in each target layer are reorganized to synchronously calculate the differential entropy index and amplitude response strength, and after normalization and fusion, a static global importance map is formed; Determine the importance threshold according to the preset pruning rate, and generate the index set of all global intermediate neurons to be pruned with scores lower than the importance threshold at one time based on the global importance map; Based on the index set, a one-time physical structured pruning is performed on the weight matrix of each target layer.

[0010] Optionally, using a calibration dataset in parallel to capture the input activation tensors of all target layers in the neural network to be pruned via a single forward propagation includes: Load the selected neural network to be pruned and obtain a calibration dataset by randomly sampling from the original training data; Input the calibration data set into the neural network to be pruned and perform a single forward propagation calculation; During the forward propagation process, the pre-implanted hooks are used to intercept the input activation tensors of each target feedforward network layer in parallel at one time, and the input activation tensors are iteratively stored to ensure that only a single stored result of the fusion of the current input activation tensor and the historical input activation tensors is always retained; Among them, the hidden state vector of each word in the input activation tensor is generated based on the corresponding context in the decoding stage and is used to predict the final output word. The word hidden state vector is the smallest operable data unit in the input activation tensor that carries independent semantic context information.

[0011] Optionally, based on the input activation tensor, the differential entropy index and amplitude response strength are synchronously calculated for the associated weights of the intermediate neurons in each target layer. After normalization and fusion, a static global importance map is formed, including: Configure the associated weight group for each intermediate neuron of the target feedforward network layer, where the associated weight group contains the specified row of the gated projection layer weight matrix, the specified row of the upstream projection layer weight matrix, and the specified column of the downstream projection layer weight matrix; Based on the input activation tensor, the product matrix of each weight in the associated weight group and the activation response strength index of the corresponding feature channel is calculated, and the aggregation operation is performed according to the input or output dimension to generate the amplitude response strength score; In the descending projection layer, the intermediate activation tensor generated by the modulation of the gated projection and the upward projection output is reconstructed in two dimensions and its feature channels are separated. The differential entropy index of each intermediate neuron is calculated by kernel density estimation and numerical integration. The amplitude response intensity score and differential entropy index were normalized respectively; The normalized amplitude results are weightedly fused with the differential entropy results to generate a static global importance map that represents the undisturbed intrinsic information.

[0012] Optionally, based on the input activation tensor, a product matrix of the weights in the associated weight group and the activation response strength index of the corresponding feature channel is calculated, and an aggregation operation is performed according to the input or output dimension to generate an amplitude response strength score including: Locate the weight matrix row corresponding to the target neuron in the gated projection layer, traverse the weight elements corresponding to all input channels of the weight matrix row, multiply the absolute value of each weight element by the L2 norm of the feature channel corresponding to the input activation tensor, and accumulate the sum along the input feature channel direction to obtain the gated row importance score; Locate the weight matrix row corresponding to the target neuron in the upstream projection layer, traverse the weight elements corresponding to all input channels in the weight matrix row, multiply the absolute value of each weight element by the L2 norm of the feature channel corresponding to the input activation tensor, and accumulate the sum along the input feature channel direction to obtain the upstream row importance score; Locate the weight matrix column corresponding to the target neuron in the downstream projection layer, traverse the weight elements corresponding to all output channels of the weight matrix column, multiply the absolute value of each weight element by the L2 norm of the feature channel corresponding to the intermediate activation tensor, and accumulate the sum along the output feature channel to obtain the downstream column importance score; the intermediate activation tensor is obtained by performing a Hadamard product operation on the output of the upstream projection layer after being transformed by the SiLU function of the gated projection layer. The gated row importance score, the upper row importance score, and the lower column importance score are summed to generate an amplitude response strength score.

[0013] Optionally, in the down-projection layer, the intermediate activation tensor generated based on the modulation of the gated projection and the up-projection output is subjected to two-dimensional reconstruction and feature channel separation, and the differential entropy index of each intermediate neuron is calculated by kernel density estimation and numerical integration, including: In the down-projection layer, the intermediate activation tensor generated in a three-dimensional form based on the modulation of the gated projection and the up-projection output is reconstructed along the batch and sequence dimensions into a two-dimensional matrix with independent observation samples as rows and feature dimensions as columns; Extract the activation value set corresponding to each intermediate neuron in the two-dimensional matrix column by column; Perform kernel function smoothing on the activation value set of each intermediate neuron to construct a continuous probability density function that characterizes the distribution characteristics of the neuron's activation value; Based on the continuous probability density function, the logarithmic probability weighted integral of the activation value distribution is calculated within the preset numerical integration interval, and the differential entropy index is approximately calculated by numerical integration.

[0014] Optionally, determining the importance threshold according to a preset pruning rate, and generating an index set of all global intermediate neurons to be pruned with scores lower than the importance threshold at one time based on the global importance map includes: According to the preset network pruning ratio, percentile statistical operations are performed on the global map containing the normalized importance scores of all neurons to determine the corresponding global importance threshold; Scan the global importance map element by element, and compare the numerical relationship between the importance score of each neuron and the global importance threshold in real time; The position information of neurons whose importance scores are continuously lower than the global importance threshold is synchronously recorded into a unified pruning index container across all target feedforward network layers to form an index set of global intermediate neurons to be pruned.

[0015] Optionally, performing a one-time physical structured pruning on the weight matrix of each target layer based on the index set includes: The target feedforward network layer containing the intermediate neurons to be pruned is retrieved layer by layer along the depth direction of the network to generate a queue of layers to be processed; Create a transactional temporary mirror copy of the weight matrix of each target feedforward network layer in the queue of pending layers to establish a backup of the original weight state; For the queue of layers to be processed, based on the coordinate information of the intermediate neurons to be pruned recorded in the index set, the weight matrices of the gated projection layer and the upstream projection layer are reduced in input dimension, and the row vectors of the weight matrices corresponding to the intermediate neurons to be pruned are deleted; At the same time, the output dimension of the weight matrix of the downstream projection layer is reduced, and the weight matrix column vector corresponding to the intermediate neurons to be pruned is deleted; Based on the original weight state backup, perform preset inter-layer connection dimension matching verification on all target feedforward network layers that have completed the reduction operation; If the dimension matching verification passes, the reduced weight matrix is ​​atomically replaced in batches to the corresponding network layer; If the dimension matching verification fails, the transaction rollback mechanism is automatically triggered to restore the weight matrices of all target feedforward network layers to the original backup state; After completing a one-time structured row and / or column dimension removal for all target feed-forward network layers, a compressed model without masking operations is generated.

[0016] In a second aspect, an embodiment of the present invention provides a global one-time structured pruning system for a neural network based on a hybrid amplitude entropy importance map, comprising: A tensor capture module is used to capture the input activation tensors of all target layers in parallel in the neural network to be pruned using the calibration dataset through a single forward propagation; The atlas output module is used to reorganize the associated weights of the intermediate neurons in each target layer according to the input activation tensor, and synchronously calculate the differential entropy index and amplitude response strength. After normalization and fusion, a static global importance atlas is formed; The neuron screening module is used to determine the importance threshold according to the preset pruning rate and generate an index set of all global intermediate neurons to be pruned whose scores are lower than the importance threshold at one time based on the global importance map; The structured pruning module is used to perform one-time physical structured pruning on the weight matrix of each target layer based on the index set.

[0017] In a third aspect, an embodiment of the present invention provides a global one-time structured pruning device for a neural network based on a hybrid amplitude entropy importance map, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the global one-time structured pruning method for a neural network based on a hybrid amplitude entropy importance map as described above.

[0018] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer-executable instructions stored thereon. When the executable instructions are executed by a processor, the global one-time structured pruning method of a neural network based on a hybrid amplitude entropy importance map as described above is implemented.

[0019] (3) Beneficial effects The beneficial effects of the present invention are as follows: first, the static input activation tensors of all target layers in the neural network to be pruned can be captured in parallel through a single forward propagation, breaking through the limitations of traditional layer-by-layer dynamic serial capture, eliminating the propagation and accumulation of errors in multi-layer networks, and simultaneously achieving an order of magnitude reduction in computing resource consumption and an improvement in the utilization of hardware execution units. On this basis, a multi-dimensional fusion evaluation strategy of differential entropy index and amplitude response intensity is adopted to innovatively realize the cross-layer joint analysis of the intermediate neuron association weight group, and eliminate the memory access fragmentation caused by unstructured sparse patterns through structured correlation modeling. The static global importance map generated thereby has spatial consistency characteristics, so that the importance score is no longer limited by the activation fluctuations of the local level, thereby effectively avoiding the problem of misjudgment accumulation caused by the inconsistent importance evaluation standards between levels in traditional methods.

[0020] Furthermore, by directly mapping the normalized score distribution of the static global importance map through a preset pruning rate, the screening threshold of all intermediate neurons to be pruned is defined at one time, avoiding the risk of evaluation fluctuations in the iterative optimization process, and achieving global optimality of pruning decisions while ensuring hardware friendliness.

[0021] Finally, based on the execution mechanism of one-time physical structured pruning, the row and column dimensions of redundant parameters are directly removed instead of relying on mask retention. This not only eliminates the additional overhead introduced by mask calculation in traditional pruning methods, but also achieves a dual improvement in storage space and computing efficiency through physical reconstruction of the weight matrix, allowing the compressed model to be directly deployed without the need for additional dimensionality calibration or mask compensation mechanism, thereby avoiding the problem of instantaneous model failure caused by step-by-step pruning in traditional methods.

[0022] Therefore, the present invention does not need to observe tiny perturbations after pruning through expensive iterations, that is, it abandons the traditional linear evaluation paradigm based on activation amplitude, and through high-dimensional spatial structured representation driven by entropy information, only a global "snapshot" is needed on the neural network to be pruned, so as to identify and prune the neurons that should be removed the most and have the least important functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic flow chart of a method provided in an embodiment of the present invention; Figure 2 A schematic diagram of a specific flow chart of step S1 of the method provided in an embodiment of the present invention; Figure 3 A schematic diagram of a specific flow chart of step S2 of the method provided in an embodiment of the present invention; Figure 4 A schematic diagram of a specific flow chart of step S22 of the method provided in an embodiment of the present invention; Figure 5 A schematic diagram of a specific flow chart of step S23 of the method provided in an embodiment of the present invention; Figure 6 A schematic diagram of a specific flow chart of step S3 of the method provided in an embodiment of the present invention; Figure 7 A schematic diagram of a specific flow chart of step S4 of the method provided in an embodiment of the present invention; Figure 8 A schematic diagram comparing the variance of activation values ​​of neurons to be pruned under a traditional pruning scheme and the pruning scheme of the present invention provided in an embodiment of the present invention; Figure 9 A schematic diagram showing a comparison of activation value variances of retained neurons under the pruning scheme of the present invention provided in an embodiment of the present invention; Figure 10 Comparison of the instant pruning performance accuracy provided by the embodiments of the present invention; Figure 11 Comparison of pruning recovery performance accuracy provided by embodiments of the present invention; Figure 12 Schematic diagram of the joint distribution of pruning channel amplitude and entropy provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation methods in conjunction with the accompanying drawings.

[0025] Prior to this, in order to facilitate understanding of the technical solution provided by the present invention, some concepts are first introduced below: FFN: Feed-Forward Network, feed-forward neural network; Hooks: Hooks are a mechanism for intercepting and recording intermediate results during the forward / backward propagation of a neural network; gate_proj: gated projection layer; up_proj: up projection layer; down_proj: down projection layer; A down_in : input activation tensor; PyTorch Hook: A callback mechanism that allows capturing or modifying intermediate layer data during forward / backward propagation of the model, often used to obtain parameters or gradients in real time; mlp.downproj: The linear projection layer in the Transformer feedforward network that compresses the high-dimensional gated activations (gate_proj⊙up_proj results) back to the original model dimensions; KDE: Kernel Density Estimation, that is, kernel density estimation; Language_model: language model module for large language models (especially multimodal models); Batch_size(B): the number of samples in one training step; seq_len(L): the length (number) of the sequence to generate tokens; Prefill: Prefill is the first step in language model inference and only processes the input content; Generate: The generation phase, the autoregressive decoding phase of the language model, processes the generated tokens; Token: word unit, the smallest language structure unit that the language model can process; InternVL3: A large multimodal model of scholarly phenomena, developed by the Shanghai Artificial Intelligence Laboratory; Qwen3: A comprehensive language model developed by Alibaba.

[0026] like Figure 1 As shown, an embodiment of the present invention proposes a global one-time structured pruning method for a neural network based on a hybrid amplitude entropy importance map, comprising: through a single forward propagation, using a calibration data set to capture the input activation tensors of all target layers in the neural network to be pruned in parallel; according to the input activation tensor, the differential entropy index and the amplitude response intensity of the associated weights of the intermediate neurons of each target layer are synchronously calculated, and a static global importance map is formed after normalization and fusion; the importance threshold is determined according to a preset pruning rate, and an index set of all global intermediate neurons to be pruned whose scores are lower than the importance threshold is generated at one time based on the global importance map; based on the index set, a one-time physical structured pruning is performed on the weight matrix of each target layer.

[0027] First, the present invention can capture the static input activation tensors of all target layers in the neural network to be pruned in parallel through a single forward propagation, breaking through the limitations of traditional layer-by-layer dynamic serial capture, eliminating the propagation and accumulation of errors in multi-layer networks, and simultaneously achieving an order of magnitude reduction in computing resource consumption and an improvement in the utilization of hardware execution units. On this basis, a multi-dimensional fusion evaluation strategy of differential entropy index and amplitude response intensity is adopted to innovatively realize the cross-layer joint analysis of the intermediate neuron association weight group, and eliminate the memory access fragmentation caused by unstructured sparse patterns through structured correlation modeling. The static global importance map generated thereby has spatial consistency characteristics, so that the importance score is no longer limited by the activation fluctuations of the local level, thereby effectively avoiding the problem of misjudgment accumulation caused by the inconsistent importance evaluation standards between levels in traditional methods.

[0028] Furthermore, by directly mapping the normalized score distribution of the static global importance map through a preset pruning rate, the screening threshold of all intermediate neurons to be pruned is defined at one time, avoiding the risk of evaluation fluctuations in the iterative optimization process, and achieving global optimality of pruning decisions while ensuring hardware friendliness.

[0029] Finally, based on the execution mechanism of one-time physical structured pruning, the row and column dimensions of redundant parameters are directly removed instead of relying on mask retention. This not only eliminates the additional overhead introduced by mask calculation in traditional pruning methods, but also achieves a dual improvement in storage space and computing efficiency through physical reconstruction of the weight matrix, allowing the compressed model to be directly deployed without the need for additional dimensionality calibration or mask compensation mechanism, thereby avoiding the problem of instantaneous model failure caused by step-by-step pruning in traditional methods.

[0030] Therefore, the present invention does not need to observe tiny perturbations after pruning through expensive iterations, that is, it abandons the traditional linear evaluation paradigm based on activation amplitude, and through high-dimensional spatial structured representation driven by entropy information, only a global "snapshot" is needed on the neural network to be pruned, so as to identify and prune the neurons that should be removed the most and have the least important functions.

[0031] Therefore, the present invention does not need to observe tiny perturbations after pruning through expensive iterations, that is, abandoning the traditional linear evaluation paradigm based on activation amplitude, by introducing a high-dimensional perspective analysis mechanism based on entropy information, only a global "snapshot" is needed on the neural network to be pruned, so as to identify and prune the neurons that should be removed the most and have the least important functions.

[0032] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0033] Specifically, an embodiment of the present invention provides a global one-time structured pruning method for a neural network based on a hybrid amplitude entropy importance map, which includes: S1. Through a single forward propagation, the calibration dataset is used to capture the input activation tensors of all target layers in parallel in the neural network to be pruned.

[0034] Further, if Figure 2 As shown, step S1 includes: S11. Load the selected neural network to be pruned and randomly sample from the original training data to obtain a calibration dataset. The neural network to be pruned is a pre-trained model, such as the InternVL series, Qwen series, or Llama series network model. The calibration dataset uses a data size of 30-128 samples.

[0035] S12. Input the calibration data set into the neural network to be pruned and perform a single complete forward propagation calculation.

[0036] S13. During the forward propagation process, the pre-implanted hooks are used to capture all target feedforward network layers and input activation tensors as target layers in parallel at one time, and the input activation tensors are iteratively stored to ensure that only a single storage result of the fusion of the current input activation tensor and the historical input activation tensor is retained. During the forward propagation process, hooks are set to capture all target FFN layers (such as down_proj) and input activation tensors (such as A down_in ), and iteratively store these captured activation tensors. Each piece of data is combined with the result of the previous one when it is calculated, so that only one tensor is always retained.

[0037] It is important to understand that the hidden state vector of each word in the input activation tensor is generated based on the corresponding context during the decoding phase and is used to predict the final output word. The word hidden state vector is the smallest operable data unit in the input activation tensor that carries independent semantic context information.

[0038] S2. Based on the input activation tensor, the correlation weights of the intermediate neurons in each target layer are reorganized to synchronously calculate the differential entropy index and amplitude response strength, and after normalization and fusion, a static global importance map is formed.

[0039] In this step, for each captured activation tensor, the entropy and amplitude scores for each intermediate feature channel are calculated independently and in parallel. This entropy modeling process involves data reshaping, KDE probability density estimation, and numerical integration, ultimately generating an entropy importance score for all neurons in the model to be pruned. Furthermore, all calculated entropy and amplitude scores are aggregated to form a static, global importance map. This map records the intrinsic information content of each target neuron in the model in its original, unperturbed state.

[0040] Further, if Figure 3 As shown, step S2 includes: S21. Configure the associated weight group for each intermediate neuron in the target feedforward network layer. The associated weight group includes the specified row of the gated projection layer weight matrix, the specified row of the upstream projection layer weight matrix, and the specified column of the downstream projection layer weight matrix. This configuration operation is based on the following considerations: the feature representation of the i-th intermediate neuron can be divided into two stages. In the expansion stage, it is represented by the i-th row of the upstream projection matrix and the gated projection matrix; in the contraction stage, it is represented by the i-th column of the downstream projection matrix. Therefore, the pruning operation of a single neuron requires the simultaneous execution of three operations: removing the i-th row of the upstream projection matrix and the gated projection matrix, and removing the i-th column of the downstream projection matrix to ensure the dimensional consistency of the data stream.

[0041] S22. Based on the input activation tensor, calculate the product matrix of each weight in the associated weight group and the activation response strength index of the corresponding feature channel, perform aggregation operations according to the input or output dimension, and generate an amplitude response strength score. The activation response strength index is selected from, but not limited to, statistics such as the L2 norm, absolute mean, or standard deviation.

[0042] Furthermore, if Figure 4 As shown, step S22 includes: S221. Locate the weight matrix row corresponding to the target neuron in the gated projection layer, traverse the weight elements corresponding to all input channels of the weight matrix row, multiply the absolute value of each weight element by the L2 norm of the feature channel corresponding to the input activation tensor, and accumulate the sum along the direction of the input feature channel to obtain the gated row importance score.

[0043] S222. Locate the weight matrix row corresponding to the target neuron in the upstream projection layer, traverse the weight elements corresponding to all input channels of the weight matrix row, multiply the absolute value of each weight element by the L2 norm of the feature channel corresponding to the input activation tensor, and accumulate the sum along the direction of the input feature channel to obtain the upstream row importance score.

[0044] S223. Locate the weight matrix column corresponding to the target neuron in the downstream projection layer, traverse the weight elements corresponding to all output channels of the weight matrix column, multiply the absolute value of each weight element by the L2 norm of the feature channel corresponding to the intermediate activation tensor, and accumulate and sum them along the output feature channel to obtain the importance score of the downstream column; wherein, the intermediate activation tensor is obtained by performing a Hadamard product operation on the output of the gated projection layer after being transformed by the SiLU activation function and then with the output of the upstream projection layer.

[0045] S224 , adding the gated row importance score, the upper row importance score, and the lower column importance score to generate an amplitude response strength score.

[0046] In one embodiment, the goal is to calculate an importance score I for each interneuron in the FFN. j , in order to perform structured pruning. Pruning the j-th interneuron means removing a set of weights associated with it, thereby maintaining the integrity of the matrix operation.

[0047] The goal of this invention is to remove entire neurons that process intermediate features, which corresponds to entire rows or columns in the matrix. Therefore, individual weights cannot be evaluated independently. A single importance score must be calculated for a group of weights. This "group" contains all weights associated with the same intermediate neuron. Failure to do this, such as independently pruning the i-th row of the up-projection layer up_proj and the k-th column of the down-projection layer down_proj, will disrupt the computational flow of the FFN and lead to structural misalignment.

[0048] For the jth intermediate neuron (j is the intermediate layer neuron index, j∈{1,...,d ffn}), its corresponding weight group includes: gate projection layer gate_proj weight matrix The jth row of d model is the input and output feature dimension, d ffn is the intermediate feature dimension; up_proj weight matrix of the up projection layer The jth row; and the down projection layer down_proj weight matrix The jth column of .

[0049] And the basic component definitions that subsequent calculations rely on: Input Activation: , B is the number of samples, L is the number of generated tokens; gate_proj weight: ; up_proj weight: ; down_proj weight: ; To obtain the amplitude response strength score, we first evaluate the importance of up_proj and gate_proj. u and W g , we need to evaluate the importance of its rows. All weights of the j-th row together determine the output of the j-th intermediate neuron. Specifically, evaluating the row includes the following steps: (1) Calculate the input activation norm: These weights act on the input X. Calculate each input feature (d model dimension), which aggregates information from both batch and sequence dimensions.

[0050]

[0051] Where, the norm vector N in It measures the overall activation strength of each output channel, and X:,:,k represents the sum of the activation values ​​of the k channel under all sample generated tokens (overall activation strength).

[0052] (2) Calculate the importance matrix of weights: First calculate a temporary importance matrix, where each element is the product of the weight size and the corresponding input activation norm.

[0053] ; These two temporary matrices M u , M g The shapes of .

[0054] (3) Aggregation to get row importance: To get a single score for the j-th row (i.e., the j-th neuron), we must aggregate across columns (input dimension d model ) are aggregated. This represents the average response amplitude of all outputs of neurons in the gate_proj and up_proj layers.

[0055] ; Where, I u [j] is the total importance score of neurons in the upward projection layer, I g [j] is the total importance score of the neurons in the gated projection layer. It's worth emphasizing that, given a fixed input dimension, the perturbation of the neuron importance ranking by summing and averaging is only a linear scaling, and does not change the relative order. This property allows for flexible computational choices when designing pruning algorithms without affecting the final pruning results.

[0056] Secondly, the importance of down_proj is evaluated. d , we need to evaluate the importance of its columns, because the j-th column receives input from the j-th intermediate neuron. Specifically, evaluating the column includes the following steps: (1) Calculate the input activation and norm of down_proj: down_proj input A down_in is the Hadamard product of the output activations of the gated projection layer and the output activations of the up-projection layer. (You can also directly obtain the input matrix of the mlp.downproj layer through PyTorch Hooks).

[0057] ; Where SiLU is the activation function and T is the transpose.

[0058] (2) Calculate each intermediate feature (d ffn dimension). This norm vector N down_in It measures the overall output activation strength of each interneuron.

[0059] ; Where A down_in:,:,j Represents the overall activation intensity of the j channel of the down_proj layer under all samples generating tokens.

[0060] (3) Calculate the importance matrix of the downlink projection layer: ; This matrix M d The shape is .

[0061] (4) Aggregation to get column importance: To get a single score for the jth column (corresponding to the jth neuron), we must aggregate across rows (output dimension d model ) for aggregation.

[0062] ; In this way, I u , I g and I d The dimensions and physical meanings of are completely aligned: their j-th elements all represent the importance of the j-th interneuron. Ultimately, the total importance score of the j-th interneuron is the sum of the contributions of the three parts: ; Expressed in vector form: ; In addition, since entropy needs to be combined to make an overall importance assessment later, normalization must be performed separately to ensure that the evaluation of a neuron is holistic and functionally balanced. Reward those neurons that continue to play an important role throughout the life cycle of the information flow, and prune those neurons that may only produce huge but ultimately ineffective activations at a certain stage: ; Where Norm represents minimum-maximum normalization.

[0063] This final vector I final Can be used to all d ffn The intermediate neurons are sorted and the neurons with the lowest scores are pruned. Assuming that the language_model module of this network has a total of k layers, an amplitude importance matrix can be obtained: .

[0064] Where, I mag [i, j] is the amplitude response strength score of the jth neuron in the i-th layer of the FFN layer of language_model.

[0065] S23. In the descending projection layer, the intermediate activation tensor generated based on the modulation of the gated projection and the upward projection output is two-dimensionally reconstructed and the feature channels are separated. The differential entropy index of each intermediate neuron is calculated by kernel density estimation and numerical integration.

[0066] Further, if Figure 5 As shown, step S23 includes: S231. In the down-projection layer, the intermediate activation tensor in three-dimensional form generated based on the modulation of the gated projection and the up-projection output is reconstructed into a two-dimensional matrix with independent observation samples as rows and feature dimensions as columns along the batch dimension and the sequence dimension.

[0067] S232. Extract the activation value set corresponding to each intermediate neuron in the two-dimensional matrix column by column.

[0068] S233. Perform kernel function smoothing on the activation value set of each intermediate neuron to construct a continuous probability density function that characterizes the activation value distribution characteristics of the neuron.

[0069] S234. Based on the continuous probability density function, the logarithmic probability weighted integral of the activation value distribution is calculated within a preset numerical integration interval, and the differential entropy index is approximately calculated through numerical integration.

[0070] S24. Normalize the amplitude response intensity score and the differential entropy index respectively.

[0071] S25. Perform weighted fusion on the normalized amplitude result and the differential entropy result to generate a static global importance map representing the undisturbed intrinsic information volume and store it persistently.

[0072] It should be clarified that this "intrinsic information content" is not a single metric, but a comprehensive score obtained by weighted fusion of features from two orthogonal dimensions. Signal strength is calculated by normalizing the norm of the neuron's output weights to characterize the influence of the neuron in the forward propagation of network information. Activation diversity is calculated by calculating the information entropy of the neuron's activation value distribution on a set of calibration data sets to characterize the richness of the information encoded by the neuron in different contexts.

[0073] In another embodiment, the importance of information entropy metrics is determined by assigning an importance score to each interneuron by evaluating the amount of information it conveys. The core idea is that a neuron with a very stable and predictable activation value distribution (low entropy) carries less information and may be redundant, while a neuron with a widely distributed activation value and high uncertainty (high entropy) carries more information and is therefore more important.

[0074] Unlike the magnitude method that evaluates the product of weights and activation values, the entropy method directly models the distribution of activation values. Its evaluation object only focuses on the input activation of the down_proj layer. This choice is crucial because the activation tensor A down_in It is the final result of the interaction between gate_proj and up_proj, and represents the complete information flow through the FFN layer. Modeling this information bottleneck allows for the most direct evaluation of the comprehensive information contribution of each intermediate channel.

[0075] The input of down_proj is the final confluence point of the up_proj and gate_proj information flows, and the uncertainty of its distribution best represents the information characteristics after the entire FFN layer is processed. This avoids unreasonable mathematical combinations of indicators with different distributions.

[0076] Specifically, the importance of information entropy measurement includes: (1) Data reshaping: In order to model the activation value distribution of each intermediate feature channel, the three-dimensional activation tensor A is reshaped. down_in Reshape into a 2D matrix: ; In this matrix, the rows ( ): represents the sample dimension. Each row is an independent observation sample. B is the number of samples, L is the number of generated tokens, and only the activation value of the decoding stage is saved. Column (d ffn ): represents the feature dimension. Each column contains the activation value of a single intermediate neuron on all samples.

[0077] (2) Separate feature channels: process the matrix A′ column by column down_in .make A′down_in The jth column of this vector C j That is, the jth interneuron is the ) generates a set of activation values ​​on the token, with the goal of calculating the entropy of the distribution represented by this vector.

[0078] (3) Probability density estimation: For each activation vector C j , first we need to estimate the continuous probability density function (PDF) behind these discrete sample points, which is recorded as , assuming that the activation vector C obeys a true distribution p, the activation vector C is modeled by KDE probability density estimation to obtain an estimated distribution to approximate the true distribution, x It is a random variable in the activation vector C.

[0079] Each sample is decoded k times to obtain a total of B×k activation value samples. Experimental verification shows that only the first 32-64 activation samples in the decoded sequence can achieve optimal modeling results. Traditional histogram methods are not robust enough and are limited by the bin selection. Therefore, kernel density estimation (KDE) is used. This is a more advanced non-parametric method that can generate smooth density curves and is particularly suitable for small sample sizes. The mathematical form of KDE is: ; Where K is the kernel function (Gaussian kernel is used in the embodiment of the present invention), h is the bandwidth hyperparameter, and C ji represents the i-th sample value of the j-th activation vector.

[0080] (4) Calculating differential entropy: With a continuous probability density function After that, we can calculate its differential entropy H(C j ), which quantifies the uncertainty of this distribution: ; And approximate this infinite integral with the sum of a finite interval.

[0081] ; Here, KDE creates a continuous function. For numerical computation, it needs to be discretized. To do this, we first collect the maximum and minimum values ​​of the original activation vector to obtain a sampling interval S. Then, we set the required number of sampling points N. Δx is the sampling interval S / N. A smaller sampling interval yields a more accurate estimate, but also increases computational overhead. The number of sampling points N is a hyperparameter, and an empirical value of 100 is used.

[0082] The entropy value calculated for each interneuron is used as its final importance score ; Assuming that the language_model module of this network has a total of k layers, we can get an entropy-aware importance matrix: ; Combining the above calculation formulas and parameters, the calculated I ent and execute and I mag The vectors are normalized in the same way (e.g., converted to the 0-1 range or rank) to eliminate scale differences.

[0083] By combining the two by weighted average, we get the final importance score vector: ; Where α is a pre-set weight parameter, α∈[0,1].

[0084] Finally, the cross-layer I model Aggregated into a global static graph, it provides a unified neuron importance assessment across the model.

[0085] It should be emphasized that in order to ensure the accuracy of the entropy model, the composition of the sample is crucial.

[0086] The first is the correct approach (modeling the distribution of activation values ​​in the generation phase): the sample should be composed of activation values ​​from different independent sequences. For example, in the decoding phase, multiple different prompts are collected to generate the activation values ​​of the token in the generation phase. That is, the vector C j Each element in comes from a different input context p(token|distinct_context). This ensures that the samples are approximately independent and identically distributed, making the estimated It can truly reflect the general activation behavior pattern of the jth neuron in various situations.

[0087] The second incorrect approach (modeling the distribution of activation values ​​during the pre-population phase) is to collect activation values ​​from different token positions within the same long sequence to form a sample. This will not measure the general distribution of neuron activations. The changes in activation values ​​will primarily reflect the internal linguistic structure of that particular sequence, rather than the inherent randomness of neurons during decoding.

[0088] Therefore, unlike the general pruning algorithm that calculates activation values ​​in the prefill phase, it is necessary to obtain the activation values ​​of tokens generated by different samples in the generate phase. Experiments have shown that modeling the initial k (16-32) tokens generated is better than random sampling and modeling using the activation values ​​of all tokens.

[0089] S3. Determine the importance threshold according to the preset pruning rate, and generate an index set of all global intermediate neurons to be pruned whose scores are lower than the importance threshold at one time based on the global importance map.

[0090] Further, if Figure 6 As shown, step S3 includes: S31. Based on a preset network pruning ratio, perform percentile statistical operations on the global graph containing the normalized importance scores of all neurons to determine a corresponding global importance threshold. Based on a preset global pruning rate (e.g., removing 25% of the neurons), determine a unified entropy importance score threshold.

[0091] S32. Scan the global importance map element by element, and compare the numerical relationship between the importance score of each neuron and the global importance threshold in real time.

[0092] S33. Synchronously record the position information of neurons whose importance scores are continuously lower than the global importance threshold across all target feedforward network layers into a unified pruning index container to form an index set of global intermediate neurons to be pruned.

[0093] S4. Based on the index set, perform a one-time physical structured pruning on the weight matrix of each target layer.

[0094] Further, if Figure 7 As shown, step S4 includes: S41. Retrieve the target feedforward network layer containing the intermediate neurons to be pruned layer by layer along the depth direction of the network, and generate a queue of layers to be processed.

[0095] S42. Create a transactional temporary mirror copy for the weight matrix of each target feedforward network layer in the queue of layers to be processed, and establish a backup of the original weight state.

[0096] S43. For the queue of layers to be processed, based on the coordinate information of the intermediate neurons to be pruned recorded in the index set, perform input dimension reduction on the weight matrices of the gated projection layer and the upstream projection layer, and delete the row vectors of the weight matrices corresponding to the intermediate neurons to be pruned.

[0097] In this step, the weight matrix of each FFN layer is physically and permanently modified according to the global set of indices to be pruned: For the gate_proj and up_proj layers, delete the corresponding rows in their weight matrices.

[0098] For the down_proj layer, delete the corresponding column in its weight matrix.

[0099] S44. Simultaneously, perform output dimension reduction on the weight matrix of the downstream projection layer, and delete the weight matrix column vector corresponding to the intermediate neurons to be pruned.

[0100] S45. Based on the original weight state backup, perform inter-layer connection dimension matching verification on all target feed-forward network layers that have completed the reduction operation. The inter-layer connection dimension matching verification includes: performing a structured check on the input / output matrix dimensions between adjacent layers of the neural network to ensure that the weight matrix after the pruning operation maintains the mathematical computability of the feed-forward calculation chain when transmitted across layers.

[0101] S46. If the dimension matching verification passes, the reduced weight matrix is ​​atomically replaced in batches to the corresponding network layer; if the dimension matching verification fails, the transaction rollback mechanism is automatically triggered to restore the weight matrices of all target feedforward network layers to the original backup state.

[0102] S47. After completing the one-time structured row and / or column dimension deletion of all target feed-forward network layers, a compressed model without masking is generated. The new model obtained at this time is smaller in size and can be directly used for subsequent fine-tuning or inference without any masking.

[0103] In a specific embodiment, dynamic autoregressive pruning is implemented based on multiple models of the InternVL3-2B and Qwen3 series. The specific steps are as follows: First, prepare the system and environment, instantiate the model, and load the pre-trained InternVL3-2B model, along with its visual encoder and word segmenter. If the model is distributed across multiple GPUs, record the device mappings for each layer to ensure hardware consistency for subsequent calculations and data collection.

[0104] Next, prepare a calibration data set D, which contains M samples, D={S1,…,S M}. Each sample S m =(I m , P m ), where I m is the image data, P m The prompt is a text prompt; furthermore, all FFN layers in the model that need to be pruned are identified. For example, for the language model of InternVL3-2B, this corresponds to the mlp module in each layer, which contains three linear sub-layers: gate_proj, up_proj, and down_proj. For the language model of InternVL3-2B, this corresponds to three linear sub-layers: w1, w2, and w3. The Qwen3 series uses a similar process and will not be further explained.

[0105] Secondly, traditional pruning methods usually only extract activation values ​​in the pre-filling stage, but the dynamic behavior of the model during the autoregressive generation process is more informative. This paper uses hook technology to capture the internal state of the decoding stage in real time. Taking InternVL3-2B as an example, the specific process is detailed as follows: (1) Layer encapsulation: Define and instantiate a Wrapper class. This class contains member variables for online statistical calculations, including: Norm_l2 / Norm_l1: used to save and update L2 / L1 norm.

[0106] min_vals, max_vals: used to determine the minimum value of the activation value.

[0107] activation_input_matrix: used to store the activation value for generating tokens.

[0108] Create a Wrapper instance for all layer targets gate_proj, up_proj, down_proj sublayers in the model and store them in the dictionary.

[0109] (2) Hook registration and execution: Traverse each sample S in the dataset D m =(I m , P m ), for each sample, execute the following loop: a. Registration hook: Before all samples are processed, a forward hook is registered for all target sub-layers.

[0110] b. Execute generation: call the generate function of the model, and input the sample S m The model will first perform a prefill, then enter the autoregressive decoding loop to generate new tokens one by one. The key point is that the wrapper distinguishes between the prefill and generate phases by checking the sequence length L of the input activation tensor. It only records activation values ​​when the sequence length L is 1 (i.e., in the decoding loop), ensuring that the collected data strictly comes from the dynamic generation process.

[0111] c. Data Accumulation: During the entire generation process, the hook is triggered at each decoding step. The wrapper instance continuously accumulates statistical data (recording channel activation values, updating the L2 norm, and updating various statistical values ​​such as the maximum, variance, and mean as needed).

[0112] d. Remove hooks: After the generate process of all samples is completed, all hooks are removed uniformly.

[0113] (3) The above data collection process only needs to be performed once: traversing the entire dataset D once. At this stage, the wrapper determines the activation value of each channel and completes the calculation of statistical values ​​such as the L2 norm.

[0114] Then, after completing the statistical data collection, the amplitude importance score and entropy importance score are calculated for each FFN layer according to the specific importance calculation content in the above step S2, and a unified importance score vector is generated through weighted fusion.

[0115] After that, structured pruning is performed to sort all intermediate feature channels according to the final importance score vector and determine the channel index to be pruned. Synchronous pruning is implemented as follows: down_proj: Remove the columns in its weight matrix that correspond to the indices of the channels to be pruned.

[0116] gate_proj: Remove the rows in its weight matrix corresponding to the indices of the channels to be pruned.

[0117] up_proj: Remove the rows in its weight matrix that correspond to the indices of the channels to be pruned.

[0118] Accordingly, the bias vectors of gate_proj and up_proj are clipped.

[0119] pass Figure 8 and Figure 9 The comparative experiment can deeply analyze the effectiveness of different pruning strategies. The experiment prunes the model using the SOTA method based only on amplitude and the method of the present invention, and records the activation value response variance of the channel to be pruned. The results show: The variance of the channels pruned using the amplitude-only method is significantly higher than that of the proposed method. This indicates that channels deemed "unimportant" by traditional amplitude methods still exhibit differential responses to different input samples, implying potential information value. Directly pruning them may lead to a loss in model performance.

[0120] The variance of the pruned channels by our method approaches zero, validating the rigor of its screening criteria—the channels marked as redundant are indeed low-response, low-information regions. Furthermore, the variance distribution trends of the highly important channels screened by the two methods closely match, demonstrating that the core functional channels of the model are stable and identifiable, regardless of whether they are based on amplitude or comprehensive evaluation.

[0121] refer to Figure 12 , which shows the amplitude and entropy distribution of neurons retained and deleted in a certain layer when the pruning ratio is 50%, further revealing the screening logic of different strategies: Pure amplitude pruning ( Figure 12(Shown in the middle rectangle): While retaining high-amplitude channels, a large number of high-entropy channels (rich in information but low in amplitude) are mistakenly deleted. These channels may play a key role in complex semantic reasoning, and their loss will affect the model's generalization ability.

[0122] Pure entropy pruning ( Figure 12 (Image in the upper middle triangle): While retaining high-entropy channels, high-amplitude channels (with high activation intensity but redundant information) are mistakenly removed, resulting in a decrease in the basic feature expression ability of the model.

[0123] The present invention ( Figure 12 (Figure 1) The lower-middle triangle comprehensively evaluates amplitude and entropy, pruning only the dual-redundant channels with low amplitude and low entropy, while retaining the channels with high amplitude (critical for the function) and high entropy (rich in information), achieving the optimal balance between performance and efficiency.

[0124] Next, to verify the effectiveness of our method, referring to Experimental Results 1, we tested an embodiment of the present invention on a professionally fine-tuned model whose core task was to recognize chart information and output structured JSON data as required. The experiment used 400 charts as test samples, and compared them using the following schemes: amplitude only, entropy only, and our proposed scheme. For a more intuitive comparison, we also compared unstructured state-of-the-art schemes (e.g., Wanda).

[0125] Evaluation indicators are strictly divided into: 1. Format accuracy: If the format is inaccurate, that is, it cannot be correctly parsed into JSON format, the content accuracy of the table is 0.

[0126] 2. Content accuracy: Compare the accuracy of key-value pairs in the chart, taking into account the accuracy of both key values ​​and key names.

[0127] 3. Real-time performance comparison: Comparing the results of pruning rates from 10% to 70%, without recovery training.

[0128] 4. Comparison of recovery performance: The results with pruning rates of 10 and 50% participated in the recovery training.

[0129] The experimental results can be referred to Figure 10 and Figure 11 ,whether pruning immediate performance or recovery training performance, the method provided by the ,presentation outperforms other existing solutions used as comparison benchmarks. ,In high pruning rate scenarios (above 60%), it can even approach or surpass the ,unstructured solutions.

[0130] To further validate the generalization capabilities of the pruned model, we tested its performance on common test metrics. Using the original unpruned model (Base) as a benchmark, we compared performance degradation at different pruning rates. As shown in Tables 1 through 4, our structured pruning solution performs at least as well as existing state-of-the-art solutions across all metrics. This structured pruning solution is not only applicable to specialized graph parsing tasks but also demonstrates significant advantages in general language tasks.

[0131] Table 1

[0132] Table 1: Zero-shot performance of the model on the language reasoning benchmark test. The model was calibrated on the Alpaca (Qwen3-1.7B) and VQA (InternVL3-2B) datasets respectively. Magnitude represents the method of applying the pure amplitude pruning standard to structured pruning. The principle is derived from Wanda. Esparse is another method that combines amplitude and entropy for pruning. It mainly uses entropy as a correction term for amplitude. Entropy is a pure entropy pruning scheme, and Ours is the scheme of the present invention. ARC-c, ARC-e, BoolQ, HellaSwag, OBQA, RTE, and WinoGrande are all benchmarks related to natural language processing, which are used to measure the language generation ability of a model from the perspective of multiple tasks. The results show that the pruning performance of the present invention is better than the SOTA pruning method whether it is on a large language model or a multimodal model.

[0133] Table 2

[0134] Table 2: This table reports the perplexity and average inference accuracy of the language benchmark on WikiText2 for models from the Qwen (LLM) and InternVL (VLM,*) series with different parameter sizes. This table clearly demonstrates that the proposed method is also a better choice when applied to larger models.

[0135] Table 3

[0136] Table 3: AI2D, ChartQA, GQA, MME-cog, MME-per, and OKVQA are all benchmarks related to multimodal tasks, used to measure a multimodal model's ability to complete image-text related tasks from the perspective of multiple tasks. This result further verifies the versatility and advantages of our method.

[0137] Table 4

[0138] Table 4: This table records the average accuracy of pruned models on language benchmarks after performing structured and unstructured pruning by capturing activation values ​​at different inference stages. The results in this table show that capturing activation values ​​at the pre-filling stage is only suitable for unstructured pruning methods. Capturing activation values ​​at the generation stage is necessary for structured pruning to achieve better results.

[0139] Additionally, an embodiment of the present invention provides a global one-time structured pruning system for neural networks based on a hybrid amplitude entropy importance map, comprising: a tensor capture module for capturing the input activation tensors of all target layers in parallel in the neural network to be pruned using a calibration data set through a single forward propagation. A map output module for synchronously calculating the differential entropy index and amplitude response intensity for the associated weights of the intermediate neurons of each target layer based on the input activation tensor, and forming a static global importance map after normalization and fusion. A neuron screening module for determining the importance threshold according to a preset pruning rate, and generating an index set of all global intermediate neurons to be pruned whose scores are lower than the importance threshold at one time based on the global importance map. A structured pruning module for performing a one-time physical structured pruning on the weight matrix of each target layer based on the index set.

[0140] Then, an embodiment of the present invention provides a global one-time structured pruning device for a neural network based on a hybrid amplitude entropy importance map, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the global one-time structured pruning method for a neural network based on a hybrid amplitude entropy importance map as described above.

[0141] Furthermore, an embodiment of the present invention provides a computer-readable storage medium having computer-executable instructions stored thereon. When the executable instructions are executed by a processor, the global one-time structured pruning method of a neural network based on a hybrid amplitude entropy importance map as described above is implemented.

[0142] In summary, the embodiments of the present invention provide a global one-time structured pruning method, system, device and medium for a neural network, which realizes efficient model compression through the following logical links: First, a single forward propagation is performed in the neural network to be pruned using a calibration data set, and the original input activation tensors of all target layers are captured in parallel to ensure that all calculations are based on the unmodified initial model state. Then, for the neuron weight group of each target layer, the differential entropy index and the amplitude response intensity are calculated synchronously, and a global static importance map is generated after normalization and dynamic fusion. Subsequently, the importance threshold is determined according to the preset pruning rate, and the global intermediate neuron index set to be pruned that is below the threshold is directly screened out from the map, and finally a one-time physical pruning operation is performed on all target layer weight matrices.

[0143] The core advantages of this approach are reflected in three aspects: First, the method of the present invention only performs one forward propagation on the original, unmodified, and completely error-free model. The importance calculations of all layers are based on their most ideal and accurate input activations. There is no dependent calculation, and the importance calculation of the L+1 layer is completely unrelated to the pruning decision of the L layer. They are computationally parallel and independent. The L+1 layer sees the original input, and the L layer also sees the original input. There is no "error propagation" path between them. What is calculated is a complete "global importance map" that represents the original state of the model. Pruning decisions are made based on this single, uncontaminated true map. All neurons to be deleted are identified at one time and then executed at one time. There is no chance for the pruning error of one layer to affect the decision-making process of the next layer.

[0144] Second, structured pruning directly reduces the parameter size without introducing complex mask management or special hardware adaptation, significantly reducing deployment complexity.

[0145] Third, based on the solid theoretical foundation of "relative information entropy ranking stability," pruning ensures its effectiveness, significantly reducing model complexity while preserving model performance to the greatest extent possible. It also incorporates token information from the autoregressive generation process to avoid performance loss caused by solely considering pre-populated text prompts. This makes it particularly suitable for pruning and compression of specialized models (i.e., models fine-tuned for specialized fields or vertical categories). (Pruning tests on general models show continued performance improvement over existing state-of-the-art structured pruning methods.)

[0146] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each process flow and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions.

[0147] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments after learning the basic creative concepts. Therefore, the technical solutions should be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0148] Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if such modifications and variations fall within the scope of the technical solution of the present invention and its equivalents, the present invention shall also include such modifications and variations.

Claims

1. A global one-shot structured pruning method for neural networks based on hybrid amplitude entropy importance maps, characterized by: include: Through a single forward propagation, the calibration dataset is used to capture the input activation tensors of all target layers in parallel in the neural network to be pruned; According to the input activation tensor, the correlation weights of the intermediate neurons in each target layer are reorganized to synchronously calculate the differential entropy index and amplitude response strength, and after normalization and fusion, a static global importance map is formed; Determine the importance threshold according to the preset pruning rate, and generate the index set of all global intermediate neurons to be pruned with scores lower than the importance threshold at one time based on the global importance map; Based on the index set, a one-time physical structured pruning is performed on the weight matrix of each target layer.

2. The global one-time structured pruning method for a neural network based on a hybrid amplitude entropy importance map according to claim 1, characterized in that: Through a single forward pass, the calibration dataset is used to capture the input activation tensors of all target layers in parallel in the neural network to be pruned, including: Load the selected neural network to be pruned and obtain a calibration dataset by randomly sampling from the original training data; Input the calibration data set into the neural network to be pruned and perform a single forward propagation calculation; During the forward propagation process, the pre-implanted hooks are used to intercept the input activation tensors of each target feedforward network layer in parallel at one time, and the input activation tensors are iteratively stored to ensure that only a single stored result of the fusion of the current input activation tensor and the historical input activation tensors is always retained; Among them, the hidden state vector of each word in the input activation tensor is generated based on the corresponding context in the decoding stage and is used to predict the final output word. The word hidden state vector is the smallest operable data unit in the input activation tensor that carries independent semantic context information.

3. The global one-time structured pruning method for a neural network based on a hybrid amplitude entropy importance map according to claim 1, characterized in that: According to the input activation tensor, the correlation weights of the intermediate neurons in each target layer are reorganized to synchronously calculate the differential entropy index and amplitude response strength. After normalization and fusion, a static global importance map is formed, including: Configure the associated weight group for each intermediate neuron of the target feedforward network layer, where the associated weight group contains the specified row of the gated projection layer weight matrix, the specified row of the upstream projection layer weight matrix, and the specified column of the downstream projection layer weight matrix; Based on the input activation tensor, the product matrix of each weight in the associated weight group and the activation response strength index of the corresponding feature channel is calculated, and the aggregation operation is performed according to the input or output dimension to generate the amplitude response strength score; In the descending projection layer, the intermediate activation tensor generated by the modulation of the gated projection and the upward projection output is reconstructed in two dimensions and its feature channels are separated. The differential entropy index of each intermediate neuron is calculated by kernel density estimation and numerical integration. The amplitude response intensity score and differential entropy index were normalized respectively; The normalized amplitude results are weightedly fused with the differential entropy results to generate a static global importance map that represents the undisturbed intrinsic information.

4. The global one-time structured pruning method for a neural network based on a hybrid amplitude entropy importance map according to claim 3, characterized in that: Based on the input activation tensor, the product matrix of each weight in the associated weight group and the activation response strength index of the corresponding feature channel is calculated, and the aggregation operation is performed according to the input or output dimension to generate the amplitude response strength score including: Locate the weight matrix row corresponding to the target neuron in the gated projection layer, traverse the weight elements corresponding to all input channels of the weight matrix row, multiply the absolute value of each weight element by the L2 norm of the feature channel corresponding to the input activation tensor, and accumulate the sum along the input feature channel direction to obtain the gated row importance score; Locate the weight matrix row corresponding to the target neuron in the upstream projection layer, traverse the weight elements corresponding to all input channels in the weight matrix row, multiply the absolute value of each weight element by the L2 norm of the feature channel corresponding to the input activation tensor, and accumulate the sum along the input feature channel direction to obtain the upstream row importance score; Locate the weight matrix column corresponding to the target neuron in the downstream projection layer, traverse the weight elements corresponding to all output channels of the weight matrix column, multiply the absolute value of each weight element by the L2 norm of the feature channel corresponding to the intermediate activation tensor, and accumulate the sum along the output feature channel to obtain the downstream column importance score; the intermediate activation tensor is obtained by performing a Hadamard product operation on the output of the upstream projection layer after being transformed by the SiLU function of the gated projection layer. The gated row importance score, the upper row importance score, and the lower column importance score are summed to generate an amplitude response strength score.

5. The global one-time structured pruning method for a neural network based on a hybrid amplitude entropy importance map according to claim 3, characterized in that: In the down-projection layer, the intermediate activation tensor generated by the modulation of the gated projection and the up-projection output is reconstructed in two dimensions and its feature channels are separated. The differential entropy indicators of each intermediate neuron are calculated by kernel density estimation and numerical integration, including: In the down-projection layer, the intermediate activation tensor generated in a three-dimensional form based on the modulation of the gated projection and the up-projection output is reconstructed along the batch and sequence dimensions into a two-dimensional matrix with independent observation samples as rows and feature dimensions as columns; Extract the activation value set corresponding to each intermediate neuron in the two-dimensional matrix column by column; Perform kernel function smoothing on the activation value set of each intermediate neuron to construct a continuous probability density function that characterizes the distribution characteristics of the neuron's activation value; Based on the continuous probability density function, the logarithmic probability weighted integral of the activation value distribution is calculated within the preset numerical integration interval, and the differential entropy index is approximately calculated by numerical integration.

6. The global one-time structured pruning method for a neural network based on a hybrid amplitude entropy importance map according to any one of claims 1 to 5, characterized in that: The importance threshold is determined according to the preset pruning rate, and the index set of all global intermediate neurons to be pruned with scores lower than the importance threshold is generated at one time based on the global importance map, including: According to the preset network pruning ratio, percentile statistical operations are performed on the global map containing the normalized importance scores of all neurons to determine the corresponding global importance threshold; Scan the global importance map element by element, and compare the numerical relationship between the importance score of each neuron and the global importance threshold in real time; The position information of neurons whose importance scores are continuously lower than the global importance threshold is synchronously recorded into a unified pruning index container across all target feedforward network layers to form an index set of global intermediate neurons to be pruned.

7. The global one-time structured pruning method for a neural network based on a hybrid amplitude entropy importance map according to any one of claims 1 to 5, characterized in that: Based on the index set, a one-time physical structured pruning is performed on the weight matrix of each target layer, including: The target feedforward network layer containing the intermediate neurons to be pruned is retrieved layer by layer along the depth direction of the network to generate a queue of layers to be processed; Create a transactional temporary mirror copy of the weight matrix of each target feedforward network layer in the queue of pending layers to establish a backup of the original weight state; For the queue of layers to be processed, based on the coordinate information of the intermediate neurons to be pruned recorded in the index set, the weight matrices of the gated projection layer and the upstream projection layer are reduced in input dimension, and the row vectors of the weight matrices corresponding to the intermediate neurons to be pruned are deleted; At the same time, the output dimension of the weight matrix of the downstream projection layer is reduced, and the weight matrix column vector corresponding to the intermediate neurons to be pruned is deleted; Based on the original weight state backup, perform preset inter-layer connection dimension matching verification on all target feedforward network layers that have completed the reduction operation; If the dimension matching verification passes, the reduced weight matrix is ​​atomically replaced in batches to the corresponding network layer; If the dimension matching verification fails, the transaction rollback mechanism is automatically triggered to restore the weight matrices of all target feedforward network layers to the original backup state; After completing a one-time structured row and / or column dimension removal for all target feed-forward network layers, a compressed model without masking operations is generated.

8. A global one-shot structured pruning system for neural networks based on hybrid amplitude entropy importance maps, characterized by: include: A tensor capture module is used to capture the input activation tensors of all target layers in parallel in the neural network to be pruned using the calibration dataset through a single forward propagation; The atlas output module is used to reorganize the associated weights of the intermediate neurons in each target layer according to the input activation tensor, and synchronously calculate the differential entropy index and amplitude response strength. After normalization and fusion, a static global importance atlas is formed; The neuron screening module is used to determine the importance threshold according to the preset pruning rate and generate an index set of all global intermediate neurons to be pruned whose scores are lower than the importance threshold at one time based on the global importance map; The structured pruning module is used to perform one-time physical structured pruning on the weight matrix of each target layer based on the index set.

9. A global one-time structured pruning device for neural networks based on a hybrid amplitude entropy importance map, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the global one-time structured pruning method of a neural network based on a hybrid amplitude entropy importance map as described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that: When the executable instructions are executed by the processor, the global one-time structured pruning method of the neural network based on the hybrid amplitude entropy importance map as described in any one of claims 1 to 7 is implemented.

Images