Adaptive pruning method for large model text processing task

Abstract

The application discloses an adaptive pruning method for a large model text processing task. Calibration data samples are extracted from a text calibration data set and input into a to-be-pruned model. Activation vectors at the input ends of each layer of the to-be-pruned model are collected. The amplitude correlation score is fused with the normalized independence suppression factor to obtain comprehensive pruning scores of each weight. The comprehensive pruning scores are sorted to obtain a comprehensive pruning score set. The weights corresponding to the comprehensive pruning scores are set to zero to realize pruning of the to-be-pruned model. The application can reasonably allocate the pruning proportion of different network layers on the basis of analyzing the response characteristics of the model under text semantic input, effectively identify and suppress redundant parameter structures within the layers, thereby reduce the calculation and storage overhead in the model inference stage without significantly impairing the text generation and text understanding capabilities, and improve the deployment efficiency and application feasibility of the large model in an actual text processing system.

Description

Technical Field

[0001] This invention relates to the field of model pruning technology, and in particular to an adaptive pruning method for large model text processing tasks. Background Technology

[0002] With the rapid development of artificial intelligence technology, large-scale language models (LLMs) have become the core foundational models in the field of natural language processing, and are widely used in text processing tasks such as text generation, text understanding, information extraction, semantic matching, and dialogue systems. These models are typically based on deep neural network structures such as Transformers, and model text sequences through multi-layer self-attention mechanisms and feedforward networks. They can effectively capture the contextual semantic relationships in the text, thereby significantly improving the performance of text processing tasks.

[0003] However, in order to achieve stronger text modeling capabilities and better generalization performance, the parameter size of large-scale language models continues to grow, typically reaching billions or even hundreds of billions of parameters. In practical text processing applications, large models usually perform token-by-token inference on text using an autoregressive approach, and their computational complexity and memory usage are highly correlated with the model parameter size and text length. This often leads to high inference latency, high computational resource consumption, and high deployment costs for large models in scenarios such as text generation, long text understanding, and real-time dialogue, severely restricting their application in computing-constrained environments or real-time text processing systems.

[0004] To reduce the inference cost of large models in text processing tasks, researchers have proposed various model compression techniques, including model pruning, weight quantization, knowledge distillation, and low-rank decomposition. Among these, model pruning, by removing parameter structures that contribute little or are redundant to the output, reduces the number of model parameters and computational complexity while maintaining text processing performance as much as possible, making it a crucial means to improve the efficiency of large model text inference. Compared to structural redesign or retraining, pruning methods typically offer advantages such as low implementation cost and high deployment flexibility, thus possessing significant application value in large model text processing scenarios.

[0005] Most existing model pruning methods evaluate the importance of model parameters based on metrics such as weight magnitude, gradient information, or activation statistics, and then perform pruning accordingly. However, these methods still have significant shortcomings when applied to text processing tasks using large-scale language models. On the one hand, many pruning methods apply a uniform pruning ratio to all layers of the model, failing to fully consider the differences in importance of different network layers in text semantic modeling and contextual information transmission. This can easily lead to over-pruning of key layers, resulting in a decrease in text generation quality or a degradation in text understanding performance. On the other hand, existing methods typically focus only on the numerical value of individual weights during the selection of parameters within layers, ignoring the correlation and redundancy between weight vectors. This can easily result in retaining a large number of parameter structures with similar orientations after pruning, leading to a decrease in the ability to represent text features. Summary of the Invention

[0006] Based on this, it is necessary to propose an adaptive pruning method, apparatus, computer equipment, and storage medium for large-scale text processing tasks to address the above problems.

[0007] An adaptive pruning method for large-scale text processing tasks, the method comprising:

[0008] Calibration data samples are extracted from the text calibration dataset and input into the model to be pruned. Activation vectors at the input of each layer of the model to be pruned are collected using multiple forward propagations. The sensitivity features of each activation dimension are statistically analyzed by sampling the activation vectors using multiple forward propagations.

[0009] The weight matrix of each layer of the pruning model is divided into multiple blocks according to a preset direction. Within each block, a number of sample vectors with no more than a preset number of samples are selected to form a sample set. Each sample vector in the sample set is normalized to obtain a normalized sample vector.

[0010] Multiple normalized sampling vectors form a normalized sampling vector matrix, and a correlation matrix between vectors is constructed based on the normalized sampling vector matrix;

[0011] Determine the vectors in the correlation matrix with vector The degree of correlation determines the average redundancy; and the average redundancy is determined based on the said average redundancy.

[0012] The hierarchical sensitivity index is determined based on the sensitivity features and parameter set; the magnitude correlation score of the weight is determined based on the sensitivity features and parameter magnitudes.

[0013] A set of hierarchical sensitivity indicators consisting of multiple hierarchical sensitivity indicators is used to determine a normalized hierarchical sensitivity indicator based on the hierarchical sensitivity indicators and the set of hierarchical sensitivity indicators; a set of hierarchical redundancy consists of multiple hierarchical redundancy, and a normalized hierarchical redundancy indicator is determined based on the hierarchical redundancy and the set of hierarchical redundancy.

[0014] Based on the normalized hierarchical sensitivity index and the normalized hierarchical redundancy index, a cross-layer structure evaluation index is determined, and the reverse weight ratio of the l-th layer of the model to be pruned is combined with the cross-layer structure evaluation index; multiple reverse weight ratios constitute a reverse weight ratio set; a normalized reverse weight ratio is determined based on the reverse weight ratio and the reverse weight ratio set.

[0015] Adjustments are made based on global sparsity constraints to obtain the target sparsity of each layer of the model to be pruned;

[0016] The independence inhibition factor is determined based on the vector in the correlation matrix, and a set of independence inhibition factors is formed by multiple independence inhibition factors; a normalized independence inhibition factor is determined based on the set of independence inhibition factors.

[0017] An independence adjustment coefficient is introduced, and the amplitude correlation score is fused with the normalized independence suppression factor to obtain the comprehensive pruning score for each weight. The multiple comprehensive pruning scores are sorted to obtain a comprehensive pruning score set. Starting from the smallest comprehensive pruning score, multiple comprehensive pruning scores with a preset proportion are selected from the comprehensive pruning score set, and the weights corresponding to the multiple comprehensive pruning scores are reset to zero to achieve pruning of the model to be pruned.

[0018] In one embodiment, the step of performing multiple forward propagation sampling on the activation vector to statistically analyze the sensitivity features of each activation dimension is achieved through the following expression:

[0019]

[0020] in, To activate the sensitivity features of a dimension, the higher the sensitivity, the greater the contribution of the parameters subsequently formed by that dimension; The average energy at which it is activated.

[0021] In one embodiment, the weight matrix of each layer of the model to be pruned is divided into multiple blocks according to a preset direction. Within each block, a number of sample vectors, not exceeding a preset number of samples, are selected to form a sample set. Each sample vector in the sample set is then normalized to obtain a normalized sample vector. This can be achieved using the following expression:

[0022]

[0023] , i=1,...,n

[0024] Where T is the sampling set; For sampling vectors; Let be the normalized sampled vector of the i-th sampled vector; It is a very small positive value to avoid numerical instability when the denominator is zero. The total number of vectors in the sample set;

[0025] The multiple normalized sampling vectors Constructing a normalized sampling vector matrix Based on the normalized sampling vector matrix Constructing the correlation matrix between vectors This can be achieved using the following expression:

[0026]

[0027] in, This is a correlation matrix; For multiple normalized sampling vectors The normalized sampling vector matrix formed.

[0028] In one embodiment, the determination of vectors in the correlation matrix with vector The degree of correlation determines the average redundancy; and the average redundancy is determined based on the average redundancy using the following expression:

[0029]

[0030]

[0031] in, This represents the average redundancy of each block. Representing the correlation matrix vectors in with vector The degree of correlation; For hierarchical redundancy; The number of blocks; This represents the total number of vectors in the sample set.

[0032] In one embodiment, the hierarchical sensitivity index is determined based on the sensitivity features and parameter set; the magnitude correlation score of the weights is determined based on the sensitivity features and parameter magnitudes through the following expression:

[0033]

[0034]

[0035] in, As a level-sensitive indicator, Sensitivity features for activating dimensions; For parameter amplitude; For parameter set; The score is related to the magnitude of the weights. For parameter amplitude; This is a sensitivity feature.

[0036] In one embodiment, a set of hierarchical sensitivity indicators comprising multiple hierarchical sensitivity indicators is used to determine a normalized hierarchical sensitivity indicator based on the hierarchical sensitivity indicators and the set of hierarchical sensitivity indicators; a set of hierarchical redundancy degrees comprising multiple hierarchical redundancy degrees is used to determine a normalized hierarchical redundancy indicator based on the hierarchical redundancy degree and the set of hierarchical redundancy degrees. This can be achieved using the following expression:

[0037]

[0038]

[0039] in, This is a normalized hierarchical sensitivity indicator; Normalized hierarchical redundancy index; As a level-sensitive indicator, For multiple levels of sensitivity indicators The set of hierarchical sensitivity indicators constituted; R represents the level of redundancy; R is the set of level redundancy consisting of multiple levels of redundancy. It is a very small positive value.

[0040] In one embodiment, the normalized hierarchical sensitivity index and the normalized hierarchical redundancy index are used. Determine the evaluation index for cross-layer structure, and combine it with the reverse weight ratio of the l-th layer of the model to be pruned in the cross-layer structure evaluation index; multiple reverse weight ratios constitute a reverse weight ratio set; the normalized reverse weight ratio is determined based on the reverse weight ratio and the reverse weight ratio set by the following expression:

[0041]

[0042]

[0043]

[0044] in, As an evaluation index for multi-layer structures; This is a normalized hierarchical sensitivity indicator; Normalized hierarchical redundancy index; The inverse weight ratio of the l-th layer of the model to be pruned; The evaluation index for the cross-layer structure of the l-th layer in the LLAMA-7B model; This represents the normalized inverse weighting ratio; The inverse weight ratio of the l-th layer of the model to be pruned; It is a set of reverse weight ratios consisting of multiple reverse weight ratios; It is a very small positive value.

[0045] In one embodiment, the adjustment based on the global sparsity constraint to obtain the target sparsity of each layer of the model to be pruned is achieved by the following expression:

[0046]

[0047] in, is the target sparsity of the l-th layer of the model to be pruned; This represents the normalized inverse weighting ratio; For global sparsity constraints; This represents the total number of layers in the model to be pruned.

[0048] In one embodiment, the step of determining the independence repression factor based on the vectors in the correlation matrix, wherein the independence repression factor set consists of multiple independence repression factors; and the independence repression factor set is based on the independence repression factor set. The normalized independence suppression factor is determined by the following expression:

[0049]

[0050]

[0051] in, It is an independent inhibitory factor; Correlation matrix Vectors in; Normalized independence inhibitory factor; It is an independent inhibitory factor; It is a set of independent repressor factors consisting of multiple independent repressor factors; It is a very small positive value.

[0052] In one embodiment, the introduction of an independence adjustment coefficient, which fuses the magnitude correlation score with the normalized independence suppression factor, yields a comprehensive pruning score for each weight, implemented through the following expression:

[0053]

[0054] in, To calculate the overall pruning score; The score is related to the magnitude of the weights. Normalized independence inhibitory factor; This is the independence adjustment coefficient.

[0055] This invention can rationally allocate the pruning ratio of different network layers based on the analysis of the model's response characteristics under text semantic input, and effectively identify and suppress redundant parameter structures within the layers. Thus, without significantly impairing text generation and text understanding capabilities, it reduces the computational and storage overhead of the model's inference stage, and improves the deployment efficiency and application feasibility of large models in practical text processing systems. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] in:

[0058] Figure 1 This is a flowchart of an adaptive pruning method for a large-model text processing task in one embodiment;

[0059] Figure 2 This is a structural block diagram of a computer device in one embodiment. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] To address the technical problems in the background art, this application provides an adaptive pruning method for large-model text processing tasks.

[0062] This invention aims to address the problems of high inference computation overhead, large storage resource consumption, and high deployment cost caused by the large number of model parameters in text processing tasks such as text generation and text understanding using large-scale language models. To address the issues of fixed inter-layer pruning ratios, neglect of the differences in importance of different network layers in text semantic modeling, and difficulty in effectively identifying intra-layer parameter redundancy in traditional pruning methods for text processing scenarios, this invention proposes a unified pruning framework that combines cross-layer adaptive sparsity rate allocation with intra-layer vector independence constraints. This framework enables the model to achieve efficient compression while maintaining stable text processing performance. By introducing a cross-layer structure evaluation mechanism and intra-block vector orthogonality constraints, this invention proposes a completely static, non-secondary fine-tuning, and directly deployable unified pruning framework. This allows the sparsed model to maintain high computational efficiency while achieving inference performance close to or even exceeding that of the original model. The overall process of this invention is as follows: Figure 1 As shown.

[0063] In the specific implementation of this invention

[0064] First, a text calibration dataset is selected for pruning statistics. This dataset simulates the input distribution of the model in real-world text processing tasks and may include natural language text, dialogue corpora, or task-related text samples. Several calibration data samples are input into the model to be pruned, and activation vectors at the input ends of each layer are collected using multiple forward propagations. Each activation vector originates from different input samples, reflecting the response changes of the current layer under different semantic scenarios. Statistical processing of these activation vectors yields activation sensitivity features representing input sensitivity, which describe the sensitivity of a layer's weight matrix to input perturbations and effectively reflect the differences in importance of hierarchical structures during information transmission.

[0065] In the structural sampling stage of the weight matrix, this invention employs block-based and sub-sampling techniques to construct vector redundancy features to characterize the degree of structural redundancy. Specifically, for the linear weight matrix of each layer in the model, it is divided into several sub-blocks with fixed column widths, and several column vectors are extracted as representative subsets within each block. These vectors are then normalized and a similarity matrix is ​​calculated to obtain the average relevance index for each block. The higher the relevance index, the stronger the redundancy between the column vectors of that layer, i.e., the weaker the vector independence; the lower the relevance, the stronger the structural representativeness of that layer, the more dispersed the directional distribution, and the greater its contribution to information encoding capability.

[0066] Based on the aforementioned activation sensitivity and vector redundancy characteristics, this invention further constructs a cross-layer structure evaluation index to characterize the relative importance of different layers in the overall structure. To achieve dynamic and reasonable allocation of cross-layer sparsity, this invention standardizes and fuses the input sensitivity and vector redundancy characteristics to form a unified layer evaluation value, used to measure the necessity of retaining that layer in the global structure. Layers with strong input sensitivity and low vector redundancy indicate that their parameters contribute more to the final output; this invention uses a structure evaluation mapping mechanism to give these layers a higher retention rate. Conversely, layers with high redundancy and weak sensitivity are given a higher pruning ratio.

[0067] To meet the global sparsity constraint required by the overall model, this invention uses normalized hierarchical structure evaluation results for linear mapping and centering balance. This ensures that the sparsity allocation of all layers meets the target sparsity in absolute value, but exhibits differentiated expressive capabilities between relative layers. The mapping process maintains consistency in the order of each layer while establishing a consistent monotonic relationship between the overall structure evaluation and the final sparsity allocation. Compared to the traditional fixed pruning ratio, this adaptive sparsity allocation method significantly reduces the performance degradation caused by over-pruning of important layers, while avoiding wasting parameter budget on unimportant layers.

[0068] During the pruning phase within the execution layer, this invention combines amplitude information and vector independence features to construct a comprehensive pruning index to select weight parameters that truly contribute to the structure. Specifically, within each sub-block of the weight matrix, this invention first calculates the amplitude correlation score of the original weights to reflect the basic importance of the parameters. Simultaneously, it calculates the correlation matrix of the column vectors within that block to characterize the degree of local redundancy. By converting this redundancy into a suppression factor, the weight scores corresponding to highly correlated (i.e., highly redundant) vector columns are appropriately weakened to avoid directional collapse or decreased semantic expressiveness in the text feature representation process after pruning.

[0069] The overall score is composed of both the magnitude score and the redundancy suppression factor. After forming the overall score within the layer, the weights of that layer are sorted according to the inter-layer sparsity determined in the previous step, and the higher-scoring weights are selected as the final non-zero weights to be retained. After pruning the low-scoring weights, a sparse weight matrix is ​​formed. Unlike traditional pruning methods that only rely on magnitude or gradient, the overall index of this invention can simultaneously capture the independence of local vector structures and their actual contribution, thereby avoiding the situation where a large number of repetitive directions are retained and the model's expressive power is limited.

[0070] Through the above implementation methods, this invention can achieve adaptive pruning of large-scale language models without secondary fine-tuning or additional training. The pruned model significantly reduces parameter size and inference computational complexity while maintaining good text semantic modeling capabilities and output stability, making it suitable for various text processing tasks such as text generation and text understanding. This implementation demonstrates that the method proposed in this invention can effectively balance model compression efficiency and text processing performance, and has high application value in the inference deployment scenarios of practical text processing systems.

[0071] The method described in this invention is applied to the weights of all Transformer layers, feedforward layers, and attention layers, achieving stable pruning under high sparsity while maintaining the model's inference accuracy and structural independence. By constructing local redundancy through block-subsampling, this invention effectively reduces the complexity of redundancy calculation, enabling efficient application on large-scale models. Through cross-layer structure evaluation metrics, this invention achieves adaptive sparsity allocation under global sparsity constraints. By introducing intra-block orthogonal constraints and redundancy suppression factors, this invention significantly improves the "directional collapse" problem in traditional amplitude pruning, enhancing the representational power after pruning. If needed, this invention can also be combined with various post-training quantization strategies to perform weight quantization on the basis of sparse structures, further improving inference efficiency. In scenarios combined with quantization, the cross-layer sparsity allocation provided by this invention can synergistically influence the control of quantization errors, enabling the final model to achieve a better balance between compression ratio and actual inference performance.

[0072] In one specific implementation, taking the LLaMA-7B model based on the Transformer architecture as an example, adaptive unstructured pruning is performed. First, a text calibration dataset is obtained for pruning statistics. This dataset is a subset of the publicly available text dataset C4, containing 128 sequences, each with 2048 tokens. The aim is to maintain consistency across different methods to ensure fair performance comparisons. Without updating any parameters, multiple forward propagations are performed on the model to collect statistical characteristics of the input activations at each layer. For the... The layer, whose input activation vector is denoted as After performing forward propagation on each calibration sample, a set of activations is obtained.

[0073] like Figure 1 As shown, in one embodiment, an adaptive pruning method for large-model text processing tasks is provided. This method can be applied to both terminals and servers; this embodiment illustrates its application to a terminal. The adaptive pruning method for large-model text processing tasks specifically includes the following steps:

[0074] S1: Extract calibration data samples from the text calibration dataset C4, input the calibration data samples into the model to be pruned, and use multiple forward propagations to collect the activation vectors at the input of each layer of the model to be pruned, i.e., the LLaMA-7B model. Perform multiple forward propagation sampling on the activation vectors to statistically analyze the sensitivity features of each activation dimension. ;

[0075] S2: Divide the weight matrix of each layer of the model to be pruned into multiple blocks according to a preset direction, and select a number of sample vectors within each block, not exceeding a preset number of samples, to form a sample set. For the sample set Each sample vector is normalized to obtain a normalized sample vector. ;

[0076] S3: Multiple normalized sampling vectors Constructing a normalized sampling vector matrix Based on the normalized sampling vector matrix Constructing the correlation matrix between vectors ;

[0077] S4: Determine the correlation matrix vectors in with vector The degree of correlation determines the average redundancy. ; and based on the average redundancy Determine the average redundancy ;

[0078] S5: Based on the aforementioned sensitivity features Determine the hierarchical sensitivity index using the parameter set. Based on the aforementioned sensitivity features and parameter amplitude Determine the magnitude of the weights related to the score ;

[0079] S6: Multiple Levels of Sensitivity Indicators The set of hierarchical sensitivity indicators Based on hierarchical sensitivity indicators and set of hierarchical sensitivity indicators Determine the normalized hierarchical sensitivity index Multiple levels of redundancy The set of hierarchical redundancy Based on hierarchical redundancy and hierarchical redundancy set Determine the normalized hierarchical redundancy index ;

[0080] S7: Based on the normalized hierarchical sensitivity index and the normalized hierarchical redundancy index Determine the evaluation indicators for multi-layer structures Combined with the aforementioned cross-layer structure evaluation indicators The reverse weight ratio of the l-th layer of the model to be pruned Multiple inverse weight ratios constitute an inverse weight ratio set. Based on the aforementioned reverse weight ratio and the inverse weight ratio set Determine the normalized inverse weight ratio ;

[0081] S8: Combining global sparsity constraints Adjustments were made to obtain the target sparsity of each layer of the model to be pruned. ;

[0082] S9: Based on the correlation matrix vectors in Identify independent inhibitory factors A set of independent repressor factors consisting of multiple independent repressor factors Based on the aforementioned independent inhibitory factor Independent inhibitory factor set Determine the normalized independence inhibitory factor ;

[0083] S10: Introduce an independence adjustment coefficient The magnitude-related score With normalized independent inhibitory factor By merging the results, a comprehensive pruning score for each weight is obtained. The multiple comprehensive pruning scores are sorted to obtain a comprehensive pruning score set. Starting from the smallest comprehensive pruning score, multiple comprehensive pruning scores with a preset proportion are selected from the comprehensive pruning score set, and the weights corresponding to the multiple comprehensive pruning scores are reset to zero, so as to prune the model to be pruned.

[0084] In one embodiment, the sensitivity features of each activation dimension are statistically analyzed by performing multiple forward propagation sampling on the activation vector. This can be achieved using the following expression:

[0085]

[0086] in, To activate the sensitivity features of a dimension, the higher the sensitivity, the greater the contribution of the parameters subsequently formed by that dimension; is the average energy of the activation values. By calculating the root mean square of the activation vector, we can measure the sensitivity of the layer to input perturbations. A higher value indicates a larger activation amplitude and greater sensitivity to gradient propagation; therefore, more parameters should be retained during sparsification.

[0087] In one embodiment, the weight matrix of each layer of the model to be pruned is divided into multiple blocks according to a preset direction, and a number of sampling vectors not exceeding a preset number of samples are selected in each block to form a sampling set. For the sample set Each sample vector is normalized to obtain a normalized sample vector. This can be achieved using the following expression:

[0088]

[0089] , i=1,...,n

[0090] in, For the sample set; For sampling vectors; Let be the normalized sampled vector of the i-th sampled vector; It is a very small positive value;

[0091] The multiple normalized sampling vectors Constructing a normalized sampling vector matrix Based on the normalized sampling vector matrix Constructing the correlation matrix between vectors This can be achieved using the following expression:

[0092]

[0093] in, This is a correlation matrix; For multiple normalized sampling vectors The normalized sampling vector matrix formed.

[0094] In one embodiment, the determination of the correlation matrix vectors in with vector The degree of correlation determines the average redundancy. ; and based on the average redundancy Determine the average redundancy This can be achieved using the following expression:

[0095]

[0096]

[0097] in, This represents the average redundancy of each block. Representing the correlation matrix vectors in with vector The degree of correlation; For hierarchical redundancy; The number of blocks; This represents the total number of vectors in the sample set.

[0098] In one embodiment, the sensitivity feature is... Determine the hierarchical sensitivity index using the parameter set. Based on the aforementioned sensitivity features and parameter amplitude Determine the magnitude of the weights related to the score This can be achieved using the following expression:

[0099]

[0100]

[0101] in, As a level-sensitive indicator, Sensitivity features for activating dimensions; For parameter amplitude; For parameter set; The score is related to the magnitude of the weights. For parameter amplitude; This is a sensitivity feature.

[0102] In one embodiment, the multiple hierarchical sensitivity indicators The set of hierarchical sensitivity indicators Based on hierarchical sensitivity indicators and set of hierarchical sensitivity indicators Determine the normalized hierarchical sensitivity index Multiple levels of redundancy The set of hierarchical redundancy Based on hierarchical redundancy and hierarchical redundancy set Determine the normalized hierarchical redundancy index This can be achieved using the following expression:

[0103]

[0104]

[0105] in, This is a normalized hierarchical sensitivity indicator; Normalized hierarchical redundancy index; As a level-sensitive indicator, For multiple levels of sensitivity indicators The set of hierarchical sensitivity indicators constituted; R represents the level of redundancy; R is the set of level redundancy consisting of multiple levels of redundancy. It is a very small positive value.

[0106] In one embodiment, the sensitivity index based on the normalized hierarchical level and the normalized hierarchical redundancy index Determine the evaluation indicators for multi-layer structures Combined with the aforementioned cross-layer structure evaluation indicators The reverse weight ratio of the l-th layer of the model to be pruned Multiple inverse weight ratios constitute an inverse weight ratio set. Based on the aforementioned reverse weight ratio and the inverse weight ratio set Determine the normalized inverse weight ratio This can be achieved using the following expression:

[0107]

[0108]

[0109]

[0110] in, As an evaluation index for multi-layer structures; This is a normalized hierarchical sensitivity indicator; Normalized hierarchical redundancy index; The inverse weight ratio of the l-th layer of the model to be pruned; The evaluation index for the cross-layer structure of the l-th layer in the LLAMA-7B model; This represents the normalized inverse weighting ratio; The inverse weight ratio of the l-th layer of the model to be pruned; It is a set of reverse weight ratios consisting of multiple reverse weight ratios; It is a very small positive value.

[0111] In one embodiment, the combination of global sparsity constraints Adjustments were made to obtain the target sparsity of each layer of the model to be pruned. This can be achieved using the following expression:

[0112]

[0113] in, is the target sparsity of the l-th layer of the model to be pruned; This represents the normalized inverse weighting ratio; For global sparsity constraints; This represents the total number of layers in the model to be pruned.

[0114] In one embodiment, the step of relying on the correlation matrix vectors in Determine independent inhibitory factors A set of independent repressor factors consisting of multiple independent repressor factors Based on the aforementioned independent inhibitory factor Independent inhibitory factor set Determine the normalized independence inhibitory factor This can be achieved using the following expression:

[0115]

[0116]

[0117] in, It is an independent inhibitory factor; Correlation matrix Vectors in; Normalized independence inhibitory factor; It is an independent inhibitory factor; It is a set of independent repressor factors consisting of multiple independent repressor factors; It is a very small positive value.

[0118] In one embodiment, the introduction of an independence adjustment coefficient The magnitude-related score With normalized independent inhibitory factor The results are then combined to obtain the overall pruning score for each weight. This can be achieved through the following expression:

[0119]

[0120] in, To calculate the overall pruning score; The score is related to the magnitude of the weights. Normalized independence inhibitory factor; This is the independence adjustment coefficient;

[0121] When the global sparsity is set to be... When the sparsity is 70%, the process of inter-layer sparsity allocation will revolve around this step, through... While differentiating the sparsity rate, we aim to bring the overall sparsity closer to the global target of 70%.

[0122] The core of the model pruning method is to construct an amplitude-sensitivity score, extract intra-block vector independence features, fuse them to obtain a comprehensive pruning score, calculate the pruning priority for each weight, and finally achieve precise pruning of redundant weights.

[0123] First, the magnitude sensitivity score is constructed: First, the weight matrix is ​​traversed, and the absolute value of each element is calculated to obtain the weight magnitude matrix. Then, for each input dimension... Sensitivity characteristics The square root operation is performed, and then the magnitude-sensitivity score matrix of each weight is calculated, which quantifies the core contribution of each weight.

[0124] Then, the extraction of the intra-block vector independence feature: first, the weight matrix... The input dimension is divided into several blocks. A weight submatrix of a given block is taken (each column represents an input vector within the block). L2 normalization is performed on each vector within the block to obtain a normalized intra-block matrix. Then, a correlation matrix is ​​constructed, and its diagonal elements are set to zero to eliminate interference from vector correlations. Finally, each column of the corrected matrix... Calculate the sum of squares of all elements to obtain the independence suppression factor for the corresponding vector, and then perform interval normalization on the suppression factors of all intra-block vectors (the formula is given, and the normalization result is...). It directly represents the degree of redundancy and the strength of independence of a vector; the larger the value, the higher the redundancy and the weaker the independence.

[0125] Finally, the comprehensive pruning score is constructed: first, a preset independence adjustment coefficient is loaded, and then for each input dimension... The normalized suppression factor is used to calculate the fusion term, which adjusts the weight importance based on vector redundancy. The higher the redundancy, the smaller the fusion term. Then, the amplitude sensitivity score matrix is... Multiply each column by the fusion term and calculate the overall pruning score of the weights in the overall pruning score matrix element by element. The lower the score, the more redundant the weights are and the more likely they are to be pruned.

[0126] Finally, based on the obtained comprehensive pruning score matrix, subsequent pruning operations are performed: the comprehensive scores of all weights in each layer are sorted in ascending order; weights at the top of the sorting order are removed according to a preset pruning rate; and this process is repeated for all layers until the overall sparsity reaches the expected target. By combining the core contribution and redundancy of the weights, precise pruning control of the model weights is achieved. The entire implementation process can be accessed through... Figure 2 The pseudocode is used to summarize this.

[0127] To verify the effectiveness of the proposed model pruning method based on cross-layer redundancy allocation and intra-block orthogonal constraints on large-scale language models, this invention selects the LLaMA series models as the baseline model and conducts perplexity evaluation on the WikiText-2 language modeling dataset. The experiments first compare the inference performance of different pruning methods under various model sizes and sparsities. Then, ablation experiments are designed to deeply analyze the importance of the cross-layer sparsity allocation strategy and the intra-block orthogonal constraint mechanism. All experiments are conducted under the same hardware environment and with the same amount of calibration dataset to ensure the comparability and reproducibility of the results.

[0128] Table 1 compares the perplexity experimental results of different methods on WikiText for the LLaMA and OPT models.

[0129]

[0130] In the main experiment, as shown in Table 1, the global sparsity constraint was set to 60% and 70% on the LLaMA-7B, LLaMA-2-7B, and LLaMA-2-13B models, respectively. The method of this invention was compared with existing representative unstructured pruning methods, including Magnitude pruning, Wanda, OWL, SDS, and DLP. The experimental results are shown in Table 1. As can be seen from the table, the method of this invention achieves optimal or near-optimal perplexity performance under all model sizes and all sparsity settings. Taking the LLaMA-7B model as an example, under a 60% global sparsity constraint, the perplexities of Magnitude, Wanda, OWL, SDS, and DLP are 559.88, 10.70, 9.35, 10.11, and 9.35, respectively, while the perplexity of the model after pruning using this invention is only 9.30, further decreasing from the best baseline. Especially under the high global sparsity constraint of 70%, the pruned model of this invention still maintains an extremely low perplexity of 20.77, while various methods, including Wanda, SDS, and DLP, show significant degradation. The method of this invention also shows consistent results on LLaMA-2-7B and LLaMA-2-13B, namely, under the same global sparsity constraint, the perplexity is consistently significantly lower than existing technologies, indicating that the cross-layer structure evaluation mechanism and intra-block orthogonal constraint proposed in this invention can effectively maintain the expressive power of the model under high global sparsity constraints. Furthermore, on the LLaMA-2-13B model, the perplexity of this invention reaches 15.23 under a 70% global sparsity constraint, further lower than DLP (16.20) and OWL (20.65), and achieves the lowest perplexity of 7.58 under a 60% global sparsity constraint, demonstrating that the method of this invention has good scalability and model size adaptability.

[0131] ablation experiment

[0132] Table 2 Experimental results of LLAMA-7B and LLaMA-2-7B under 70% global sparsity constraint

[0133]

[0134] To verify the two core mechanisms of this invention, as shown in Table 2, and their specific contributions to the overall performance improvement of the cross-layer sparsity allocation strategy and intra-block vector orthogonality constraints, this invention further designed an ablation experiment to compare three configurations: removing only the inter-layer sparsity allocation strategy, removing only the intra-block orthogonality constraints, removing both, and using the full version of the method. The experiment was conducted on both LLaMA-7B and LLaMA-2-7B with a sparsity of 70%, and the results are shown in Table 2. It is clear from the table that removing the cross-layer sparsity allocation significantly worsens the perplexity (86.90 for LLaMA-7B and 72.33 for LLaMA-2-7B), indicating that the differences in the importance of parameters across different layers cannot be ignored, and forcibly adopting a uniform sparsity will directly disrupt gradient flow and feature extraction. After removing the independence suppression factor, the perplexity is 24.83 and 28.74, which is still significantly better than the version that removes the inter-layer strategy, but still significantly lower than the complete method's 20.77 and 22.92. This indicates that local directional orthogonality plays a crucial role in suppressing local redundancy and maintaining information independence. When both mechanisms are removed simultaneously, the performance degrades sharply to 87.88 and 75.81, even worse than the basic pruning method Wanda. This demonstrates that both the cross-layer structure evaluation mechanism and the intra-block directional orthogonality constraint are indispensable components, and their combined use produces significant performance gains.

[0135] Overall, the main experiments and ablation analyses demonstrate that the model pruning method proposed in this invention exhibits significant advantages compared to different model sizes, sparsities, and baseline methods. Particularly under high sparsity (70%) conditions, it significantly reduces perplexity compared to mainstream methods such as Wanda, SDS, and DLP, effectively mitigating the problem of decreased model accuracy after pruning. Furthermore, the cross-layer sparsity allocation strategy and intra-block orthogonal constraints proposed in this invention both show independent and stable performance contributions in the ablation experiments. Their synergy significantly improves pruning quality and the stability of the text generation task. These experimental results fully demonstrate that this invention possesses excellent theoretical feasibility and practical application value, and can be widely applied to text processing and lightweight scenarios for large-scale language models.

[0136] Embodiment 2 of the present invention addresses the typical application of language modeling in text processing tasks using large language models. Building upon Embodiment 1, this embodiment provides an application of an adaptive pruning method for large-model text processing tasks, including:

[0137] The input text sequence information is obtained as the input of the target large model. Through language modeling, the completed text information is output according to the text context. During the inference optimization process, the target large model adopts the large model hierarchical pruning method based on pruning influence rate described in Example 1 to perform adaptive hierarchical pruning, so as to dynamically adjust the pruning ratio of each layer parameter, thereby maximizing the compression of model parameters while ensuring model performance.

[0138] Taking text generation as an example: the user inputs a text sequence into the model, which internally completes the text based on the context information of the input text and outputs the completed text back to the user. This adaptive pruning method can automatically and selectively retain or prune parameters based on the degree of influence of each layer's parameters on the output result, thereby significantly reducing the computational resources and time required for model operation.

[0139] For example, QWEN-2.57B, developed by Qwen, boasts approximately 2.57 billion parameters and delivers excellent performance in language modeling tasks. However, running this model on conventional hardware typically still requires high-performance GPU support. Applying this adaptive pruning method can reduce the number of parameters in a large model to about half of its original size while maintaining the output quality for language modeling tasks, thereby significantly reducing hardware resource requirements and improving inference efficiency.

[0140] As can be seen from the above embodiments, the adaptive pruning method for large model text processing tasks described in this embodiment can maintain the performance and output quality of the model in text generation tasks such as language modeling while reducing the computational overhead and hardware requirements of large models.

[0141] Figure 2 An internal structural diagram of a computer device in one embodiment is shown. This computer device can specifically be a terminal or a server. Figure 2 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program. When executed by the processor, this computer program enables the processor to implement an adaptive pruning method for large-model text processing tasks. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to implement an adaptive pruning method for large-model text processing tasks. Those skilled in the art will understand that… Figure 2The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0142] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0143] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An adaptive pruning method for large-scale text processing tasks, characterized in that, The method includes: Calibration data samples are extracted from the text calibration dataset and input into the model to be pruned. Activation vectors at the input of each layer of the model to be pruned are collected using multiple forward propagations. The sensitivity features of each activation dimension are statistically analyzed by sampling the activation vectors using multiple forward propagations. The weight matrix of each layer of the pruning model is divided into multiple blocks according to a preset direction. Within each block, a number of sample vectors with no more than a preset number of samples are selected to form a sample set. Each sample vector in the sample set is normalized to obtain a normalized sample vector. Multiple normalized sampling vectors form a normalized sampling vector matrix, and a correlation matrix between vectors is constructed based on the normalized sampling vector matrix; Determine the vectors and vectors in the correlation matrix The degree of correlation determines the average redundancy; and the average redundancy is determined based on the said average redundancy. The hierarchical sensitivity index is determined based on the sensitivity features and parameter set; the magnitude correlation score of the weight is determined based on the sensitivity features and parameter magnitudes. A set of hierarchical sensitivity indicators consisting of multiple hierarchical sensitivity indicators is used to determine a normalized hierarchical sensitivity indicator based on the hierarchical sensitivity indicators and the set of hierarchical sensitivity indicators; a set of hierarchical redundancy consists of multiple hierarchical redundancy, and a normalized hierarchical redundancy indicator is determined based on the hierarchical redundancy and the set of hierarchical redundancy. Based on the normalized hierarchical sensitivity index and the normalized hierarchical redundancy index, a cross-layer structure evaluation index is determined, and the reverse weight ratio of the l-th layer of the model to be pruned is combined with the cross-layer structure evaluation index; multiple reverse weight ratios constitute a reverse weight ratio set; a normalized reverse weight ratio is determined based on the reverse weight ratio and the reverse weight ratio set. Adjustments are made based on global sparsity constraints to obtain the target sparsity of each layer of the model to be pruned; The independence inhibition factor is determined based on the vector in the correlation matrix, and a set of independence inhibition factors is formed by multiple independence inhibition factors; a normalized independence inhibition factor is determined based on the set of independence inhibition factors. An independence adjustment coefficient is introduced, and the amplitude correlation score is fused with the normalized independence suppression factor to obtain the comprehensive pruning score for each weight. The multiple comprehensive pruning scores are sorted to obtain a comprehensive pruning score set. Starting from the smallest comprehensive pruning score, multiple comprehensive pruning scores with a preset proportion are selected from the comprehensive pruning score set, and the weights corresponding to the multiple comprehensive pruning scores are reset to zero to achieve pruning of the model to be pruned.

2. The adaptive pruning method for large-model text processing tasks according to claim 1, characterized in that, The process of sampling and calculating the sensitivity features of each activation dimension through multiple forward propagation of the activation vector is achieved by the following expression: in, To activate the sensitivity features of a dimension, the higher the sensitivity, the greater the contribution of the parameters subsequently formed by that dimension; The average energy at which it is activated.

3. The adaptive pruning method for large-model text processing tasks according to claim 1, characterized in that, The weight matrix of each layer of the model to be pruned is divided into multiple blocks according to a preset direction. Within each block, a number of sample vectors, not exceeding a preset number of samples, are selected to form a sample set. Each sample vector in the sample set is then normalized to obtain a normalized sample vector. This can be achieved using the following expression: ,i=1,...,n Where T is the sampling set; For sampling vectors; Let be the normalized sampled vector of the i-th sampled vector; It is a very small positive value to avoid numerical instability when the denominator is zero. The total number of vectors in the sample set; The multiple normalized sampling vectors Constructing a normalized sampling vector matrix Based on the normalized sampling vector matrix Constructing the correlation matrix between vectors This can be achieved using the following expression: in, This is a correlation matrix; For multiple normalized sampling vectors The normalized sampling vector matrix formed.

4. The adaptive pruning method for large-model text processing tasks according to claim 1, characterized in that, The vector in the correlation matrix is ​​determined with vector The degree of correlation determines the average redundancy; and the average redundancy is determined based on the average redundancy using the following expression: in, This represents the average redundancy of each block. Representing the correlation matrix vectors in with vector The degree of correlation; For hierarchical redundancy; The number of blocks; This represents the total number of vectors in the sample set.

5. The adaptive pruning method for large-model text processing tasks according to claim 1, characterized in that, The hierarchical sensitivity index is determined based on the sensitivity features and parameter set; the magnitude correlation score of the weights is determined based on the sensitivity features and parameter magnitudes through the following expression: in, As a level-sensitive indicator, Sensitivity features for activating dimensions; For parameter amplitude; For parameter set; The score is related to the magnitude of the weights. For parameter amplitude; This is a sensitivity feature.

6. The adaptive pruning method for large-model text processing tasks according to claim 1, characterized in that, The hierarchical sensitivity index set, which consists of multiple hierarchical sensitivity indices, is used to determine the normalized hierarchical sensitivity index based on the hierarchical sensitivity index and the hierarchical sensitivity index set. A set of hierarchical redundancy levels, comprising multiple levels of redundancy, is used to determine a normalized hierarchical redundancy index. This can be achieved using the following expression: in, This is a normalized hierarchical sensitivity indicator; Normalized hierarchical redundancy index; As a level-sensitive indicator, For multiple levels of sensitivity indicators The set of hierarchical sensitivity indicators constituted; R represents the level of redundancy; R is the set of level redundancy consisting of multiple levels of redundancy. It is a very small positive value.

7. The adaptive pruning method for large-model text processing tasks according to claim 1, characterized in that, The normalized hierarchical sensitivity index and the normalized hierarchical redundancy index are used as the basis for this. Determine the evaluation index for cross-layer structure, and combine it with the reverse weight ratio of the l-th layer of the model to be pruned in the cross-layer structure evaluation index; multiple reverse weight ratios constitute a reverse weight ratio set; the normalized reverse weight ratio is determined based on the reverse weight ratio and the reverse weight ratio set by the following expression: in, As an evaluation index for multi-layer structures; This is a normalized hierarchical sensitivity indicator; Normalized hierarchical redundancy index; The inverse weight ratio of the l-th layer of the model to be pruned; The evaluation index for the cross-layer structure of the l-th layer in the LLAMA-7B model; This represents the normalized inverse weighting ratio; The inverse weight ratio of the l-th layer of the model to be pruned; It is a set of reverse weight ratios consisting of multiple reverse weight ratios; It is a very small positive value.

8. The adaptive pruning method for large-model text processing tasks according to claim 1, characterized in that, The adjustment, which incorporates global sparsity constraints, yields the target sparsity of each layer in the model to be pruned, achieved through the following expression: in, is the target sparsity of the l-th layer of the model to be pruned; This represents the normalized inverse weighting ratio; For global sparsity constraints; This represents the total number of layers in the model to be pruned.

9. The adaptive pruning method for large-model text processing tasks according to claim 1, characterized in that, The independent suppression factor is determined based on the vector in the correlation matrix, and the set of independent suppression factors consists of multiple independent suppression factors; Based on the aforementioned set of independent inhibitory factors The normalized independence suppression factor is determined by the following expression: in, It is an independent inhibitory factor; Correlation matrix Vectors in; Normalized independence inhibitory factor; It is an independent inhibitory factor; It is a set of independent repressor factors consisting of multiple independent repressor factors; It is a very small positive value.

10. The adaptive pruning method for large-model text processing tasks according to claim 1, characterized in that, The introduction of an independence adjustment coefficient, which integrates the amplitude correlation score with the normalized independence suppression factor, yields a comprehensive pruning score for each weight, expressed as follows: in, To calculate the overall pruning score; The score is related to the magnitude of the weights. Normalized independence inhibitory factor; This is the independence adjustment coefficient.

Images