Lightweight inference acceleration large model layer pruning method and device, equipment and medium

Abstract

The application belongs to the field of computer artificial intelligence, and provides a large model layer pruning method and device for light inference acceleration, equipment and medium, wherein the method comprises the following steps: initializing model parameters and pruning states of an original model; obtaining an output tensor of a Transformer layer of the original model, performing layer-by-layer cosine similarity calculation on the Transformer layer according to the output tensor, and obtaining a similarity matrix between adjacent Transformer layer pairs; performing dynamic hierarchical clustering grouping processing on the similarity matrix by using a clustering algorithm to obtain a category clustering result of the Transformer layer; and processing the original model by using an iterative gradual layer pruning strategy according to a target function and the category clustering result to obtain a layer pruning result of the original model. The application has the beneficial effects of improving the inference speed of a large model and reducing the system resource consumption during model inference.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer artificial intelligence, and in particular to a light inference acceleration large model layer pruning method, device, equipment and medium. BACKGROUND

[0002] With the wide application of large language models (such as GPT-3, GPT-4, etc.), inference speed and resource consumption have become a bottleneck in real-world applications. Large language models (LLM) usually require a lot of computing resources during inference, especially in edge devices, mobile devices or real-time inference scenarios, which often cannot meet the demand for efficient inference, making its deployment and application face high cost and delay problems.

[0003] Currently, existing model compression technologies can be roughly divided into two main directions: model pruning and quantization. Model pruning is further divided into two different types: unstructured pruning and structured pruning. The strategy of unstructured pruning is to accurately remove individual weights, which can often achieve ideal performance improvement. However, due to the inherent shortcomings of existing hardware architecture in processing sparse weights, the goal of model acceleration is difficult to achieve. In contrast, structured pruning is a radical approach that removes entire pre-defined model structures at once. This approach may cause a slight drop in model accuracy, but it can greatly optimize hardware processing efficiency. In existing technologies, the Layer Ablation method simplifies the model structure by deleting redundant layers, but its core problem is:

[0004] Importance evaluation is one-sided, and a single indicator (such as gradient norm) cannot fully measure the impact of layers on the overall performance of the model; Similarity measurement is insufficient, and the functional redundancy of adjacent layers is not fully considered, leading to the deletion of key layers; Dynamic adjustment capability is weak, and static pruning strategies are difficult to adapt to the differentiated needs of different input tasks. SUMMARY

[0005] To at least solve one of the technical problems existing in the prior art, the present application provides a light inference acceleration large model layer pruning method, device, equipment and medium, which improves the inference speed of the large language model and reduces the system resource consumption during inference of the large language model.

[0006] One aspect of the present application provides a light inference acceleration large model layer pruning method, comprising:

[0007] Initializing the model parameters and pruning state of the original model, the original model being a large language model for question answering, the large language model being used to convert input text signals into semantic feature representations;

[0008] An output tensor of a Transformer layer of an original model is obtained, and a cosine similarity calculation is performed on the Transformer layer layer by layer according to the output tensor, to obtain an inter-layer similarity matrix of any adjacent Transformer layer pair, wherein the inter-layer similarity matrix is used to represent the semantic similarity of the large language model when inferring the input text signal; and the output tensor is used to represent the semantic feature distribution feature tensor obtained after the Transformer layer processes the input text signal;

[0009] According to the similarity matrix, a clustering algorithm is used to perform a dynamic hierarchical clustering grouping processing from bottom to top, to obtain a category clustering result of the Transformer layer;

[0010] According to the target function and the category clustering result, an iterative and progressive layer pruning strategy is used to process the original model, to obtain a layer pruning result of the original model, wherein the target function is used to guide the iterative and progressive layer pruning to minimize the target function under the premise of satisfying the double-threshold judgment, to represent that the model performance of the original model is not lower than a preset value and the number of Transformer layers retained after pruning is minimized; and the layer pruning is used to represent that the calculation instructions during inference of the Transformer layer are removed, and the occupied video memory space is released.

[0011] According to the layer pruning result of the original model, a modified original model is obtained, and the target problem sentence is inferred through the modified original model, to obtain a problem inference result.

[0012] According to the light-weight inference acceleration large model layer pruning method, the model parameters and the pruning state of the original model are initialized, including:

[0013] The model weights of the original model are frozen, and the original model is inferred through a question and answer test set to obtain an initial accuracy rate;

[0014] The initial accuracy rate is used as a baseline value to determine the initialized pruning state, wherein the initialized pruning state includes an accuracy difference threshold, an absolute accuracy lower limit, and a layer freezing judgment threshold, wherein the accuracy difference threshold represents the maximum decrease in model accuracy allowed by the Transformer layer deleted due to pruning; the absolute accuracy lower limit represents the absolute minimum value of the model accuracy allowed by the Transformer layer deleted due to pruning; and the layer freezing judgment threshold represents a threshold of the number of Transformer layers that are not deleted due to pruning, wherein the number of Transformer layers that are not deleted is determined by a counter dictionary set in each Transformer layer.

[0015] According to the lightweight inference acceleration large model layer pruning method, an output tensor of a Transformer layer of the original model is obtained, cosine similarity calculation is performed on the Transformer layer layer by layer to obtain a similarity matrix between adjacent Transformer layer pairs, including:

[0016] An output tensor of a Transformer layer of the original model is obtained , The sequence number of the Transformer layer is identified, wherein the output tensor Semantic feature aggregation is performed according to the sequence number of the Transformer layer to obtain

[0017] The output tensor is subjected to mean pooling processing:

[0018] ;

[0019] Wherein, The total time step of the mean pooling processing is represented by T, The output vector at the i-th time step is represented by xi, The output vector obtained after the mean pooling is represented by xi T.

[0020] The cosine similarity calculation is performed on the Transformer layer layer by layer to obtain a similarity matrix is:

[0021] ;

[0022] Wherein, The dot product operation of the output vectors of the i-th Transformer layer and the j-th Transformer layer after the mean pooling is represented by , and ; And represent adjacent Transformer layers, The norm of the vector is represented by ||.

[0023] According to the lightweight inference acceleration large model layer pruning method, a clustering algorithm is used to perform a bottom-up dynamic hierarchical clustering grouping processing according to the similarity matrix to obtain a category clustering result of the Transformer layer, including:

[0024] According to the similarity matrix, the similarity of adjacent Transformer layers is determined to be ;

[0025] According to the similarity​​ , the average value of the similarity between all adjacent Transformer layers in any two categories is calculated to obtain the inter-class similarity is:

[0026] ;

[0027] and represents a category pair composed of any two categories, each Transformer layer has a corresponding category, represents the number of Transformer layers including category . represents the number of Transformer layers including category . represents the Transformer layer conforming to category . represents the Transformer layer conforming to category ; wherein the category is used to classify the Transformer layers processing different question and answer tasks.

[0028] If the inter-class similarity between the category pair is greater than the preset adaptive threshold, the category pair is merged, the new category includes all Transformer layers of the category pair, and the category list is updated at the same time;

[0029] Repeat the dynamic hierarchical clustering grouping until no category pair meeting the merging condition can be found or the category list includes a preset number of categories, and obtain the category clustering result of the Transformer layers.

[0030] According to the lightweight inference acceleration large model layer pruning method, wherein the method further comprises:

[0031] The category clustering result of the Transformer layer is sequentially processed by using intra-class layer sorting and inter-class sorting to obtain a category sorting result, wherein the intra-class layer sorting is:

[0032] ;

[0033] wherein, represents the Transformer layer included in the clustered category is arranged in descending order according to its depth in the original model, and ; ;

[0034] The sorting of the intra-class layer sorting is executed by the inter-class sorting to obtain the category sorting result:

[0035] ;

[0036] wherein, represents the deepest depth of the clustered class is arranged in descending order from large to small, and , is the total number of classes obtained by clustering.

[0037] According to the lightweight inference acceleration large model layer pruning method, wherein according to the similarity matrix, a clustering algorithm is used to perform a bottom-up dynamic hierarchical clustering grouping process to obtain a class clustering result of the Transformer layer, including:

[0038] obtain a class ordering result wherein , only one class of the Transformer layer is included in the class ordering result, and the class including only one Transformer layer is added to the frozen layer set wherein the frozen layer set does not enter the candidate layer, i.e., does not perform a deletion operation;

[0039] initialize a counter dictionary of the Transformer layer , all Transformer layers record the number of consecutive unsuccessful pruning rounds, wherein ;

[0040] evaluate the candidate layer and make a pruning decision for each class of the Transformer layer of the copy of the original model according to the class ordering result:

[0041] the deepest layer is selected as the candidate layer for pruning, the candidate layer is added with a deletion identifier, and the output of the previous reserved layer of the candidate layer is transmitted to the input of the next reserved layer; compare the accuracy of the copy of the original model with the initial accuracy to obtain a model accuracy drop rate, if the model accuracy drop rate does not satisfy one of the accuracy difference threshold and the absolute accuracy lower limit, the candidate layer is not deleted; otherwise, the candidate layer is deleted;

[0042] perform pruning processing in a loop, and each time the loop is executed, freeze all classes with a class list length of 1, and add the Transformer layer in the class list length of 1 to the frozen class set , if no Transformer layer is deleted in the loop or all Transformer layers of the class are frozen, terminate the main iteration pruning loop.

[0043] According to the lightweight inference acceleration large model layer pruning method, the method comprises the following steps:

[0044] Wherein the objective function is:

[0045]

[0046] Wherein represents the model accuracy drop caused by layer ablation, is a binary variable, indicating whether the i-th layer is retained, represents the calculation amount of the i-th layer; is a hyperparameter, used to balance the relative importance of accuracy loss and model calculation;

[0047] Step 3: Step 2 is performed step by step to optimize the objective function, and at each step, whether to perform the deletion is determined according to the accuracy difference threshold.

[0048] Another aspect of the present application provides a lightweight inference acceleration large model layer pruning device, comprising:

[0049] The first module is used for initializing the model parameters and pruning state of the original model, wherein the original model is a large language model for question and answer, and the large language model is used for converting input text signals into semantic feature representations.

[0050] The second module is used for obtaining the output tensor of the Transformer layer of the original model, and performing cosine similarity calculation on the Transformer layer layer by layer according to the output tensor to obtain an inter-layer similarity matrix of any adjacent Transformer layer pair.

[0051] The third module is used for performing bottom-up dynamic hierarchical clustering grouping processing on the similarity matrix by using a clustering algorithm to obtain a category clustering result of the Transformer layer.

[0052] The fourth module is used for processing the original model by using an iterative and progressive layer pruning strategy according to the objective function and the category clustering result, to obtain a layer pruning result of the original model, wherein the objective function is used for iterative and progressive layer pruning guided by minimizing the objective function under the premise of satisfying the double-threshold judgment, to represent that the model performance of the original model is not lower than a preset value and the number of retained Transformer layers after pruning is minimized; and the layer pruning is used for representing the calculation instructions during inference of the removed Transformer layer, and releasing the occupied video memory space.

[0053] ​The fifth module is used to obtain a modified original model based on the layer pruning results of the original model, and then to reason about the target question statement using the modified original model to obtain the question reasoning result.

[0054] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0055] The memory is used to store programs;

[0056] The processor executes the program to implement the method as described above.

[0057] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the methods described above.

[0058] The beneficial effects of this invention are as follows: By capturing layer outputs and calculating similarity and dynamically clustering similar layers, when progressively pruning these similar layers, the large language model is ensured to reduce redundant computation while preserving performance; the large language model is effectively prevented from crashing through dual threshold verification (Transformer layer number threshold and model performance threshold) and a class-freezing strategy; and the connection compatibility is verified and the normalization layer is recalibrated after fine-tuning the large language model based on the accuracy difference threshold. When using the pruned large language model for question-answering inference, the number of parameters of the large language model is reduced, the efficiency and inference speed of the large language model are improved, and the system resource consumption during the inference of the large language model is reduced. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the large model layer pruning process for lightweight inference acceleration according to an embodiment of the present invention;

[0060] Figure 2 This is a comparison chart of the inference time of the original model and the pruned model in an embodiment of the present invention;

[0061] Figure 3 This is a graph showing the score curves of the original model and the pruned model in an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram of the overall process of pruning a large language model layer for lightweight inference acceleration according to an embodiment of the present invention;

[0063] Figure 5 This is a schematic diagram of a large model layer pruning device for lightweight inference acceleration according to an embodiment of the present invention. Detailed Implementation

[0064] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings. Throughout the description, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" can be used interchangeably. Terms such as "first," "second," etc., are used only to distinguish technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the sequential relationship of the indicated technical features. In the following description, the consecutive reference numerals for method steps are for ease of review and understanding. Adjusting the implementation order of steps, in conjunction with the overall technical solution of the present invention and the logical relationship between the various steps, will not affect the technical effect achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0065] refer to Figure 1 , Figure 1 This is a schematic diagram of the lightweight inference acceleration large model layer pruning process according to an embodiment of the present invention, which includes, but is not limited to, steps S100~S400:

[0066] S100: Initialize the model parameters and pruning state of the original model. The original model is a large language model for question answering, which is used to convert the input text signal into a semantic feature representation.

[0067] In some embodiments, semantic feature representation captures core semantic information of words, phrases, or sentences used to characterize input or question-and-answer sessions in a large language model, enabling the computer to perform mathematical processing and computation, such as context encoding and coding.

[0068] In some embodiments, large language models such as DeepSeek, GPT, etc., are used for question answering (natural language processing).

[0069] In some embodiments, the model weights of the original model are frozen, and the original model is used for inference on a question-answering test set to obtain an initial accuracy. The initial accuracy is used as a baseline value to determine the initial pruning state, which includes an accuracy difference threshold, an absolute accuracy lower bound, and a layer freeze judgment threshold. The accuracy difference threshold represents the maximum allowable decrease in model accuracy due to the removal of Transformer layers during pruning. The absolute accuracy lower bound is the absolute minimum value of model accuracy allowed due to the removal of Transformer layers during pruning. The layer freeze judgment threshold represents the threshold for the number of Transformer layers that were not removed during pruning, where the number of unremoved Transformer layers is determined by a counter dictionary set for each Transformer layer.

[0070] In some embodiments, the model weights of the large language model are feature representations learned by the model. By adjusting the weights, the model can understand the relationships and context between words. The Transformer layers corresponding to the model weight representations are not trained to reduce computation.

[0071] In some embodiments, the model weights of the large language model include processing the semantic features of the input text, with the weights capturing long-distance dependencies through a self-attention mechanism; and generating response text, with the weights determining the generation strategy and contextual associations.

[0072] In some embodiments, through a question-and-answer test set Construct a test set containing multiple questions. Load an original, unpruned model, keeping the model weights frozen to ensure the pruning process does not affect the original parameters. Define a string of length... A binary vector representing the total number of layers in the model. (Initialize the pruned state vector), initially all elements are 1, i.e. For all Established,

[0073] At the start of pruning, all layers of the model are preserved:

[0074] ;

[0075] Using the original, unpruned model In the test set Infer from above and record its initial accuracy. and original output In each subsequent pruning attempt, the performance of the current model will be used as a baseline for comparison.

[0076] Set accuracy difference threshold This is the maximum allowable decrease in model accuracy when attempting to remove a layer causes a drop in accuracy. If this happens, the deletion operation will be rejected.

[0077] Set a lower limit for absolute accuracy This is the absolute minimum allowable model accuracy, even if However, the trimmed model Model accuracy The deletion operation will also be rejected. These two thresholds together constitute a "dual threshold verification" mechanism to prevent model performance crashes caused by a single metric.

[0078] To avoid situations where certain categories remain undeleted across multiple iterations, leading to inefficient pruning, this embodiment of the invention initializes a counter dictionary for each layer. If a certain layer If a layer is not deleted in multiple consecutive iterations, and the number of undeleted layers exceeds the layer freezing threshold, then the layer freeze condition is considered to be in a state of failure. Then add this layer to the frozen category set. The pruning loop ends when no level is deleted or all categories are frozen during a single iteration.

[0079] In some embodiments, the accuracy difference threshold and the lower limit of absolute accuracy The settings vary depending on the type of text signal processing task being performed, such as the accuracy difference threshold for sentiment analysis question answering. and the lower limit of absolute accuracy It has a higher tolerance and can be set to a lower value, such as 0.3 and 0.7, while knowledge-based question answering has a lower tolerance and requires a higher value, such as 0.1 and 0.85.

[0080] S200, Obtain the output tensor of the Transformer layer of the original model, and perform cosine similarity calculation on the Transformer layer layer by layer based on the output tensor to obtain the inter-layer similarity matrix of any adjacent Transformer layer pair. The inter-layer similarity matrix is ​​used to characterize the semantic similarity of text when the large language model performs text signal inference.

[0081] In some embodiments, the inter-layer similarity of the Transformer layer mainly represents the semantic association strength between the query and the key, indicating that the large language model should focus on parts of the text input sequence when processing the words of the current question and answer, that is, dynamically capturing long-distance dependencies in the Transformer layer through the semantic association strength between the query and the key.

[0082] It should be noted that the large language model achieves vector transformation through weights in the Query matrix, Key matrix, and Value matrix:

[0083] The Query matrix converts the input vector into a retrieval instruction (Query), determining which contextual information to focus on.

[0084] The key matrix generates index labels (Keys) that identify the semantic features of each token.

[0085] Value matrix: Extracts the content value, providing a carrier of actual information.

[0086] The following is a text example of the inter-layer similarity matrix of a Transformer layer:

[0087] For example, in the question 1, "What is the highest mountain in the world?" and in the question 2, "What is the highest mountain in my country?", it can be determined that these two sentences are different in text, but by calculating their semantic vector similarity (e.g., a cosine similarity as high as 0.97), the large language model can determine that they are semantically equivalent questions, and thus can be answered with the same knowledge. This indicates that the two Transformer layers have a high degree of similarity.

[0088] The above embodiments only illustrate the similarity between questions. The similarity between questions and answers or between answers can also be calculated on the Transformer layer.

[0089] In some embodiments, forward hooks are registered at the output position of each Transformer block to obtain the output tensor of each layer without changing the original model structure and computation flow. Intermediate layer outputs are captured when inputting the test set, and the output tensors of the original model's Transformer layers are obtained. , This is the index identifier for the Transformer layer, where the output tensor... The semantic features are aggregated based on the sequence number of the Transformer layer.

[0090] For output tensor Perform mean pooling:

[0091] ;

[0092] in, This represents the total time steps for mean pooling. Indicates the first The output vector at each time step This represents the output vector obtained after mean pooling.

[0093] In some embodiments, the output tensor of the Transformer layer It is a sequence vector representation (i.e., semantic feature) processed by a multi-layer encoder of a large language model, incorporating positional information and contextual dependencies. It is a core feature for question-answering models to generate answers or make decisions for downstream tasks. For example, when... When =10, it represents the semantic feature aggregation result of the Transformer layer from 0 to 10.

[0094] In some embodiments, the aggregated semantic features include parts of speech, syntactic structure, and topic information (such as news).

[0095] The cosine similarity is calculated layer by layer in the Transformer layer to obtain the similarity matrix. for:

[0096] ;

[0097] in, Indicates the first The Transformer layer and the first The dot product operation of the output vectors of each Transformer layer after mean pooling, and ; and Indicates adjacent Transformer layers. The norm of a vector.

[0098] In some embodiments, The closer the value of is to 1, the more similar the output vectors are between the two layers; the closer the value is to -1, the less similar the output vectors are between the two layers. By analyzing this similarity matrix, we can identify which layers have similar functions, thus providing a basis for subsequent pruning operations.

[0099] S300: Based on the similarity matrix, a bottom-up dynamic hierarchical clustering grouping process is performed using a clustering algorithm to obtain the category clustering results of the Transformer layer.

[0100] In some embodiments, the similarity between adjacent Transformer layers is determined based on the similarity matrix. ;

[0101] Based on similarity Calculate the average similarity between all adjacent Transformer layers in any two categories to obtain the inter-class similarity. for:

[0102] ;

[0103] and This represents any pair of categories, where each Transformer layer has a corresponding category. Indicates including categories The number of Transformer layers, Indicates including categories The number of Transformer layers, Indicates conformity to category The Transformer layer, Indicates conformity to category The Transformer layer; where categories are used to classify Transformer layers that handle different question-answering tasks.

[0104] In some embodiments, the category refers to the different encoder layers or decoder layers in the Transformer model being divided into several groups, each group being called a category. For example, a 12-layer BERT model may be divided into 3 categories (such as 2-layer / 3-layer / 7-layer).

[0105] Different categories are used to handle different question-answering tasks, such as a certain category handling personalized question answering and sentiment analysis.

[0106] If the inter-class similarity between category pairs is greater than a preset adaptive threshold, then the category pairs are merged, and the new category is obtained. This includes all Transformer layers for the category pairs, while updating the category list; the dynamic hierarchical clustering grouping is repeated until no category pairs that meet the merging conditions can be found or the category list includes the preset number of categories, thus obtaining the category clustering results of the Transformer layers.

[0107] In some embodiments, a calculated N×N inter-layer cosine similarity matrix S is received, wherein... Indicates the first Layer and first The similarity between layers will be used to determine the similarity between each Transformer layer in the model. (in This is considered a separate category (or cluster). At this point, there are N initial categories, and each category contains one and only one Transformer layer.

[0108] In some embodiments, a preset adaptive threshold is used. This enhances the adaptability of clustering results to different tasks and model structures, and adjusts the clustering error caused by a fixed threshold under different tasks. The value of can be set to a certain quantile of the inter-class similarity in the current iteration (such as the median or 75th percentile), or adjusted according to the desired final pruning granularity. This dynamism allows for the generation of different numbers and granularities of layer categories under different tasks, better reflecting the inter-layer relationships of different models.

[0109] In some embodiments, the clustering results of the Transformer layer are processed sequentially by intra-category sorting and inter-category sorting to obtain the category sorting results, wherein the intra-category sorting is as follows:

[0110] ;

[0111] in, This represents the Transformer layers included in the clustered categories. According to its depth in the original model Sort in descending order, and ;

[0112] Perform inter-category sorting on the sorted content within each category to obtain the category sorting result:

[0113] ;

[0114] in, Represents the deepest clustering depth according to the already clustered categories. Sort in descending order from largest to smallest, and , This represents the total number of clusters obtained through clustering.

[0115] S400, based on the objective function and the clustering results of the categories, the original model is processed using an iterative progressive layer pruning strategy to obtain the layer pruning result of the original model. The objective function is used to perform iterative progressive layer pruning guided by minimizing the objective function under the premise of satisfying the dual threshold judgment, so as to characterize that the model performance of the original model is not lower than the preset value and the number of Transformer layers retained after pruning is minimized. The layer pruning is used to characterize the removal of the computation instructions during Transformer layer inference and to release the occupied GPU memory space.

[0116] In some embodiments, the preset value of model performance is used to ensure that the model performance meets the requirements of the actual scenario. Whether the model performance meets the preset value can be calculated and compared by means such as accuracy, mean squared error (MSE), and mean absolute error (MAE). It is understood that the preset value of model performance is set according to the required scenario.

[0117] In some embodiments, the iterative progressive layer pruning strategy performs the following processing steps:

[0118] Get category sorting results ,in Retrieve categories that contain only one Transformer layer from the category sorting results, and add these categories to the frozen layer set. The set of frozen layers It will not enter the candidate layer, meaning the deletion operation will not be performed;

[0119] Initialize the counter dictionary of the Transformer layer Use a counter dictionary for all Transformer layers The number of consecutive rounds in which pruning failed is recorded for all Transformer layers. (in This is used to record the number of consecutive rounds in which pruning was unsuccessful in each layer;

[0120] The copies of the original model are sorted by category, and candidate layers are evaluated and pruning decisions are made for each category's Transformer layer:

[0121] The deepest layer As a candidate layer for pruning, a deletion flag is added to the candidate layer, and the output of the previous retained layer is passed to the input of the next retained layer. The accuracy of the copy of the original model is compared with the initial accuracy to obtain the model accuracy decline rate. If the model accuracy decline rate does not meet either the accuracy difference threshold or the lower limit of absolute accuracy, the candidate layer is not deleted; otherwise, the candidate layer is deleted.

[0122] In some embodiments, for the currently selected category Select from those that have not yet been marked for deletion (i.e. And the deepest layer As candidate layers for this pruning, since step S300 performed intra-class sorting, the pruning proceeds sequentially from the first layer of each category. A temporary model copy is constructed by replicating the original model. , will candidate layer The temporary status at this location is set to delete. The output of the previous retained layer is directly passed to the input of the next retained layer to ensure model connectivity. The accuracy difference is then compared with the original model. If any of the following conditions are met, the candidate layer is rejected for deletion:

[0123] (1) The model accuracy drops beyond the accuracy difference threshold. ;

[0124] (2) The model accuracy is lower than the lower limit of absolute accuracy. and will .

[0125] If both threshold validations pass, then modify the pruning vector: Remove this layer from its category and add it to the frozen layer set. This category list is then moved forward to subsequent layers, returning to step S300 for inter-category sorting. This process of intra- and inter-category sorting, candidate layer evaluation, and pruning is repeated. In each loop, if a certain layer... Then add this layer to the frozen category set F.

[0126] The intra-category sorting is achieved by calculating the similarity of each layer's output using cosine similarity in the above embodiments and arranging them in descending order to enhance inter-layer consistency. The inter-category sorting is based on the similarity of the clustering results of the categories, arranging the categories in descending order of total similarity to optimize the inter-category structure.

[0127] Category Freeze and Loop Termination Conditions:

[0128] During the loop pruning process, in each iteration, freeze all categories with a total length of 1 in the category list and add the Transformer layers with a total length of 1 in the category list to the frozen category set. If no Transformer layer is deleted during the loop or all types of Transformer layers are frozen, the main iteration pruning loop terminates.

[0129] In some embodiments, the objective function is:

[0130] ;

[0131] in This indicates a decrease in model accuracy due to layer ablation. It is a binary variable, representing the first... Whether the layer is preserved, Indicates the first The computational cost of each layer; This is a hyperparameter used to balance the relative importance of accuracy loss and model computation.

[0132] This invention reduces the computational load and number of parameters of the model by freezing and pruning the Transformer layer and freezing multiple layers, while retaining the training capability of other layers, thus adapting to the needs of different question-answering tasks.

[0133] S500: Based on the layer pruning results of the original model, a modified original model is obtained. The modified original model is then used to reason about the target question statement to obtain the question reasoning result.

[0134] In some embodiments, reference Figure 2The diagram shows a comparison of inference time between the original model and the pruning model. The 0,1 pruning state vector is obtained through this embodiment of the invention. This is the final layer pruning strategy. Based on this vector, corresponding layer deletion operations can be performed on the model to obtain the layer-pruned model. To verify the effectiveness of the method, inference acceleration and scoring validation were performed on a cloud server using a new mini-batch question-answering dataset. Experiments were conducted on the original model and the pruned model. The Hunyuan-A13B-Instruct model and the DeepSeek-R1-Distill-Qwen-14B model were used as base models for model pruning, resulting in speedups of 24.1% and 22.6%, respectively. In the pruning experiment using the DeepSeek-R1-Distill-Qwen-14B model as the base model, an accuracy of 94.9% was achieved. The speedup was determined by comparing the inference time of the two models, and the accuracy was determined by the scores of the original model and the pruned model on the question-answering test set.

[0135] refer to Figure 3 The score curves of the original model and the pruned model shown are statistically analyzed from the test score curves of the original model and the pruned model on 19 mini-batch test sets. The total score of the original model is 177, and the score of the pruned model is 168. The pruned model can reach 94.9% of the original model, which proves that this method can effectively speed up the inference of the original model while still maintaining a good model accuracy.

[0136] refer to Figure 4 The diagram illustrates the overall process of layer pruning for a large language model with lightweight inference acceleration. The process, as shown in the pseudocode, includes layer output acquisition, similarity matrix establishment, and iterative progressive pruning (i.e., pruning state vector update).

[0137] Input: Original large language model M_original, test set R, thresholds θ, β, α.

[0138] Output: Pruned model M_pruned, pruned state vector V_prune.

[0139] FUNCTION main():

[0140] / / Step 1: Initialization

[0141] V_prune = [1,1,...,1] / / Length equals the total number of layers L in the model

[0142] F = / / Collection of frozen layers

[0143] Count_no_prune = [0,0,...,0] / / Unpruned counter

[0144] Acc_baseline = evaluate(M_original, R)

[0145] / / Step 2: Layer output capture and similarity calculation

[0146] S = compute_similarity_matrix(M_original, R)

[0147] / / Step 3: Dynamic Hierarchical Clustering

[0148] sorted_categories = hierarchical_clustering(S)

[0149] / / Step 4: Iterative Progressive Pruning

[0150] WHILE (Pruning not completed):

[0151] / / Traverse the sorted categories

[0152] FOR EACH category IN sorted_categories:

[0153] IF category.size == 1:

[0154] F.add(category.layers[0]) / / Freeze single-layer categories

[0155] CONTINUE

[0156] candidate_layer = get_deepest_layer(category) / / Get the deepest layer

[0157] IF candidate_layer IN F:

[0158] CONTINUE

[0159] / / Try deleting candidate layers

[0160] M_temp=create_temp_model(M_original,V_prune,candidate_layer)

[0161] Acc_temp = evaluate(M_temp, R)

[0162] Δ_Acc = Acc_baseline - Acc_temp

[0163] / / Dual threshold verification

[0164] IF (Δ_Acc ≤ θ) AND (Acc_temp ≥ β):

[0165] / / Accept pruning

[0166] V_prune[candidate_layer] = 0

[0167] F.add(candidate_layer)

[0168] Count_no_prune[candidate_layer] = 0

[0169] update_categories(sorted_categories, candidate_layer)

[0170] ELSE:

[0171] / / Refuse to prune

[0172] Count_no_prune[candidate_layer] += 1

[0173] IF Count_no_prune[candidate_layer] > α:

[0174] F.add(candidate_layer)

[0175] / / Check termination conditions

[0176] IF (No pruning in this round) OR (All layers are frozen):

[0177] BREAK

[0178] / / Recalculate the similarity matrix and clustering (optional)

[0179] If clustering needs to be updated:

[0180] S = recompute_similarity_matrix(M_current, R)

[0181] sorted_categories = hierarchical_clustering(S)

[0182] / / Step 5: Apply pruning and verify

[0183] M_pruned = apply_pruning(M_original, V_prune)

[0184] fine_tune(M_pruned) / / Fine-tune the model

[0185] evaluate_performance(M_pruned, R)

[0186] END FUNCTION

[0187] Figure 5 This is a schematic diagram of a large model layer pruning device for lightweight inference acceleration according to an embodiment of the present invention. The device includes a first module 510, a second module 520, a third module 530, a fourth module 540, and a fifth module 550.

[0188] The system comprises three modules: First, initializing the model parameters and pruning state of the original model, which is a large language model for question answering; second, obtaining the output tensors of the Transformer layers of the original model, and performing layer-by-layer cosine similarity calculations on the Transformer layers based on the output tensors to obtain the inter-layer similarity matrix for any pair of adjacent Transformer layers, where the inter-layer similarity matrix represents the semantic similarity of texts during question-answering inference using the large language model; and third, performing bottom-up dynamic hierarchical clustering grouping based on the similarity matrix using a clustering algorithm to obtain the Transformer layer similarity matrix. The first module is used to process the original model using an iterative progressive layer pruning strategy based on the objective function and the category clustering results, resulting in the layer pruning results of the original model. The objective function is used to minimize the objective function under the premise of satisfying the dual threshold judgment, so as to characterize that the model performance of the original model is not lower than the preset value and the number of Transformer layers retained after pruning is minimized. The second module is used to obtain the modified original model based on the layer pruning results of the original model, and to reason about the target question statement using the modified original model to obtain the question reasoning results.

[0189] Exemplarily, with the cooperation of the first, second, third, fourth, and fifth modules in the device, the embodiment device can implement any of the aforementioned lightweight inference acceleration large model layer pruning methods, namely, initializing the model parameters and pruning state of the original model, where the original model is a large language model for question answering, used to convert the input text signal into a semantic feature representation; obtaining the output tensor of the Transformer layer of the original model, and performing layer-by-layer cosine similarity calculation on the Transformer layer based on the output tensor to obtain the inter-layer similarity matrix of any adjacent Transformer layer pairs, where the inter-layer similarity matrix is ​​used to characterize the semantic similarity when the large language model infers the input text signal; the output tensor is used to characterize the semantic similarity obtained by the Transformer layer after processing the input text signal. Feature distribution feature tensor; based on the similarity matrix, a bottom-up dynamic hierarchical clustering grouping process is performed using a clustering algorithm to obtain the category clustering results of the Transformer layers; based on the objective function and the category clustering results, an iterative progressive layer pruning strategy is applied to the original model to obtain the layer pruning results of the original model. The objective function is used to perform iterative progressive layer pruning guided by minimizing the objective function under the premise of satisfying the double threshold judgment, so as to characterize that the model performance of the original model is not lower than the preset value and the number of Transformer layers retained after pruning is minimized; layer pruning is used to characterize the removal of computation instructions during Transformer layer inference and release the occupied GPU memory space; based on the layer pruning results of the original model, a modified original model is obtained, and the target question statement is inferred through the modified original model to obtain the question inference result. The beneficial effects of this invention are as follows: By capturing layer outputs and calculating similarity and dynamically clustering similar layers, when progressively pruning these similar layers, the large language model is ensured to reduce redundant computation while preserving performance; the large language model is effectively prevented from crashing through dual threshold verification (Transformer layer number threshold and model performance threshold) and a class-freezing strategy; and the connection compatibility is verified and the normalization layer is recalibrated after fine-tuning the large language model based on the accuracy difference threshold. When using the pruned large language model for question-answering inference, the number of parameters of the large language model is reduced, the efficiency and inference speed of the large language model are improved, and the system resource consumption during the inference of the large language model is reduced.

[0190] This invention also provides an electronic device, which includes a processor and a memory;

[0191] The memory stores the program;

[0192] The processor executes a program to perform the aforementioned lightweight inference acceleration large model layer pruning method; the electronic device has the function of carrying and running the software system for lightweight inference acceleration large model layer pruning provided in the embodiments of the present invention, such as a personal computer, minicomputer, mainframe, workstation, network or distributed computing environment, standalone or integrated computer platform, or communicating with charged particle tools or other imaging devices, etc.

[0193] This invention also provides a computer-readable storage medium storing a program that is executed by a processor to implement the large model layer pruning method for lightweight inference acceleration as described above.

[0194] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented in the embodiments of this invention. Alternative embodiments are contemplated, in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0195] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned lightweight inference acceleration method for large model layers.

[0196] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, considering the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed in the embodiments of the invention, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0197] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0198] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can include, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0199] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0200] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0201] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0202] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0203] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A lightweight inference acceleration large model layer pruning method, characterized in that, The method comprises the following steps: initializing model parameters and pruning states of an original model, the original model being a large language model for question answering, the large language model being used to convert an input text signal into a semantic feature representation; obtaining an output tensor of a Transformer layer of the original model, performing layer-by-layer cosine similarity calculation on the Transformer layer according to the output tensor to obtain an inter-layer similarity matrix of any adjacent Transformer layer pair, wherein the inter-layer similarity matrix is used to represent the semantic similarity of the large language model during inference on the input text signal; the output tensor is used to represent the semantic feature distribution tensor obtained after the Transformer layer processes the input text signal; performing a dynamic hierarchical clustering grouping processing from bottom to top on the Transformer layer according to the similarity matrix by using a clustering algorithm to obtain a category clustering result of the Transformer layer; processing the original model by using an iterative and progressive layer pruning strategy according to a target function and the category clustering result to obtain a layer pruning result of the original model, wherein the target function is used to guide the iterative and progressive layer pruning to minimize the target function under the premise of satisfying a double-threshold judgment, so as to represent that the model performance of the original model is not lower than a preset value and the number of Transformer layers retained after pruning is minimized; the layer pruning is used to represent that the calculation instructions during inference of the Transformer layer are removed and the occupied video memory space is released; wherein the double threshold includes an accuracy difference threshold and an absolute accuracy lower limit; obtaining a modified original model according to the layer pruning result of the original model, and performing inference on a target problem sentence by using the modified original model to obtain a problem inference result; The method further comprises: frozen the model weight of the original model, and performed inference on the original model through a question answering test set to obtain an initial accuracy; determining the initialized pruning state by taking the initial accuracy as a baseline value, wherein the initialized pruning state includes an accuracy difference threshold, an absolute accuracy lower limit and a layer freezing judgment threshold, wherein the accuracy difference threshold represents the maximum decrease amplitude of the model accuracy allowed by the removed Transformer layer due to pruning; the absolute accuracy lower limit represents the absolute minimum value of the model accuracy allowed by the removed Transformer layer due to pruning; and the layer freezing judgment threshold represents the number threshold of the Transformer layer not removed due to pruning, wherein the number of the Transformer layer not removed is determined by a counter dictionary set in each Transformer layer; The method further comprises: processing the category clustering result of the Transformer layer by using category-in-layer sorting and category-to-category sorting in sequence to obtain a category sorting result, wherein the category-in-layer sorting is: wherein, denotes the Transformer layers included in the clustered class according to their depth in the original model are arranged in descending order, and ; performing category-to-category sorting on the category-in-layer sorting to obtain the category sorting result: wherein, represents the deepest depth according to the clustered categories arranged in descending order from large to small, and , is the total number of categories obtained by clustering; Obtaining a category ranking result wherein only including a category with one Transformer layer in the category ranking result, adding the category with one Transformer layer to the frozen layer set wherein the frozen layer set does not enter the candidate layer, that is, does not perform the deletion operation; Counter dictionary to initialize the counters of the Transformer layers All Transformer layers are passed through the counter dictionary The number of consecutive unsuccessful prunings is recorded, where ; performing candidate layer evaluation and pruning decision on the Transformer layer of each category of the original model according to the category sorting result: the deepest layer add a deletion mark to the candidate layer, keep the output of the previous reserved layer of the candidate layer as the input of the next reserved layer; compare the accuracy of the copy of the original model with the initial accuracy to obtain the model accuracy drop rate, if the model accuracy drop exceeds the accuracy difference threshold or the model accuracy is lower than the absolute accuracy lower limit, the candidate layer is not deleted; otherwise, the candidate layer is deleted; The pruning process is performed in a loop, and at each loop, the class list is processed, all classes with a class list length of 1 are frozen, and the Transformer layers in the class list length of 1 are added to the frozen class set The main iterative pruning loop is terminated if no Transformer layer is removed at the loop or all Transformer layers of all classes are frozen.

2. The method of claim 1, wherein, The output tensor of the Transformer layer of the original model is obtained, and cosine similarity calculation is performed on the Transformer layer layer by layer according to the output tensor to obtain a similarity matrix between adjacent Transformer layer pairs, including: obtaining an output tensor of a Transformer layer of the original model , is a sequence number identifier of the Transformer layer, wherein the output tensor is obtained according to semantic feature aggregation of the sequence number identifier of the Transformer layer; performing mean pooling on the output tensor : wherein, denotes the total time steps of the mean pooling process, denotes the output vector at the time step, denotes the output vector after the mean pooling. Performing cosine similarity computation on the Transformer layers layer by layer to obtain a similarity matrix is: wherein, denotes the dot product operation of the output vectors of the th Transformer layer and the th Transformer layer after mean-pooling, and ; and denote adjacent Transformer layers, denotes the norm of a vector.

3. The method of claim 2, wherein, The similarity matrix is used to perform bottom-up dynamic hierarchical clustering grouping processing by using a clustering algorithm to obtain a category clustering result of the Transformer layer, including: According to the similarity matrix, the similarity of adjacent Transformer layers is determined as ; According to the similarity , the average value of the similarity between all adjacent Transformer layers in any two categories is calculated to obtain the inter-class similarity is: and denotes a pair of categories consisting of any two categories, each Transformer layer has a corresponding category, denotes the number of Transformer layers including category , denotes the number of Transformer layers including category , denotes the Transformer layer meeting category , denotes the Transformer layer meeting category ; wherein the category is used to classify the Transformer layers for processing different text signal tasks; If the inter-class similarity between the category pair is greater than the preset adaptive threshold, the category pair is merged, and a new category includes all Transformer layers in the category pair, and the category list is updated; The dynamic hierarchical clustering grouping is repeatedly performed until no category pair meeting the merging condition can be found or the category list includes a preset category number, and the category clustering result of the Transformer layer is obtained.

4. The method of claim 1, wherein, The method comprises: The objective function is: wherein represents the accuracy drop of the model due to layer ablation, is a binary variable indicating whether the layer is kept, represents the computational load of the layer; is a hyperparameter balancing the relative importance of accuracy loss and model computation. The Transformer layer of the original model is gradually deleted to optimize the objective function, and whether the deletion is performed according to the accuracy difference threshold is determined at each deletion.

5. A lightweight inference acceleration large model layer pruning apparatus, characterized in that, It includes: The first module is configured to initialize the model parameters and pruning state of the original model, wherein the original model is a large language model for question answering, and the large language model is configured to convert an input text signal into a semantic feature representation; The second module is configured to obtain an output tensor of a Transformer layer of the original model, and perform cosine similarity calculation on the Transformer layer layer by layer according to the output tensor to obtain an inter-layer similarity matrix of any adjacent Transformer layer pair, wherein the inter-layer similarity matrix is configured to represent the semantic similarity of the large language model when inferring the input text signal; and the output tensor is configured to represent a semantic feature distribution feature tensor obtained after the Transformer layer processes the input text signal; The third module is configured to perform bottom-up dynamic hierarchical clustering grouping processing by using a clustering algorithm according to the similarity matrix to obtain a category clustering result of the Transformer layer; The fourth module is configured to process the original model by using an iterative and progressive layer pruning strategy according to the objective function and the category clustering result to obtain a layer pruning result of the original model, wherein the objective function is configured to perform the iterative and progressive layer pruning in the premise of satisfying a double-threshold judgment and in the direction of minimizing the objective function, so as to represent that the model performance of the original model is not lower than a preset value and the number of Transformer layers retained after pruning is minimized; and the layer pruning is configured to represent removal of the calculation instruction of the Transformer layer during inference and release of the occupied video memory space; The double threshold includes an accuracy difference threshold and an absolute accuracy lower limit; The fifth module is configured to obtain a modified original model according to the layer pruning result of the original model, and perform inference on a target problem sentence by using the modified original model to obtain a problem inference result; The initialization of the model parameters and the pruning state of the original model comprises: The model weights of the original model are frozen, and the original model is inferred by using a question answering test set to obtain an initial accuracy; Determine an initialized pruning state with the initial accuracy as a baseline value, wherein the initialized pruning state includes an accuracy difference threshold, an absolute accuracy lower limit, and a layer freezing judgment threshold, wherein the accuracy difference threshold represents the maximum decrease in model accuracy allowed by the pruning of the Transformer layer; the absolute accuracy lower limit represents the absolute minimum value of the model accuracy allowed by the pruning of the Transformer layer; and the layer freezing judgment threshold represents a threshold number of Transformer layers that are not deleted by the pruning, wherein the number of Transformer layers that are not deleted is determined by a counter dictionary set in each Transformer layer; The device further includes: The category clustering result of the Transformer layer is sequentially processed by using intra-category layer sorting and inter-category sorting to obtain a category sorting result, wherein the intra-category layer sorting is: wherein, denotes the Transformer layers included in the clustered class according to their depth in the original model are arranged in descending order, and ; The sorting of the intra-category layer sorting is executed by using the inter-category sorting to obtain the category sorting result: wherein, represents the deepest depth according to the clustered categories arranged in descending order from large to small, and , is the total number of categories obtained by clustering; Obtaining a category ranking result wherein only including a category with one Transformer layer in the category ranking result, adding the category with one Transformer layer to the frozen layer set wherein the frozen layer set does not enter the candidate layer, i.e., does not perform the deletion operation; Counter dictionary to initialize the counters of the Transformer layers All Transformer layers are passed through the counter dictionary The number of consecutive unsuccessful prunings is recorded, where ; The copy of the original model is used to perform candidate layer evaluation and pruning decision on the Transformer layer of each category according to the category sorting result: the deepest layer add a deletion mark to the candidate layer, keep the output of the previous reserved layer of the candidate layer as the input of the next reserved layer; compare the accuracy of the copy of the original model with the initial accuracy to obtain the model accuracy drop rate, if the model accuracy drop exceeds the accuracy difference threshold or the model accuracy is lower than the absolute accuracy lower limit, the candidate layer is not deleted; otherwise, the candidate layer is deleted; The pruning process is performed in a loop, and at each loop, the class list is processed, all classes with a class list length of 1 are frozen, and the Transformer layers in the class list length of 1 are added to the frozen class set The main iterative pruning loop is terminated if no Transformer layer is removed at the loop or all Transformer layers of all classes are frozen.

6. An electronic device, comprising: The device includes a processor and a memory; The memory is used to store a program; The processor executes the program to implement the light-weight inference acceleration large model layer pruning method according to any one of claims 1-4.

7. A computer readable storage medium characterized in that, The storage medium stores a program, and the program is executed by the processor to implement the light-weight inference acceleration large model layer pruning method according to any one of claims 1-4.

Images