Lightweight reasoning accelerated large model layer pruning method, apparatus and device, and medium

Abstract

The invention belongs to the field of computer artificial intelligence, and provides a lightweight reasoning accelerated large model layer pruning method, device and equipment and a medium, and the method comprises the steps: initializing model parameters and pruning states of an original model; obtaining an output tensor of a Transform layer of the original model, and performing layer-by-layer cosine similarity calculation on the Transform layer according to the output tensor to obtain a similarity matrix between adjacent Transform layer pairs; according to the similarity matrix, carrying out dynamic hierarchical clustering grouping processing by adopting a clustering algorithm to obtain a category clustering result of a Transform layer; and processing the original model by adopting an iterative progressive layer pruning strategy according to the target function and the category clustering result to obtain a layer pruning result of the original model. The method has the beneficial effects that the reasoning speed of the large model is increased, and the system resource consumption during model reasoning is reduced.

Description

Technical Field

[0001] This invention relates to the field of computer artificial intelligence technology, and in particular to a method, apparatus, device and medium for lightweight inference acceleration of large model layer pruning. Background Technology

[0002] With the widespread application of large language models (such as GPT-3, GPT-4, etc.), inference speed and resource consumption have become bottlenecks in real-world applications. Large language models (LLMs) usually require a lot of computing resources during the inference phase, especially in edge devices, mobile terminals, or real-time inference scenarios, which often makes it difficult to meet the needs of efficient inference. This makes their deployment and application face high costs and latency issues.

[0003] Currently, existing model compression technologies can be broadly categorized into two main directions: model pruning and quantization. Model pruning is further subdivided into unstructured pruning and structured pruning. Unstructured pruning employs a strategy of precisely removing individual weights, which often achieves a relatively ideal performance improvement. However, due to the inherent limitations of current hardware architectures in handling sparse weights, achieving model acceleration is difficult. In stark contrast, structured pruning involves drastically cutting off the entire pre-defined model structure at once. This approach may cause a slight decrease in model accuracy to some extent, but from the perspective of hardware processing efficiency, it can achieve significant optimization. Among existing technologies, layer ablation simplifies the model structure by removing redundant layers, but its core problem lies in: Importance assessment is one-sided; a single indicator (such as gradient norm) cannot comprehensively measure the impact of a layer on the overall performance of the model. Similarity measurement is insufficient; the functional redundancy of adjacent layers is not fully considered, leading to the accidental deletion of key layers. Dynamic adjustment capability is weak; static pruning strategies are difficult to adapt to the differentiated needs of different input tasks. Summary of the Invention

[0004] Aimed at at least in solving one of the technical problems existing in the prior art, the present invention provides a lightweight reasoning acceleration method, apparatus, device and medium for pruning large model layers, which improves the reasoning speed of large language models and reduces the system resource consumption during large language model reasoning.

[0005] One aspect of the present invention provides a lightweight inference acceleration method for large model layer pruning, comprising: Initialize the model parameters and pruning state of the original model, which is a large language model for question answering, and the large language model is used to convert the input text signal into a semantic feature representation; Obtain the output tensor of the Transformer layer of the original model, and calculate the cosine similarity of the Transformer layer layer by layer based on the output tensor to obtain the inter-layer similarity matrix of any adjacent Transformer layer pair. The inter-layer similarity matrix is ​​used to characterize the semantic similarity of the large language model when it infers the input text signal. The output tensor is used to characterize the semantic feature distribution feature tensor obtained by the Transformer layer after processing the input text signal. Based on the similarity matrix, a bottom-up dynamic hierarchical clustering grouping process is performed using a clustering algorithm to obtain the category clustering results of the Transformer layer; Based on the objective function and the clustering results, the original model is processed using an iterative progressive layer pruning strategy to obtain the layer pruning result of the original model. The objective function is used to minimize the objective function in the iterative progressive layer pruning under the premise of satisfying the dual threshold judgment, so as to characterize that the model performance of the original model is not lower than the preset value and the number of Transformer layers retained after pruning is minimized. The layer pruning is used to characterize the removal of the computation instructions during Transformer layer inference and the release of occupied GPU memory space.

[0006] Based on the layer pruning results of the original model, a modified original model is obtained. The modified original model is then used to reason about the target question statement to obtain the question reasoning result.

[0007] According to the lightweight inference acceleration method for large model layers, the initialization of the original model's model parameters and pruning state includes: The model weights of the original model are frozen, and the original model is used for inference on the question-answering test set to obtain the initial accuracy. The initial pruning state is determined using the initial accuracy as a baseline value. The initial pruning state includes an accuracy difference threshold, an absolute accuracy lower limit, and a layer freeze judgment threshold. The accuracy difference threshold represents the maximum allowable decrease in model accuracy due to the removal of Transformer layers during pruning. The absolute accuracy lower limit is the absolute minimum value of model accuracy allowed due to the removal of Transformer layers during pruning. The layer freeze judgment threshold represents the threshold for the number of Transformer layers that were not removed during pruning. The number of times the Transformer layers were not removed is determined by a counter dictionary set for each Transformer layer.

[0008] According to the lightweight inference acceleration method for large model layer pruning, the output tensor of the Transformer layer of the original model is obtained, and the cosine similarity of the Transformer layers is calculated layer by layer based on the output tensor to obtain the similarity matrix between adjacent Transformer layer pairs, including: Obtain the output tensor of the Transformer layer of the original model , This is the index identifier for the Transformer layer, where the output tensor... The semantic features are aggregated based on the sequence number of the Transformer layer. For output tensor Perform mean pooling: ; in, This represents the total time steps for mean pooling. Indicates the first The output vector at each time step This represents the output vector obtained after mean pooling. The cosine similarity is calculated layer by layer in the Transformer layer to obtain the similarity matrix. for: ; in, Indicates the first The Transformer layer and the first The dot product operation of the output vectors of each Transformer layer after mean pooling, and ; and Indicates adjacent Transformer layers. The norm of a vector.

[0009] According to the lightweight inference acceleration method for large model layers, a bottom-up dynamic hierarchical clustering grouping process is performed using a clustering algorithm based on the similarity matrix to obtain the category clustering results of the Transformer layer, including: Based on the similarity matrix, the similarity between adjacent Transformer layers is determined as follows: ; Based on similarity Calculate the average similarity between all adjacent Transformer layers in any two categories to obtain the inter-class similarity. for: ; and This represents any pair of categories, where each Transformer layer has a corresponding category. Indicates including categories The number of Transformer layers, Indicates including categories The number of Transformer layers, Indicates conformity with category The Transformer layer, Indicates conformity with category The Transformer layer; where categories are used to classify Transformer layers that handle different question-answering tasks.

[0010] If the inter-class similarity between category pairs is greater than the preset adaptive threshold, the category pairs are merged, and the new category includes all Transformer layers of the category pair, while the category list is updated. Repeatedly perform dynamic hierarchical clustering grouping until no category pair that meets the merging conditions can be found or the category list includes the preset number of categories, to obtain the category clustering results of the Transformer layer.

[0011] According to the lightweight inference acceleration large model layer pruning method described above, the method further includes: The clustering results of the Transformer layer are processed sequentially by intra-category sorting and inter-category sorting to obtain the category sorting results, where the intra-category sorting is as follows: ; in, This represents the Transformer layers included in the clustered categories. According to its depth in the original model Sort in descending order, and ; Perform inter-category sorting on the sorted content within each category to obtain the category sorting result: ; in, Represents the deepest clustering depth according to the already clustered categories. Sort in descending order from largest to smallest, and , This represents the total number of clusters obtained through clustering.

[0012] According to the lightweight inference acceleration method for large model layers, a bottom-up dynamic hierarchical clustering grouping process is performed using a clustering algorithm based on the similarity matrix to obtain the category clustering results of the Transformer layer, including: Get category sorting results ,in Retrieve categories that contain only one Transformer layer from the category sorting results, and add these categories to the frozen layer set. The set of frozen layers It will not enter the candidate layer, meaning the deletion operation will not be performed; Initialize the counter dictionary of the Transformer layer Use a counter dictionary for all Transformer layers The number of consecutive rounds in which pruning was unsuccessful was recorded. ; The copies of the original model are sorted by category, and candidate layers are evaluated and pruning decisions are made for each category's Transformer layer: The deepest layer As a candidate layer for pruning, a deletion flag is added to the candidate layer, and the output of the previous retained layer is passed to the input of the next retained layer. The accuracy of the copy of the original model is compared with the initial accuracy to obtain the model accuracy decline rate. If the model accuracy decline rate does not meet either the accuracy difference threshold or the lower limit of absolute accuracy, the candidate layer is not deleted; otherwise, the candidate layer is deleted. The pruning process is performed iteratively. In each iteration, the category list is frozen, and all categories with a length of 1 are added to the frozen category set. If no Transformer layer is deleted during the loop or all types of Transformer layers are frozen, the main iteration pruning loop terminates.

[0013] According to the aforementioned lightweight inference acceleration method for large model layers, the method includes: The objective function is: ; in This indicates a decrease in model accuracy due to layer ablation. It is a binary variable, representing the first... Whether the layer is preserved, Indicates the first The computational cost of each layer; These are hyperparameters used to balance the relative importance of accuracy loss and model computation; The original model is progressively deleting Transformer layers to optimize the objective function, and at each deletion, the decision to perform the deletion is based on an accuracy difference threshold.

[0014] Another aspect of the present invention provides a lightweight inference acceleration large model layer pruning device, comprising: The first module is used to initialize the model parameters and pruning state of the original model, which is a large language model for question answering. The large language model is used to convert the input text signal into a semantic feature representation. The second module is used to obtain the output tensor of the Transformer layer of the original model, and to perform cosine similarity calculation on the Transformer layer layer by layer based on the output tensor to obtain the inter-layer similarity matrix of any adjacent Transformer layer pair. The third module is used to perform bottom-up dynamic hierarchical clustering grouping based on the similarity matrix using a clustering algorithm to obtain the category clustering results of the Transformer layer. The fourth module is used to process the original model using an iterative progressive layer pruning strategy based on the objective function and the clustering results of the categories, to obtain the layer pruning result of the original model. The objective function is used to minimize the objective function in the iterative progressive layer pruning under the premise of satisfying the dual threshold judgment, so as to characterize that the model performance of the original model is not lower than the preset value and the number of Transformer layers retained after pruning is minimized. The layer pruning is used to remove the computation instructions during Transformer layer inference and release the occupied GPU memory space. The fifth module is used to obtain a modified original model based on the layer pruning results of the original model, and then to reason about the target question statement using the modified original model to obtain the question reasoning result.

[0015] Another aspect of the present invention provides an electronic device, including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method as described above.

[0016] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the methods described above.

[0017] The beneficial effects of this invention are as follows: By capturing layer outputs and calculating similarity and dynamically clustering similar layers, when progressively pruning these similar layers, the large language model is ensured to reduce redundant computation while preserving performance; the large language model is effectively prevented from crashing through dual threshold verification (Transformer layer number threshold and model performance threshold) and a class-freezing strategy; and the connection compatibility is verified and the normalization layer is recalibrated after fine-tuning the large language model based on the accuracy difference threshold. When using the pruned large language model for question-answering inference, the number of parameters of the large language model is reduced, the efficiency and inference speed of the large language model are improved, and the system resource consumption during the inference of the large language model is reduced. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the large model layer pruning process for lightweight inference acceleration according to an embodiment of the present invention; Figure 2 This is a comparison chart of the inference time of the original model and the pruned model in an embodiment of the present invention; Figure 3 This is a graph showing the score curves of the original model and the pruned model in an embodiment of the present invention; Figure 4 This is a schematic diagram of the overall process of pruning a large language model layer for lightweight inference acceleration according to an embodiment of the present invention; Figure 5 This is a schematic diagram of a large model layer pruning device for lightweight inference acceleration according to an embodiment of the present invention. Detailed Implementation

[0019] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings. Throughout the description, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" can be used interchangeably. Terms such as "first," "second," etc., are used only to distinguish technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the sequential relationship of the indicated technical features. In the following description, the consecutive reference numerals for method steps are for ease of review and understanding. Adjusting the implementation order of steps, in conjunction with the overall technical solution of the present invention and the logical relationship between the various steps, will not affect the technical effect achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0020] refer to Figure 1 , Figure 1This is a schematic diagram of the lightweight inference acceleration large model layer pruning process according to an embodiment of the present invention, which includes, but is not limited to, steps S100~S400: S100: Initialize the model parameters and pruning state of the original model. The original model is a large language model for question answering, which is used to convert the input text signal into a semantic feature representation.

[0021] In some embodiments, semantic feature representation captures core semantic information of words, phrases, or sentences used to characterize input or question-and-answer sessions in a large language model, enabling the computer to perform mathematical processing and computation, such as context encoding and coding.

[0022] In some embodiments, large language models such as DeepSeek, GPT, etc. are used for question answering (natural language processing).

[0023] In some embodiments, the model weights of the original model are frozen, and the original model is used for inference on a question-answering test set to obtain an initial accuracy. The initial accuracy is used as a baseline value to determine the initial pruning state, which includes an accuracy difference threshold, an absolute accuracy lower bound, and a layer freeze judgment threshold. The accuracy difference threshold represents the maximum allowable decrease in model accuracy due to the removal of Transformer layers during pruning. The absolute accuracy lower bound is the absolute minimum value of model accuracy allowed due to the removal of Transformer layers during pruning. The layer freeze judgment threshold represents the threshold for the number of Transformer layers that were not removed during pruning, where the number of unremoved Transformer layers is determined by a counter dictionary set for each Transformer layer.

[0024] In some embodiments, the model weights of the large language model are feature representations learned by the model. By adjusting the weights, the model can understand the relationships and context between words. The Transformer layers corresponding to the model weight representations are not trained to reduce computation.

[0025] In some embodiments, the model weights of the large language model include processing the semantic features of the input text, with the weights capturing long-distance dependencies through a self-attention mechanism; and generating response text, with the weights determining the generation strategy and contextual associations.

[0026] In some embodiments, through a question-and-answer test set Construct a test set containing multiple questions. Load an original, unpruned model, keeping the model weights frozen to ensure the pruning process does not affect the original parameters. Define a string of length... A binary vector representing the total number of layers in the model. (Initialize the pruned state vector), initially all elements are 1, i.e. For all Established, At the start of pruning, all layers of the model are preserved: ; Using the original, unpruned model In the test set The above reasoning is used to record the initial accuracy. and original output In each subsequent pruning attempt, the performance of the current model will be used as a baseline for comparison.

[0027] Set accuracy difference threshold This is the maximum allowable decrease in model accuracy when attempting to remove a layer causes a drop in accuracy. If this happens, the deletion operation will be rejected.

[0028] Set a lower limit for absolute accuracy This is the absolute minimum allowable model accuracy, even if However, the trimmed model Model accuracy The deletion operation will also be rejected. These two thresholds together constitute a "dual threshold verification" mechanism to prevent model performance crashes caused by a single metric.

[0029] To avoid situations where certain categories remain undeleted across multiple iterations, leading to inefficient pruning, this embodiment of the invention initializes a counter dictionary for each layer. If a certain layer If a layer is not deleted in multiple consecutive iterations, and the number of undeleted layers exceeds the layer freezing threshold, then the layer freeze condition is considered to be in a state of failure. Then add this layer to the frozen category set. The pruning loop ends when no level is deleted or all categories are frozen during a single iteration.

[0030] In some embodiments, the accuracy difference threshold and the lower limit of absolute accuracy The settings vary depending on the type of text signal processing task being performed, such as the accuracy difference threshold for sentiment analysis question answering. and the lower limit of absolute accuracy It has a higher tolerance and can be set to a lower value, such as 0.3 and 0.7, while knowledge-based question answering has a lower tolerance and requires a higher value, such as 0.1 and 0.85.

[0031] S200, Obtain the output tensor of the Transformer layer of the original model, and perform cosine similarity calculation on the Transformer layer layer by layer based on the output tensor to obtain the inter-layer similarity matrix of any adjacent Transformer layer pair. The inter-layer similarity matrix is ​​used to characterize the semantic similarity of text when the large language model performs text signal inference.

[0032] In some embodiments, the inter-layer similarity of the Transformer layer mainly represents the semantic association strength between the query and the key, indicating that the large language model should focus on parts of the text input sequence when processing the words of the current question and answer, that is, dynamically capturing long-distance dependencies in the Transformer layer through the semantic association strength between the query and the key.

[0033] It should be noted that the large language model achieves vector transformation through weights in the Query matrix, Key matrix, and Value matrix: The Query matrix converts the input vector into a retrieval instruction (Query), determining which contextual information to focus on. A key matrix is ​​generated to produce index labels (Keys) that identify the semantic features of each token. Value matrix: Extracts the content value, providing a carrier of actual information.

[0034] The following is a textual example of the inter-layer similarity matrix of a Transformer layer: For example, in the question 1, "What is the highest mountain in the world?" and in the question 2, "What is the highest mountain in my country?", it can be determined that these two sentences are different in text, but by calculating their semantic vector similarity (e.g., a cosine similarity as high as 0.97), the large language model can determine that they are semantically equivalent questions, and thus can be answered with the same knowledge. This indicates that the two Transformer layers have a high degree of similarity.

[0035] The above embodiments only illustrate the similarity between questions. The similarity between questions and answers or between answers can also be calculated on the Transformer layer.

[0036] In some embodiments, forward hooks are registered at the output position of each Transformer block to obtain the output tensor of each layer without changing the original model structure and computation flow. Intermediate layer outputs are captured when inputting the test set, and the output tensors of the original model's Transformer layers are obtained. , This is the index identifier for the Transformer layer, where the output tensor... The semantic features are aggregated based on the sequence number of the Transformer layer. For output tensor Perform mean pooling: ; in, This represents the total time steps for mean pooling. Indicates the first The output vector at each time step This represents the output vector obtained after mean pooling. In some embodiments, the output tensor of the Transformer layer It is a sequence vector representation (i.e., semantic feature) processed by a multi-layer encoder of a large language model, incorporating positional information and contextual dependencies. It is a core feature for question-answering models to generate answers or make decisions for downstream tasks. For example, when... When =10, it represents the semantic feature aggregation result of the Transformer layer from 0 to 10.

[0037] In some embodiments, the aggregated semantic features include parts of speech, syntactic structure, and topic information (such as news).

[0038] The cosine similarity is calculated layer by layer in the Transformer layer to obtain the similarity matrix. for: ; in, Indicates the first The Transformer layer and the first The dot product operation of the output vectors of each Transformer layer after mean pooling, and ; and Indicates adjacent Transformer layers. The norm of a vector.

[0039] In some embodiments, The closer the value of is to 1, the more similar the output vectors are between the two layers; the closer the value is to -1, the less similar the output vectors are between the two layers. By analyzing this similarity matrix, we can identify which layers have similar functions, thus providing a basis for subsequent pruning operations.

[0040] S300: Based on the similarity matrix, a bottom-up dynamic hierarchical clustering grouping process is performed using a clustering algorithm to obtain the category clustering results of the Transformer layer.

[0041] In some embodiments, the similarity between adjacent Transformer layers is determined based on the similarity matrix. ; Based on similarity Calculate the average similarity between all adjacent Transformer layers in any two categories to obtain the inter-class similarity. for: ; and This represents any pair of categories, where each Transformer layer has a corresponding category. Indicates including categories The number of Transformer layers, Indicates including categories The number of Transformer layers, Indicates conformity with category The Transformer layer, Indicates conformity with category The Transformer layer; where categories are used to classify Transformer layers that handle different question-answering tasks.

[0042] In some embodiments, the category refers to the different encoder layers or decoder layers in the Transformer model being divided into several groups, each group being called a category. For example, a 12-layer BERT model may be divided into 3 categories (such as 2-layer / 3-layer / 7-layer).

[0043] Different categories are used to handle different question-answering tasks, such as a certain category handling personalized question answering and sentiment analysis.

[0044] If the inter-class similarity between category pairs is greater than a preset adaptive threshold, then the category pairs are merged, and the new category is obtained. This includes all Transformer layers for the category pairs, while updating the category list; the dynamic hierarchical clustering grouping is repeated until no category pairs that meet the merging conditions can be found or the category list includes the preset number of categories, thus obtaining the category clustering results of the Transformer layers.

[0045] In some embodiments, a calculated N×N inter-layer cosine similarity matrix S is received, wherein... Indicates the first Layer and first The similarity between layers will be used to determine the similarity between each Transformer layer in the model. (in This is considered a separate category (or cluster). At this point, there are N initial categories, and each category contains one and only one Transformer layer.

[0046] In some embodiments, a preset adaptive threshold is used. This enhances the adaptability of clustering results to different tasks and model structures, and adjusts the clustering error caused by a fixed threshold under different tasks. The value of can be set to a certain quantile of the inter-class similarity in the current iteration (such as the median or 75th percentile), or adjusted according to the desired final pruning granularity. This dynamism allows for the generation of different numbers and granularities of layer categories under different tasks, better reflecting the inter-layer relationships of different models.

[0047] In some embodiments, the clustering results of the Transformer layer are processed sequentially by intra-category sorting and inter-category sorting to obtain the category sorting results, wherein the intra-category sorting is as follows: ; in, This represents the Transformer layers included in the clustered categories. According to its depth in the original model Sort in descending order, and ; Perform inter-category sorting on the sorted content within each category to obtain the category sorting result: ; in, Represents the deepest clustering depth according to the already clustered categories. Sort in descending order from largest to smallest, and , This represents the total number of clusters obtained through clustering.

[0048] S400, based on the objective function and the clustering results of the categories, the original model is processed using an iterative progressive layer pruning strategy to obtain the layer pruning result of the original model. The objective function is used to perform iterative progressive layer pruning guided by minimizing the objective function under the premise of satisfying the dual threshold judgment, so as to characterize that the model performance of the original model is not lower than the preset value and the number of Transformer layers retained after pruning is minimized. The layer pruning is used to characterize the removal of the computation instructions during Transformer layer inference and to release the occupied GPU memory space.

[0049] In some embodiments, the preset value of model performance is used to ensure that the model performance meets the requirements of the actual scenario. Whether the model performance meets the preset value can be calculated and compared by means such as accuracy, mean squared error (MSE), and mean absolute error (MAE). It is understood that the preset value of model performance is set according to the required scenario.

[0050] In some embodiments, the iterative progressive layer pruning strategy performs the following processing steps: Get category sorting results ,in Retrieve categories that contain only one Transformer layer from the category sorting results, and add these categories to the frozen layer set. The set of frozen layers It will not enter the candidate layer, meaning the deletion operation will not be performed; Initialize the counter dictionary of the Transformer layer Use a counter dictionary for all Transformer layers The number of consecutive rounds in which pruning failed is recorded for all Transformer layers. (in This is used to record the number of consecutive rounds in which pruning was unsuccessful in each layer; The copies of the original model are sorted by category, and candidate layers are evaluated and pruning decisions are made for each category's Transformer layer: The deepest layer As a candidate layer for pruning, a deletion flag is added to the candidate layer, and the output of the previous retained layer is passed to the input of the next retained layer. The accuracy of the copy of the original model is compared with the initial accuracy to obtain the model accuracy decline rate. If the model accuracy decline rate does not meet either the accuracy difference threshold or the lower limit of absolute accuracy, the candidate layer is not deleted; otherwise, the candidate layer is deleted.

[0051] In some embodiments, for the currently selected category Select from those that have not yet been marked for deletion (i.e. And the deepest layer As candidate layers for this pruning, since step S300 performed intra-class sorting, the pruning proceeds sequentially from the first layer of each category. A temporary model copy is constructed by replicating the original model. , will candidate layer The temporary status at this location is set to delete. The output of the previous retained layer is directly passed to the input of the next retained layer to ensure model connectivity. The accuracy difference is then compared with the original model. If any of the following conditions are met, the candidate layer is rejected for deletion: (1) The model accuracy drops beyond the accuracy difference threshold. ; (2) The model accuracy is lower than the lower limit of absolute accuracy. and will .

[0052] If both threshold validations pass, then modify the pruning vector: Remove this layer from its category and add it to the frozen layer set. This category list is then moved forward to subsequent layers, returning to step S300 for inter-category sorting. This process of intra- and inter-category sorting, candidate layer evaluation, and pruning is repeated. In each loop, if a certain layer... Then add this layer to the frozen category set F.

[0053] The intra-category sorting is achieved by calculating the similarity of each layer's output using cosine similarity in the above embodiments and arranging them in descending order to enhance inter-layer consistency. The inter-category sorting is based on the similarity of the clustering results of the categories, arranging the categories in descending order of total similarity to optimize the inter-category structure.

[0054] Category Freeze and Loop Termination Conditions: During the loop pruning process, in each iteration, freeze all categories with a total length of 1 in the category list and add the Transformer layers with a total length of 1 in the category list to the frozen category set. If no Transformer layer is deleted during the loop or all types of Transformer layers are frozen, the main iteration pruning loop terminates.

[0055] In some embodiments, the objective function is: ; in This indicates a decrease in model accuracy due to layer ablation. It is a binary variable, representing the first... Whether the layer is preserved, Indicates the first The computational cost of each layer; This is a hyperparameter used to balance the relative importance of accuracy loss and model computation.

[0056] This invention reduces the computational load and number of parameters of the model by freezing and pruning the Transformer layer and freezing multiple layers, while retaining the training capability of other layers, thus adapting to the needs of different question-answering tasks.

[0057] S500: Based on the layer pruning results of the original model, a modified original model is obtained. The modified original model is then used to reason about the target question statement to obtain the question reasoning result.

[0058] In some embodiments, reference Figure 2 The diagram shows a comparison of inference time between the original model and the pruning model. The 0,1 pruning state vector is obtained through this embodiment of the invention. This is the final layer pruning strategy. Based on this vector, corresponding layer deletion operations can be performed on the model to obtain the layer-pruned model. To verify the effectiveness of the method, inference acceleration and scoring validation were performed on a cloud server using a new mini-batch question-answering dataset. Experiments were conducted on the original model and the pruned model. The Hunyuan-A13B-Instruct model and the DeepSeek-R1-Distill-Qwen-14B model were used as base models for model pruning, resulting in speedups of 24.1% and 22.6%, respectively. In the pruning experiment using the DeepSeek-R1-Distill-Qwen-14B model as the base model, an accuracy of 94.9% was achieved. The speedup was determined by comparing the inference time of the two models, and the accuracy was determined by the scores of the original model and the pruned model on the question-answering test set.

[0059] refer to Figure 3 The score curves of the original model and the pruned model shown are statistically analyzed from the test score curves of the original model and the pruned model on 19 mini-batch test sets. The total score of the original model is 177, and the score of the pruned model is 168. The pruned model can reach 94.9% of the original model, which proves that this method can effectively speed up the inference of the original model while still maintaining a good model accuracy.

[0060] refer to Figure 4 The diagram illustrates the overall process of layer pruning for a large language model with lightweight inference acceleration. The process, as shown in the pseudocode, includes layer output acquisition, similarity matrix establishment, and iterative progressive pruning (i.e., pruning state vector update). Input: Original large language model M_original, test set R, thresholds θ, β, α.

[0061] Output: Pruned model M_pruned, pruned state vector V_prune.

[0062] FUNCTION main(): / / Step 1: Initialization V_prune = [1,1,...,1] / / Length equals the total number of layers L in the model F = / / Collection of frozen layers Count_no_prune = [0,0,...,0] / / Unpruned counter Acc_baseline = evaluate(M_original, R) / / Step 2: Layer output capture and similarity calculation S = compute_similarity_matrix(M_original, R) / / Step 3: Dynamic Hierarchical Clustering sorted_categories = hierarchical_clustering(S) / / Step 4: Iterative Progressive Pruning WHILE (Pruning not completed): / / Traverse the sorted categories FOR EACH category IN sorted_categories: IF category.size == 1: F.add(category.layers[0]) / / Freeze single-layer categories CONTINUE candidate_layer = get_deepest_layer(category) / / Get the deepest layer IF candidate_layer IN F: CONTINUE / / Try deleting candidate layers M_temp=create_temp_model(M_original,V_prune,candidate_layer) Acc_temp = evaluate(M_temp, R) Δ_Acc = Acc_baseline - Acc_temp / / Dual threshold verification IF (Δ_Acc ≤ θ) AND (Acc_temp ≥ β): / / Accept pruning V_prune[candidate_layer] = 0 F.add(candidate_layer) Count_no_prune[candidate_layer] = 0 update_categories(sorted_categories, candidate_layer) ELSE: / / Refuse to prune Count_no_prune[candidate_layer] += 1 IF Count_no_prune[candidate_layer] > α: F.add(candidate_layer) / / Check termination conditions IF (No pruning in this round) OR (All layers are frozen): BREAK / / Recalculate the similarity matrix and clustering (optional) If clustering needs to be updated: S = recompute_similarity_matrix(M_current, R) sorted_categories = hierarchical_clustering(S) / / Step 5: Apply pruning and verify M_pruned = apply_pruning(M_original, V_prune) fine_tune(M_pruned) / / Fine-tune the model evaluate_performance(M_pruned, R) END FUNCTION Figure 5 This is a schematic diagram of a lightweight inference acceleration large model layer pruning device according to an embodiment of the present invention. The device includes a first module 510, a second module 520, a third module 530, a fourth module 540, and a fifth module 550.

[0063] The system comprises three modules: First, initializing the model parameters and pruning state of the original model, which is a large language model for question answering; second, obtaining the output tensors of the Transformer layers of the original model, and performing layer-by-layer cosine similarity calculations on the Transformer layers based on the output tensors to obtain the inter-layer similarity matrix for any pair of adjacent Transformer layers, where the inter-layer similarity matrix represents the semantic similarity of texts during question-answering inference using the large language model; and third, performing bottom-up dynamic hierarchical clustering grouping based on the similarity matrix using a clustering algorithm to obtain the Transformer layer similarity matrix. The first module is used to process the original model using an iterative progressive layer pruning strategy based on the objective function and the category clustering results, resulting in the layer pruning results of the original model. The objective function is used to minimize the objective function under the premise of satisfying the dual threshold judgment, so as to characterize that the model performance of the original model is not lower than the preset value and the number of Transformer layers retained after pruning is minimized. The second module is used to obtain the modified original model based on the layer pruning results of the original model, and to reason about the target question statement using the modified original model to obtain the question reasoning results.

[0064] Exemplarily, with the cooperation of the first, second, third, fourth, and fifth modules in the device, the embodiment device can implement any of the aforementioned lightweight inference acceleration large model layer pruning methods, namely, initializing the model parameters and pruning state of the original model, where the original model is a large language model for question answering, used to convert the input text signal into a semantic feature representation; obtaining the output tensor of the Transformer layer of the original model, and performing layer-by-layer cosine similarity calculation on the Transformer layer based on the output tensor to obtain the inter-layer similarity matrix of any adjacent Transformer layer pairs, where the inter-layer similarity matrix is ​​used to characterize the semantic similarity when the large language model infers the input text signal; the output tensor is used to characterize the semantic similarity obtained by the Transformer layer after processing the input text signal. Feature distribution feature tensor; based on the similarity matrix, a bottom-up dynamic hierarchical clustering grouping process is performed using a clustering algorithm to obtain the category clustering results of the Transformer layers; based on the objective function and the category clustering results, an iterative progressive layer pruning strategy is applied to the original model to obtain the layer pruning results of the original model. The objective function is used to perform iterative progressive layer pruning guided by minimizing the objective function under the premise of satisfying the double threshold judgment, so as to characterize that the model performance of the original model is not lower than the preset value and the number of Transformer layers retained after pruning is minimized; layer pruning is used to characterize the removal of computation instructions during Transformer layer inference and release the occupied GPU memory space; based on the layer pruning results of the original model, a modified original model is obtained, and the target question statement is inferred through the modified original model to obtain the question inference result. The beneficial effects of this invention are as follows: By capturing layer outputs and calculating similarity and dynamically clustering similar layers, when progressively pruning these similar layers, the large language model is ensured to reduce redundant computation while preserving performance; the large language model is effectively prevented from crashing through dual threshold verification (Transformer layer number threshold and model performance threshold) and a class-freezing strategy; and the connection compatibility is verified and the normalization layer is recalibrated after fine-tuning the large language model based on the accuracy difference threshold. When using the pruned large language model for question-answering inference, the number of parameters of the large language model is reduced, the efficiency and inference speed of the large language model are improved, and the system resource consumption during the inference of the large language model is reduced.

[0065] This invention also provides an electronic device, which includes a processor and a memory; The memory stores the program; The processor executes a program to perform the aforementioned lightweight inference acceleration large model layer pruning method; the electronic device has the function of carrying and running the software system for lightweight inference acceleration large model layer pruning provided in the embodiments of the present invention, such as a personal computer, minicomputer, mainframe, workstation, network or distributed computing environment, standalone or integrated computer platform, or communicating with charged particle tools or other imaging devices, etc.

[0066] This invention also provides a computer-readable storage medium storing a program that is executed by a processor to implement the large model layer pruning method for lightweight inference acceleration as described above.

[0067] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented in the embodiments of this invention. Alternative embodiments are contemplated, in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0068] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned lightweight inference acceleration method for large model layers.

[0069] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, considering the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed in the embodiments of the invention, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0070] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can include, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0072] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0073] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0074] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0075] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0076] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A lightweight inference acceleration method for large model layer pruning, characterized in that, include: Initialize the model parameters and pruning state of the original model, which is a large language model for question answering, and the large language model is used to convert the input text signal into a semantic feature representation; Obtain the output tensor of the Transformer layer of the original model, and calculate the cosine similarity of the Transformer layer layer by layer based on the output tensor to obtain the inter-layer similarity matrix of any adjacent Transformer layer pair. The inter-layer similarity matrix is ​​used to characterize the semantic similarity of the large language model when it infers the input text signal. The output tensor is used to characterize the semantic feature distribution feature tensor obtained by the Transformer layer after processing the input text signal. Based on the similarity matrix, a bottom-up dynamic hierarchical clustering grouping process is performed using a clustering algorithm to obtain the category clustering results of the Transformer layer; Based on the objective function and the clustering results, the original model is processed using an iterative progressive layer pruning strategy to obtain the layer pruning result of the original model. The objective function is used to minimize the objective function in the iterative progressive layer pruning under the premise of satisfying the dual threshold judgment, so as to characterize that the model performance of the original model is not lower than the preset value and the number of Transformer layers retained after pruning is minimized. The layer pruning is used to characterize the removal of the computation instructions during Transformer layer inference and the release of occupied GPU memory space. Based on the layer pruning results of the original model, a modified original model is obtained. The modified original model is then used to reason about the target question statement to obtain the question reasoning result.

2. The lightweight inference acceleration method for large model layers according to claim 1, characterized in that, The initialization of the original model's model parameters and pruning state includes: The model weights of the original model are frozen, and the original model is used for inference on the question-answering test set to obtain the initial accuracy. The initial pruning state is determined using the initial accuracy as a baseline value. The initial pruning state includes an accuracy difference threshold, an absolute accuracy lower limit, and a layer freeze judgment threshold. The accuracy difference threshold represents the maximum allowable decrease in model accuracy due to the removal of Transformer layers during pruning. The absolute accuracy lower limit is the absolute minimum value of model accuracy allowed due to the removal of Transformer layers during pruning. The layer freeze judgment threshold represents the threshold for the number of Transformer layers that were not removed during pruning. The number of times the Transformer layers were not removed is determined by a counter dictionary set for each Transformer layer.

3. The method for lightweight inference acceleration of large model layers according to claim 1, characterized in that, The process of obtaining the output tensor of the Transformer layer of the original model, and calculating the cosine similarity of each Transformer layer based on the output tensor to obtain the similarity matrix between adjacent Transformer layer pairs includes: Obtain the output tensor of the Transformer layer of the original model , This is the index identifier for the Transformer layer, where the output tensor... The semantic features are aggregated based on the sequence number of the Transformer layer. For output tensor Perform mean pooling: ； in, This represents the total time steps for mean pooling. Indicates the first The output vector at each time step This represents the output vector obtained after mean pooling. The cosine similarity is calculated layer by layer in the Transformer layer to obtain the similarity matrix. for: ； in, Indicates the first The Transformer layer and the first The dot product operation of the output vectors of each Transformer layer after mean pooling, and ; and Indicates adjacent Transformer layers. The norm of a vector.

4. The method for lightweight inference acceleration of large model layers according to claim 3, characterized in that, The process involves performing bottom-up dynamic hierarchical clustering grouping based on the similarity matrix using a clustering algorithm to obtain the category clustering results of the Transformer layer, including: Based on the similarity matrix, the similarity between adjacent Transformer layers is determined as follows: ; Based on similarity Calculate the average similarity between all adjacent Transformer layers in any two categories to obtain the inter-class similarity. for: ； and This represents any pair of categories, where each Transformer layer has a corresponding category. Indicates including categories The number of Transformer layers, Indicates including categories The number of Transformer layers, Indicates conformity with category The Transformer layer, Indicates conformity with category The Transformer layer; where categories are used to classify the Transformer layers that process different text signals. If the inter-class similarity between category pairs is greater than the preset adaptive threshold, the category pairs are merged, and the new category includes all Transformer layers of the category pair, while the category list is updated. Repeatedly perform dynamic hierarchical clustering grouping until no category pair that meets the merging conditions can be found or the category list includes the preset number of categories, to obtain the category clustering results of the Transformer layer.

5. The method for lightweight inference acceleration of large model layers according to claim 4, characterized in that, The method further includes: The clustering results of the Transformer layer are processed sequentially by intra-category sorting and inter-category sorting to obtain the category sorting results, where the intra-category sorting is as follows: ； in, This represents the Transformer layers included in the clustered categories. According to its depth in the original model Sort in descending order, and ; Perform inter-category sorting on the sorted content within each category to obtain the category sorting result: ； in, Represents the deepest clustering depth according to the already clustered categories. Sort in descending order from largest to smallest, and , This represents the total number of clusters obtained through clustering.

6. The method for lightweight inference acceleration of large model layers according to claim 5, characterized in that, The process involves performing bottom-up dynamic hierarchical clustering grouping based on the similarity matrix using a clustering algorithm to obtain the category clustering results of the Transformer layer, including: Get category sorting results ,in Retrieve categories that contain only one Transformer layer from the category sorting results, and add these categories to the frozen layer set. The set of frozen layers It will not enter the candidate layer, meaning the deletion operation will not be performed; Initialize the counter dictionary of the Transformer layer Use a counter dictionary for all Transformer layers The number of consecutive rounds in which pruning was unsuccessful was recorded. ; The copies of the original model are sorted by category, and candidate layers are evaluated and pruning decisions are made for each category's Transformer layer: The deepest layer As a candidate layer for pruning, a deletion flag is added to the candidate layer, and the output of the previous retained layer is used as the input of the next retained layer. The accuracy of the copy of the original model is compared with the initial accuracy to obtain the model accuracy decline rate. If the model accuracy decline rate does not meet either the accuracy difference threshold or the lower limit of absolute accuracy, the candidate layer is not deleted; otherwise, the candidate layer is deleted. The pruning process is performed iteratively. In each iteration, the category list is frozen, and all categories with a length of 1 are added to the frozen category set. If no Transformer layer is deleted during the loop or all types of Transformer layers are frozen, the main iteration pruning loop terminates.

7. The method for lightweight inference acceleration of large model layers according to claim 6, characterized in that, The method includes: The objective function is: ； in This indicates a decrease in model accuracy due to layer ablation. It is a binary variable, representing the first... Whether the layer is preserved, Indicates the first The computational cost of each layer; These are hyperparameters used to balance the relative importance of accuracy loss and model computation; The original model is progressively deleting Transformer layers to optimize the objective function, and at each deletion, the decision to perform the deletion is based on an accuracy difference threshold.

8. A lightweight inference acceleration device for large model layer pruning, characterized in that, include: The first module is used to initialize the model parameters and pruning state of the original model, which is a large language model for question answering. The large language model is used to convert the input text signal into a semantic feature representation. The second module is used to obtain the output tensor of the Transformer layer of the original model, and to perform cosine similarity calculation on the Transformer layer layer by layer based on the output tensor to obtain the inter-layer similarity matrix of any adjacent Transformer layer pairs. The inter-layer similarity matrix is ​​used to characterize the semantic similarity of the large language model when it infers the input text signal. The output tensor is used to characterize the semantic feature distribution feature tensor obtained by the Transformer layer after processing the input text signal. The third module is used to perform bottom-up dynamic hierarchical clustering grouping based on the similarity matrix using a clustering algorithm to obtain the category clustering results of the Transformer layer. The fourth module is used to process the original model using an iterative progressive layer pruning strategy based on the objective function and the clustering results of the categories, to obtain the layer pruning result of the original model. The objective function is used to minimize the objective function in the iterative progressive layer pruning under the premise of satisfying the dual threshold judgment, so as to characterize that the model performance of the original model is not lower than the preset value and the number of Transformer layers retained after pruning is minimized. The layer pruning is used to remove the computation instructions during Transformer layer inference and release the occupied GPU memory space. The fifth module is used to obtain a modified original model based on the layer pruning results of the original model, and then to reason about the target question statement using the modified original model to obtain the question reasoning result.

9. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the lightweight inference acceleration large model layer pruning method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the lightweight inference acceleration method for large model layers as described in any one of claims 1-7.

Images