Structured layer pruning method and system for large language models

Abstract

The application discloses a large language model structured layer pruning method and system executed by an electronic device. The electronic device reads the same type weight matrix of each Transformer layer of a pre-training model, calculates an inter-layer similarity matrix, and determines a continuous layer cluster group according to a main diagonal line sliding window score; calculates a redundancy score according to the average similarity of a layer and other layers in the cluster and the normalized inter-layer distance of the layer to the cluster group boundary, protects the boundary layer and the central representative layer of each cluster group, and sorts the remaining layers according to the score; when the target number of deleted layers does not exceed the number of deletable layers, the high-score layers are replaced in turn by an identity mapping, and the model layer sequence and connection relationship are updated; then, the training is resumed on a GPU calculation unit, and the pruned model is saved. The method reduces the model blocks, corresponding parameters and inference calculation according to a predetermined number of layers.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a structured layer pruning method and system for a large language model executed by an electronic device. Background Technology

[0002] Large language models typically consist of dozens to hundreds of stacked Transformer layers, resulting in a large number of parameters and computational demands for inference. During deployment, the complete model places high demands on storage space, GPU memory capacity, and computational throughput. Model compression techniques reduce deployment costs by decreasing redundant parameters or structures, with layer pruning achieving structured compression by removing some Transformer layers.

[0003] Existing structured layer pruning methods can evaluate Transformer layers based on layer output, gradient, loss perturbation, routing records, or inter-layer similarity, and delete some layers according to the evaluation results. Different methods employ different restrictions on the number of boundary layers, representative layers, and target layers to be deleted.

[0004] In implementations that uniformly sort all layers, the sorting result may lead to the deletion of multiple layers within the same contiguous region. Without explicit protection rules for the boundary and representative layers of contiguous regions, the final model structure depends on the sorting result, making it difficult to verify which layers should be retained for each contiguous region before pruning.

[0005] Therefore, a structured pruning method is needed that can obtain continuous layer clusters from model weights, form a defined set of protective layers and a set of deletable layers, and update the model layer order and parameter tensors according to the number of layers to be deleted according to feasible objectives. Summary of the Invention

[0006] The technical problem this invention aims to solve is: when performing structured layer pruning on a large language model in an electronic device, how to convert inter-layer weight similarity into deterministic layer protection rules and deletion order, and how to ensure that the target number of deleted layers, the model structure after deletion, and the reduced parameter tensors can be directly verified before and after pruning. This invention determines continuous layer clusters based on the inter-layer similarity matrix, calculates redundancy scores using the average intra-cluster similarity and the normalized distance from the layer to the cluster boundary, and excludes boundary layers and central representative layers from the set of layers that can be deleted.

[0007] To achieve the above objectives, the technical solution adopted in this application is: a method for pruning the structured layer of a large language model executed on an electronic device including a processor, a memory, and at least one GPU computing unit, comprising the following steps.

[0008] Read the weight matrices of the same type from each Transformer layer of the pre-trained large language model from the memory, perform pairwise comparisons on each layer using the same matrix similarity calculation rules, and arrange the obtained similarity values ​​according to the layer number to construct an inter-layer similarity matrix;

[0009] The square window of a preset order is moved layer by layer along the main diagonal of the inter-layer similarity matrix. The average value of the elements in each window except for the elements on the main diagonal is calculated. The boundary of the layer cluster is determined according to the change of the average value of the adjacent windows, and all Transformer layers are divided into several continuous layer clusters accordingly.

[0010] For each layer in each layer cluster, calculate the average similarity between that layer and other layers in the same layer cluster, and calculate the normalized boundary distance based on the layer number distance from that layer to the two boundaries of the layer cluster. The average similarity and the normalized boundary distance are weighted and summed according to two preset non-negative weights to obtain the redundancy score of that layer, where the sum of the two weights is one. The layer with the largest average similarity is selected as the central representative layer of that layer cluster.

[0011] The starting, ending, and central representative layers of each layer cluster, as well as the first and last layers of the entire model, are designated as protective layers, and the remaining layers are designated as removable layers. The product of the preset deletion ratio and the total number of layers in the model is rounded down to obtain the target number of layers to be deleted. When the target number of layers to be deleted is not greater than the total number of removable layers, all removable layers are arranged in descending order of redundancy score. If the scores are the same, they are arranged in ascending order of layer number, and the corresponding number of layers at the top of the list are selected as the target number of layers to be deleted. When the target number of layers to be deleted is greater than the total number of removable layers, the state that the target number of layers to be deleted is not feasible is output, and pruning is stopped.

[0012] The target deletion layer is removed from the layer sequence of the pre-trained large language model. The input of each deleted layer is passed to the next retained layer using an identity mapping. The layer number, retained layer order, and parameter key mapping in the model configuration are updated. The updated model is then retrained on the GPU computing unit, and the retrained pruned model is written to the memory. The deletion operation not only generates a list of layer numbers but also removes the parameter tensor corresponding to the target layer from the model parameter state, modifies the total number of layers and layer number mapping in the model configuration, and makes the pruning result a model file that can be loaded by the inference program.

[0013] Furthermore, the inter-layer weight similarity acquisition preferably adopts a measurement method based on cosine similarity distribution: for any two Transformer layers with the same type of weight matrix, calculate the cosine similarity between column vectors, fit the set of maximum absolute values ​​obtained in the two directions with Gumbel distribution, and then take the arithmetic mean of the position parameters in the two directions as the similarity value between the two Transformer layers.

[0014] Furthermore, the boundary detection threshold can be determined using an adaptive method: calculate the mean and standard deviation of the average similarity difference between all adjacent windows, and use the sum of the two as the boundary detection threshold.

[0015] Furthermore, in S3, a higher redundancy score indicates that the layer is closer to other layers in the same cluster and relatively far from the boundary protection layer, and therefore can be given priority as a candidate pruning layer; the representative layer is used to ensure that each layer cluster retains at least one functional carrying layer.

[0016] When deleting multiple layers consecutively, the output of the previous retained layer is still directly used as the input of the next retained layer, without inserting an additional projection layer that changes the pruning parameters.

[0017] Resuming training uses preset calibration data and a fixed validation set. The initial pruned model is saved before starting resuming training; during training, the validation loss is calculated at fixed intervals, and the checkpoint with the lowest validation loss is saved. The number of resuming training steps, batch size, and learning rate are written as configuration parameters to the training record.

[0018] Secondly, the present invention provides a structured layer pruning system for a large language model, comprising a processor, at least one GPU computing unit, a memory connected to the processor and the GPU computing unit, and a layer similarity acquisition module, a layer cluster identification module, a redundancy evaluation module, a structured pruning module, and a recovery fine-tuning module stored in the memory and executed by the processor.

[0019] Thirdly, the present invention provides a computer-readable storage medium storing computer program instructions that, when executed by a processor, cause the processor to perform the steps of the method described in the present invention.

[0020] Fourthly, the present invention provides an electronic device including a memory, a processor, and at least one GPU computing unit, wherein the memory stores computer program instructions executable by the processor, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to perform the steps of the method described in the present invention.

[0021] Compared to schemes that only rank the importance of output layers, this invention generates a set of guard layers, a set of deletable layers, and a defined number of target deletion layers. The target number of deletion layers is compared with the number of deletable layers before execution, thus preventing the deletion of guard layers simply to meet a deletion ratio. After deletion, the number of Transformer layers in the model equals the total number of layers in the original model minus the actual number of deleted layers; the parameter tensors and forward computations corresponding to the deleted layers are removed from the output model, and the reduction can be directly obtained by summing the parameter count and computational cost of each target deletion layer. This paper does not claim a specific task accuracy preservation range without experimental data. Attached Figure Description

[0022] Figure 1 This is an overall flowchart for layer cluster identification, redundancy scoring, protection layer locking, feasibility verification of target layer deletion, global selection, and structure updating.

[0023] Figure 2 This is a schematic diagram of inter-layer similarity matrix and layer cluster identification.

[0024] Figure 3 This is a schematic diagram of the average similarity within the cluster, the normalized boundary distance, the central representative layer, and the set of protective layers.

[0025] Figure 4 This diagram illustrates a pruning strategy that first locks the set of protected layers, then checks whether the target number of layers to be deleted does not exceed the number of layers that can be deleted, and then globally selects the corresponding number of layers from the layers that can be deleted.

[0026] Figure 5 This is a system architecture diagram for modules such as inter-layer similarity acquisition, layer cluster identification, redundancy assessment, pruning selection, structure update, and training recovery.

[0027] Figure 6 This is a structural diagram of an electronic device. Detailed Implementation

[0028] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0029] Several specific details are set forth in the following description to aid in understanding the invention. However, the invention may also be practiced in other ways than those described herein, and therefore the invention is not limited to the specific embodiments disclosed in the following specification. Example 1: Complete Method Flow

[0030] Example 1 details the complete process of the method of the present invention. For example... Figure 1As shown, this invention proposes a structured layer pruning method for large language models based on inter-layer weight similarity, including steps S1 to S5.

[0031] Step S1: Obtain inter-layer weight similarity. For a pre-trained large language model, read the same type of weight matrix from memory for each Transformer layer, and select the optimal value matrix or up-projection matrix; after calculating the similarity value for all unordered layer pairs, construct an inter-layer similarity matrix with the total number of layers in the model as the number of rows and columns according to the layer number.

[0032] In a preferred embodiment, the inter-layer similarity calculation employs a measurement method based on cosine similarity distribution. For any two Transformer layers with the same type of weight matrices, the cosine similarity matrix between column vectors is calculated; using one of the weight matrices as a reference, the maximum absolute value of the cosine similarity for each column vector is obtained and a Gumbel distribution is fitted; after swapping the two weight matrices, the calculation is repeated, and then the arithmetic mean of the position parameters in the two directions is taken to obtain the similarity between the two Transformer layers.

[0033] Step S2: Layer cluster identification. For example... Figure 2 As shown, the window order is set to an integer not less than two and not greater than the total number of layers in the model; the square window is moved layer by layer along the main diagonal of the inter-layer similarity matrix, and the average value of the non-main diagonal elements in each window and the difference between the latter and former terms of the average value of adjacent windows are calculated in turn. The layer cluster boundary is determined according to the boundary detection threshold and the continuous candidate boundary merging rule; when the window order is equal to the total number of layers in the model or there is no candidate boundary, all Transformer layers are determined as a layer cluster.

[0034] The boundary detection threshold can be a fixed threshold or an adaptive threshold. When using an adaptive threshold, the mean and standard deviation of the average similarity difference between all adjacent windows are calculated, and the sum of the two is used as the boundary detection threshold.

[0035] For any target layer in a layer cluster, first calculate the average similarity between that layer and other layers within the same layer cluster. When the layer cluster contains only one layer, the average similarity is set to zero.

[0036] Calculate the layer number distances from the target layer to both sides of the layer cluster boundary, take the smaller of the two distances, and normalize them using the maximum and minimum boundary distances achievable within the layer cluster to obtain the normalized boundary distance. The normalized boundary distance for the boundary layer is zero, and the closer to the center of the layer cluster, the larger the normalized boundary distance.

[0037] The average similarity and normalized boundary distance are summed using two preset non-negative weights to obtain a redundancy score, where the sum of the two weights is one. A higher score indicates that the layer has both high intra-cluster similarity and is located within the cluster.

[0038] For each layer cluster, the layer with the highest average similarity is selected as the central representative layer. When layers are side by side, they are determined sequentially according to the rule of being closer to the center of the layer cluster and having a smaller layer number.

[0039] The starting layer, ending layer, and central representative layer of a layer cluster are all protective layers. The first and last layers of the model are added to the protective layer set regardless of whether they have been used as the boundary of the layer cluster.

[0040] After excluding the protective layers from all layers, a set of layers that can be deleted is obtained. The product of the preset deletion ratio and the total number of layers in the model is rounded down to obtain the target number of layers to be deleted. If the target number of layers to be deleted exceeds the number of layers that can be deleted, the current objective is not feasible, the electronic device outputs an infeasibility status and stops, instead of removing the protective constraints.

[0041] If the objective is feasible, the layers that can be deleted are sorted from highest to lowest redundancy score. If the scores are the same, they are sorted from smallest to largest original layer number. The corresponding number of layers at the top of the sorted list are selected as the target deletion layers. For each target deletion layer, the corresponding Transformer block is removed from the model layer container and parameter state, and adjacent retained layers are connected by an identity mapping.

[0042] After deletion, regenerate consecutive layer numbers for the retained layers according to the original layer order, modify the total number of layers in the model configuration, and establish a mapping between the original layer numbers, consecutive new layer numbers, and parameter keys. Resume training using preset calibration data and a fixed validation set, and output the checkpoint with the lowest validation loss. Example 2: Feasibility Example of Target Deletion Layers

[0043] This embodiment only illustrates the process of determining the pruning set and does not advocate for unmeasured task performance results. For the model to be pruned, the electronic device forms a set of guard layers and a set of removable layers based on the layer cluster partitioning results.

[0044] Given a deletion ratio, the product of the deletion ratio and the total number of layers in the model is rounded down to obtain the target number of layers to be deleted. When the target number of layers to be deleted does not exceed the number of layers that can be deleted, an appropriate number of layers are selected from the deletable layers based on redundancy scores; when the target number of layers to be deleted exceeds the number of layers that can be deleted, an infeasibility status is output, and the process can be re-executed after reducing the deletion ratio.

[0045] After deletion, the number of model layers equals the original total number of layers minus the number of layers actually deleted. The reduction in parameters equals the sum of the number of parameter tensors corresponding to all target deleted layers; the reduction in forward computation equals the sum of the computational cost corresponding to all target deleted layers. Task metrics, training recovery time, and actual inference latency should be measured and recorded separately on the specific model. Example 3: Implementation details of similarity calculation

[0046] Example 3 describes the similarity calculation process. For any two Transformer layers with the same type of weight matrix, the column vectors are first normalized, and then the cosine similarity matrix between all column vector pairs of the two weight matrices is calculated.

[0047] Using the first weight matrix as a benchmark, the maximum value of the absolute cosine similarity between each column vector and all column vectors of the second weight matrix is ​​obtained, thus forming the first maximum value set.

[0048] The first set of maximum values ​​is subjected to Gumbel distribution parameter estimation to obtain the first location parameter and scale parameter. After swapping the two weight matrices, the same calculation is performed symmetrically to obtain the second location parameter. The arithmetic mean of the two location parameters is used as the similarity value between the two Transformer layers. When parameter estimation fails to converge, the arithmetic mean of the corresponding set of maximum values ​​can be used as the similarity value.

[0049] The aforementioned similarity calculations mainly involve matrix column vector inner products and distribution parameter estimation, which can be executed in parallel on GPU computing units. For typical Transformer models, the calculation of the inter-layer similarity matrix can be completed as a pre-pruning step. Example 4: System Implementation

[0050] Example 4 Figure 5 As shown, the system includes a processor, at least one GPU computing unit, memory, and modules for inter-layer similarity acquisition, layer cluster identification, redundancy assessment, pruning selection, structure update, and training recovery.

[0051] The inter-layer similarity acquisition module reads weight matrices of the same type and generates a symmetric inter-layer similarity matrix with the total number of layers in the model as the number of rows and columns.

[0052] The layer cluster identification module performs window analysis along the main diagonal and determines continuous layer clusters based on the difference in the average value of non-diagonal elements in adjacent windows.

[0053] The redundancy assessment module calculates redundancy scores based on intra-cluster average similarity and normalized boundary distance, and selects a central representative layer based on intra-cluster average similarity.

[0054] The pruning selection module first locks the starting layer, ending layer, central representative layer, and the first and last layers of the model for each cluster, then checks whether the target number of deleted layers does not exceed the number of deletable layers, and selects the corresponding number of layers from the deletable layers according to the score.

[0055] The structure update module deletes the target Transformer block and its parameter tensors to directly pass the connection to adjacent retained layers, and regenerates the consecutive layer numbers, model layer number, and parameter key mapping.

[0056] The recovery training module performs training on the GPU computing unit using fixed calibration data and validation set, and writes the checkpoint with the lowest validation loss and the mapping from the original layer number to the new layer number into memory. Example 5: Storage Media and Electronic Devices

[0057] This embodiment provides a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the steps of the method described in any embodiment of the present invention.

[0058] This embodiment also provides an electronic device, such as... Figure 6 As shown, the system includes a processor, a GPU computing unit, a memory, and a communication unit, with the components communicating via a bus. The memory stores computer program instructions executable by the processor, which implements the aforementioned structured layer pruning method when executing these instructions.

[0059] The GPU computing unit is used to perform steps involving large-scale matrix operations, including weight matrix reading, column vector normalization, cosine similarity calculation, and resuming fine-tuning training; the memory is used to store the original pre-trained model, the pruned model, the similarity matrix, the layer cluster partitioning results, and the pruning configuration.

[0060] The method of this invention is not limited to the specific model architecture, number of layers, and compression ratio settings in the above embodiments, and can be applied to large language models based on the Transformer architecture. The specific values ​​mentioned above are merely illustrative and do not constitute a limitation on the scope of protection of this invention.

[0061] This invention can also be used in conjunction with other model compression techniques such as quantization. After pruning the structured layers, the retained layers can be further quantized with low bit depth to further reduce deployment overhead in terms of both inference speed and storage efficiency.

[0062] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any modifications, equivalent changes, and modifications made based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for structuring layer pruning of a large language model, executed on an electronic device including a processor, memory, and at least one GPU computing unit, characterized in that, include: S1, read the weight matrix of the same type from each Transformer layer of the pre-trained large language model from the memory, compare each layer pairwise using the same matrix similarity calculation rule, and arrange the obtained similarity values ​​according to the layer number to construct the inter-layer similarity matrix; S2, move a square window of a preset order layer by layer along the main diagonal of the inter-layer similarity matrix, calculate the average value of elements other than the main diagonal elements in each window, determine the layer cluster boundary based on the change of the average value of adjacent windows, and divide all Transformer layers into several continuous layer clusters accordingly. S3. For each layer in each layer cluster, calculate the average similarity between that layer and other layers in the same layer cluster, and calculate the normalized boundary distance based on the layer number distance from that layer to the two boundaries of the layer cluster. The average similarity and the normalized boundary distance are weighted and summed according to two preset non-negative weights to obtain the redundancy score of that layer, where the sum of the two weights is one. The layer with the largest average similarity is selected as the central representative layer of that layer cluster. S4, the starting layer, ending layer, and central representative layer of each layer cluster, as well as the first and last layers of the entire model, are determined as protective layers, and the remaining layers are determined as removable layers. The product of the preset deletion ratio and the total number of layers in the model is rounded down to obtain the target number of layers to be deleted. When the target number of layers to be deleted is not greater than the total number of all deletable layers, all deletable layers are arranged in descending order of redundancy score. When the scores are the same, they are arranged in ascending order of layer number, and the corresponding number of layers at the top of the sort are selected as the target layers to be deleted. When the target number of layers to be deleted is greater than the total number of all layers that can be deleted, output a state that the target number of layers to be deleted is not feasible and stop pruning; S5, the target deletion layer is removed from the layer sequence of the pre-trained large language model, the input of each deleted layer is passed to the next retained layer by an identity mapping, the number of layers, the order of retained layers and the parameter key mapping in the model configuration are updated, the updated model is retrained on the GPU computing unit, and the pruned model after retraining is written to the memory.

2. The method according to claim 1, characterized in that, The matrix similarity calculation rules include: for any two Transformer layers with the same type of weight matrices, calculating the cosine similarity between each column vector of the first weight matrix and each column vector of the second weight matrix; for each column vector of the first weight matrix, obtaining the maximum value of the absolute value of its cosine similarity with each column vector of the second weight matrix, fitting the obtained maximum value set with a Gumbel distribution and obtaining the first position parameter; after swapping the two weight matrices, repeating the calculation and obtaining the second position parameter; using the arithmetic mean of the first position parameter and the second position parameter as the similarity value between the two Transformer layers; when the Gumbel distribution fitting does not converge, using the arithmetic mean of the corresponding maximum value set to replace the position parameter.

3. The method according to claim 1, characterized in that, S2 includes: setting the window order to an integer not less than two and not greater than the total number of layers in the model; moving the square window one layer at a time along the main diagonal of the inter-layer similarity matrix, and sequentially calculating the average value of the non-main diagonal elements in each window and the difference between the latter and former terms of the average value of adjacent windows; when the difference is not less than the boundary detection threshold, marking the position after the center of the corresponding window as a candidate boundary; when multiple candidate boundaries are consecutively adjacent, only the candidate boundary with the largest difference is retained, and when the differences are the same, the candidate boundary with the smaller layer number is retained; when the window order is equal to the total number of layers in the model or there are no candidate boundaries, all Transformer layers are determined as a layer cluster.

4. The method according to claim 1, characterized in that: When a layer cluster contains two or more layers, the sum of the similarities between the target layer and all other layers in the same layer cluster is divided by the number of layers in the layer cluster minus one to obtain the average similarity. When a layer cluster contains only one layer, the average similarity is set to zero. The smaller of the layer number distances from the target layer to the two side boundaries is divided by the maximum and minimum boundary distances that can be obtained within the layer cluster to obtain the normalized boundary distance. When the average similarities of multiple layers are the same, the layer closer to the center of the layer cluster is determined as the central representative layer. If the distances are still the same, the layer with the smaller layer number is selected.

5. The method according to claim 1, characterized in that, Before performing deletion, the electronic device calculates the total number of protective layers and the number of removable layers; when the target number of deleted layers is greater than the number of removable layers, it outputs a state that the target number of deleted layers is infeasible and stops pruning, without satisfying the deletion ratio by deleting protective layers.

6. The method according to claim 1, characterized in that, The updated model configuration includes: deleting the parameter tensor corresponding to the target deleted layer, regenerating consecutive layer numbers according to the order of the retained layers in the original model, modifying the total number of Transformer layers in the model configuration file, and synchronously updating the mapping relationship between the parameter keys used for model loading and the regenerated consecutive layer numbers.

7. The method according to claim 1, characterized in that, The recovery training uses a preset calibration dataset to perform 100 to 500 parameter update steps on the pruned model. Before the recovery training begins, the initial checkpoint of the pruned model is saved. During the recovery training, the model checkpoint with the lowest validation loss is saved based on the loss of the fixed validation set, and this model checkpoint is used as the output model.

8. A structured layer pruning system for large language models, characterized in that, It includes a processor, at least one GPU computing unit, and a memory connected to the processor and GPU computing unit. The memory stores: an inter-layer similarity acquisition module, used to read the same type of weight matrix of each Transformer layer and construct an inter-layer similarity matrix; The layer cluster identification module is used to determine continuous layer clusters based on the change in the average value of non-diagonal elements in the main diagonal sliding window of the inter-layer similarity matrix; the redundancy evaluation module is used to calculate the redundancy score of each layer based on the average intra-cluster similarity and normalized boundary distance, and to determine the central representative layer of each layer cluster. The pruning selection module is used to protect the starting layer, ending layer, and central representative layer of each cluster, as well as the first and last layers of the model, and to select the target deletion layer from the remaining layers according to the redundancy score. The structure update module is used to delete the target deletion layer, connect adjacent retained layers with identity mapping, and update the model layer number, layer order, and parameter key mapping; The recovery training module is used to perform recovery training on the updated model on the GPU computing unit and write the recovered pruned model into the memory.

9. A computer-readable storage medium having stored thereon computer program instructions, which, when executed by a processor, cause an electronic device to perform the method of any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and at least one GPU computing unit, characterized in that, The memory stores computer program instructions, which, when executed by the processor, cause the electronic device to perform the method of any one of claims 1 to 7.

Images