Parameter Pruning Method, Device, Equipment and Readable Storage Medium for Large Language Model

Abstract

An embodiment of the present application provides a method, apparatus, device, and readable storage medium for parameter pruning of a large language model. The method includes: obtaining a first metric matrix of each network layer in a preset large language model and determining a pruning rate of the corresponding network layer; determining multiple importance scores for multiple attention heads included in each network layer and constructing a second metric matrix; determining a first calibration matrix of the second metric matrix according to the pruning rate and the second metric matrix, and obtaining a first mask matrix based on the sparse distribution characteristics of the first calibration matrix; pruning the key channels and value channels; determining a second calibration matrix of the second metric matrix according to a target quantity and the second metric matrix, and obtaining a second mask matrix based on the sparse distribution characteristics of the second calibration matrix; pruning the query channels and output channels associated with the second mask matrix to obtain a large language model. In this way, the storage space of the large language model can be reduced and the computing efficiency of the large language model can be accelerated.

Description

Technical Field

[0001] This application relates to the technical field of network model compression, and particularly to a method, apparatus, device and readable storage medium for parameter pruning of large language models. Background Art

[0002] Large Language Model (LLM) is an artificial intelligence technology based on deep learning and is also one of the core research contents of natural language processing. By training on a vast amount of text data, large language models can understand and generate natural language texts, and perform various natural language processing tasks including text generation, translation, question answering, etc.

[0003] However, due to the huge number of parameters of large language models, which often start from billions, the requirements for computational volume and storage overhead are very high. Therefore, it is necessary to compress the weight space of large language models to reduce the computational complexity of the models and improve the computational efficiency.

[0004] In related technologies, through unstructured pruning, some elements in the weights of large language models can be set to zero to reduce the computational complexity of the models. However, the large language models after unstructured pruning still need to store the complete weight matrix, which means that the total storage requirement has not been substantially reduced; moreover, the models still need to execute the same computational steps as the original models. Therefore, unstructured pruning can neither compress the storage space of large language models nor accelerate the computational efficiency of large language models. Summary of the Invention

[0005] The main objective of the embodiments of this application is to propose a method, apparatus, device and readable storage medium for parameter pruning of large language models, which can reduce the storage space of large language models and accelerate the computational efficiency of large language models.

[0006] To achieve the above objective, in the first aspect of the embodiments of this application, a method for parameter pruning of large language models is proposed, and the method includes:

[0007] Obtain a first metric matrix for each network layer in a plurality of network layers of a preset large language model, and determine a pruning rate corresponding to the network layer based on the first metric matrix;

[0008] For a plurality of attention heads included in each network layer, determine a plurality of importance scores according to the corresponding first metric matrix, and construct a second metric matrix according to the plurality of importance scores;

[0009] Determine the first calibration matrix of the second metric matrix according to the pruning rate and the magnitude relationship of multiple importance scores in the second metric matrix, and based on the sparse distribution characteristics of the first calibration matrix, determine the corresponding first mask positions to obtain a first mask matrix;

[0010] Prune the key channels and value channels associated with the attention heads indicated by the first mask matrix;

[0011] Determine the target number of the attention heads indicated by the first mask matrix, determine the second calibration matrix of the second metric matrix according to the target number and the magnitude relationship of multiple importance scores in the second metric matrix, and based on the sparse distribution characteristics of the second calibration matrix, determine the corresponding second mask positions to obtain a second mask matrix;

[0012] Prune the query channels and output channels associated with the attention heads indicated by the second mask matrix to obtain a large language model.

[0013] Correspondingly, a second aspect of the embodiments of the present application proposes a parameter pruning device for a large language model, and the device includes:

[0014] An acquisition module, configured to acquire a first metric matrix of each network layer in multiple network layers of a preset large language model, and determine a pruning rate corresponding to the network layer based on the first metric matrix;

[0015] A construction module, configured to, for multiple attention heads included in each network layer, determine multiple importance scores according to the corresponding first metric matrix, and construct a second metric matrix according to the multiple importance scores;

[0016] A first determination module, configured to determine the first calibration matrix of the second metric matrix according to the pruning rate and the magnitude relationship of multiple importance scores in the second metric matrix, and based on the sparse distribution characteristics of the first calibration matrix, determine the corresponding first mask positions to obtain a first mask matrix;

[0017] A first pruning module, configured to prune the key channels and value channels associated with the attention heads indicated by the first mask matrix;

[0018] A second determination module, configured to determine the target number of the attention heads indicated by the first mask matrix, determine the second calibration matrix of the second metric matrix according to the target number and the magnitude relationship of multiple importance scores in the second metric matrix, and based on the sparse distribution characteristics of the second calibration matrix, determine the corresponding second mask positions to obtain a second mask matrix;

[0019] A second pruning module, configured to prune the query channels and output channels associated with the attention heads indicated by the second mask matrix, so as to obtain a large language model.

[0020] In some embodiments, the construction module is further configured to:

[0021] For each of the multiple attention heads included in each network layer, average the matrix columns corresponding to the linear layers of the first metric matrix to obtain a first average metric matrix;

[0022] Determine the correspondence between the multiple matrix features in the first average metric matrix and the multiple attention heads, and adjust the first average metric matrix based on the correspondence to obtain a second average metric matrix;

[0023] Based on the second average metric matrix, determine the importance scores corresponding to each attention head, and construct a second metric matrix based on the multiple importance scores corresponding to the multiple attention heads.

[0024] In some embodiments, the construction module is further configured to:

[0025] Obtain the target group corresponding to each attention head;

[0026] Based on the grouping relationship between the multiple attention heads, adjust the positions of the importance scores corresponding to each attention head in the second average metric matrix to obtain a second metric matrix arranged according to the target group.

[0027] In some embodiments, the first determination module is further configured to:

[0028] According to the pruning rate, determine a first quantity of attention heads to be pruned;

[0029] According to the first quantity, determine a first pruning threshold from the second metric matrix according to the magnitude relationship of the multiple importance scores;

[0030] Compare the first pruning threshold with each importance score in the second metric matrix respectively, and determine the calibration value corresponding to each importance score according to the magnitude relationship between the first pruning threshold and each importance score;

[0031] Based on the multiple calibration values corresponding to the multiple importance scores, determine a first calibration matrix of the second metric matrix.

[0032] In some embodiments, the first determination module is further configured to:

[0033] Obtain the multiple calibration values corresponding to the multiple attention heads included in each target group in the first calibration matrix;

[0034] Determine a target group value corresponding to the target group based on the multiple calibration values corresponding to each target group;

[0035] Obtain a preset mask threshold, and compare the target group value with the mask threshold to obtain a comparison result; wherein, the comparison result of each target group of the first calibration matrix is associated with the sparse distribution characteristic of the corresponding target group;

[0036] Based on the comparison result, adjust the multiple calibration values corresponding to the multiple attention heads included in the target group in the first calibration matrix to obtain multiple target calibration values corresponding to the multiple attention heads;

[0037] Determine a corresponding first mask position based on the multiple target calibration values corresponding to the multiple target groups to obtain a first mask matrix.

[0038] In some embodiments, the obtaining module is further configured to:

[0039] Obtain multiple linear layers corresponding to each network layer included in the large language model, and in each network layer, determine multiple weight matrices corresponding to each linear layer;

[0040] Determine the element positions of the matrix elements in each weight matrix;

[0041] Obtain the input data of the linear layer corresponding to each weight matrix, and calculate multiple norm values corresponding to the multiple input dimensions of the input data;

[0042] Based on the element position of each matrix element, determine a target norm value of the corresponding input dimension from the multiple norm values;

[0043] Determine a measurement criterion for each matrix element according to each matrix element and the target norm value;

[0044] Generate a layer measurement criterion corresponding to each linear layer based on the multiple measurement criteria of the multiple matrix elements corresponding to the multiple weight matrices;

[0045] Obtain a first measurement matrix corresponding to each network layer based on the multiple layer measurement criteria corresponding to the multiple linear layers.

[0046] In some embodiments, the obtaining module is further configured to:

[0047] Calculate a measurement average value for the first measurement matrix;

[0048] For each linear layer in each of the said network layers, obtain a preset threshold constant, and obtain a target threshold according to the product of the metric average value and the threshold constant;

[0049] Determine a second quantity of a plurality of target metric elements greater than the target threshold from among the plurality of metric elements of the first metric matrix;

[0050] Obtain a third quantity of the plurality of metric elements of the first metric matrix, and obtain a singular value ratio based on the ratio of the second quantity to the third quantity;

[0051] Perform a normalization process on the singular value ratio to obtain a target singular value ratio;

[0052] Determine the pruning rate of each of the said network layers based on the target singular value ratios corresponding to the plurality of linear layers corresponding to each of the said network layers.

[0053] Correspondingly, a third aspect of the embodiments of the present application provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the parameter pruning method of the large language model according to any one of the embodiments of the first aspect of the present application.

[0054] Correspondingly, a fourth aspect of the embodiments of the present application provides a computer-readable storage medium, the storage medium stores a computer program, and when the computer program is executed by a processor, it implements the parameter pruning method of the large language model according to any one of the embodiments of the first aspect of the present application.

[0055] In the embodiments of the present application, the pruning rate of each network layer in a preset large language model is determined by obtaining the first metric matrix of each network layer and based on the first metric matrix; for the multiple attention heads included in each network layer, multiple importance scores are determined according to the corresponding first metric matrix, and a second metric matrix is constructed based on the multiple importance scores; a first calibration matrix of the second metric matrix is determined according to the pruning rate and the magnitude relationship of the multiple importance scores in the second metric matrix, and based on the sparse distribution characteristics of the first calibration matrix, the corresponding first mask positions are determined to obtain a first mask matrix; pruning is performed on the key channels and value channels associated with the attention heads indicated by the first mask matrix; the target number of the attention heads indicated by the first mask matrix is determined, a second calibration matrix of the second metric matrix is determined according to the target number and the magnitude relationship of the multiple importance scores in the second metric matrix, and based on the sparse distribution characteristics of the second calibration matrix, the corresponding second mask positions are determined to obtain a second mask matrix; pruning is performed on the query channels and output channels associated with the attention heads indicated by the second mask matrix to obtain the large language model. In this way, the pruning rate of each network layer of the large language model can be determined through the first metric matrix of each network layer, so as to adaptively adjust the pruning granularity according to different layers of the model and avoid the decline of model performance caused by excessive pruning. Moreover, by determining the contribution degree of each attention head to the model output through the importance scores of each attention head, it can be ensured that the pruning operation will not significantly affect the performance of the model. Further, the key channels and value channels of the attention heads that need to be pruned are determined through the first mask matrix, effectively reducing the number of parameters of the model. After pruning the key channels and value channels, the query channels and output channels of the corresponding attention heads are further pruned according to the second mask matrix. In this way, by pruning based on the structural characteristics of the output channels of the key projection layer and value projection layer, query projection layer and output projection layer of the model, the pruning process of different parts can be controlled more finely, ensuring that important parts are retained, so that the pruned large language model does not require additional fine-tuning training and can still maintain the original performance, thereby accelerating the computational efficiency of the model. In summary, the present application can reduce the storage space of the large language model and accelerate the computational efficiency of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a schematic structural diagram of a parameter pruning system for a large language model provided by an embodiment of the present application;

[0057] Figure 2 is a flowchart of a parameter pruning method for a large language model provided by an embodiment of the present application;

[0058] Figure 3 is a flowchart of calculating the first mask matrix and the second mask matrix provided by an embodiment of the present application;

[0059] Figure 4It is the overall flowchart of the parameter pruning method for the large language model provided by the embodiments of the present application;

[0060] Figure 5 It is the schematic diagram of the functional modules of the parameter pruning device for the large language model provided by the embodiments of the present application;

[0061] Figure 6 It is the schematic diagram of the hardware structure of the computer device provided by the embodiments of the present application. Detailed implementation manners

[0062] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device or the flowchart. Terms such as "first" and "second" in the specification, claims and the above-mentioned drawings are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0065] A large language model (LLM) is an artificial intelligence technology based on deep learning and is also one of the core research contents of natural language processing. By training on a large amount of text data, the large language model can understand and generate natural language text and perform various natural language processing tasks including text generation, translation, question answering, etc.

[0066] However, due to the huge number of parameters of the large language model, which often starts from billions, the requirements for computational volume and storage overhead are very high. Therefore, it is necessary to compress the weight space of the large language model to reduce the complexity of model calculation and improve the computational efficiency.

[0067] In the related art, through unstructured pruning, some elements in the weights of a large language model can be set to zero to reduce the computational complexity of the model. However, the large language model after unstructured pruning still needs to store the complete weight matrix, which means that the total storage requirement is not substantially reduced; moreover, the model still needs to execute the same computational steps as the original model. Therefore, unstructured pruning can neither compress the storage space of the large language model nor accelerate the computational efficiency of the large language model.

[0068] Based on this, the embodiments of the present application provide a method, device, equipment and readable storage medium for parameter pruning of a large language model, which can reduce the storage space of the large language model and accelerate the computational efficiency of the large language model.

[0069] The method, device, equipment and readable storage medium for parameter pruning of the large language model provided by the embodiments of the present application are specifically described through the following embodiments. First, the parameter pruning system of the large language model in the embodiments of the present application is described.

[0070] Please refer to Figure 1 , in some embodiments, the embodiments of the present application provide a parameter pruning system for a large language model, including a terminal 11 and a server side 12.

[0071] Exemplarily, the terminal 11 can be a smart phone, a personal computer, a workstation, etc. The terminal 11 can provide a graphical interface or a command-line tool, enabling users to load the large language model, select a calibration data set, set pruning parameters (such as pruning rate, pruning threshold, etc.). The terminal 11 can also perform model loading and data preprocessing, call the pruning algorithm, and display the pruning results to help users evaluate the pruning effect, etc.

[0072] Furthermore, the server side 12 can be a high-performance computing server, a cluster server, a dedicated AI server, etc. The server side 12 can receive the model and parameters sent by the terminal 11, load the large language model and the data set, execute the parameter pruning method, and allocate computing resources according to the task requirements to ensure the efficient operation of the pruning task. After pruning is completed, the server side 12 can store the pruned large language model and related results on the disk or in the database, and return the results to the terminal 11 for the terminal 11 to display the results.

[0073] The parameter pruning method of the large language model in the embodiments of the present application can be described through the following embodiments.

[0074] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for enabling the normal operation of the embodiment of the present application will be obtained.

[0075] In the embodiment of the present application, the parameter pruning device of a large language model will be described from the dimension of the parameter pruning device of the large language model, and the parameter pruning device of the large language model can be specifically integrated in a computer device. Figure 2 , Figure 2 The step flow chart of the parameter pruning method of the large language model provided in the embodiment of the present application, the embodiment of the present application takes the parameter pruning device of the large language model as an example, and when the processor on the terminal or server executes the program instructions corresponding to the parameter pruning method of the large language model, the specific process is as follows:

[0076] Step 101: obtain a first metric matrix of each network layer in a plurality of network layers of a preset large language model, and determine a pruning rate of the corresponding network layer based on the first metric matrix.

[0077] In some embodiments, in order to retain the effectiveness and accuracy of the model as much as possible while reducing the computing cost, the first metric matrix of each network layer in the multiple network layers of the preset large language model can be obtained, and the pruning rate of the corresponding network layer can be determined based on this, so as to effectively reduce the model size in the subsequent pruning process and improve the computing speed of the model.

[0078] Among them, the preset large language model can be an LLaMA-3 model, or other models based on the Transformer architecture. The core components of the preset large language model include a multi-head attention module and a feedforward neural network (Multilayer Perceptron, MLP) module, which are used to process complex natural language processing tasks.

[0079] Among them, the network layer can be the basic building block in a large language model, such as DecoderLayer. Each network layer can contain multiple submodules, such as multi-head self-attention modules and MLP modules. Among them, the multi-head attention module includes linear layers such as q_proj (query projection), k_proj (key projection), v_proj (value projection) and o_proj (output projection), and the MLP module can contain linear layers such as up_proj (upward projection), down_proj (downward projection) and gate_proj (gated projection).

[0080] Among them, the first metric matrix can be used to measure the importance of each part in the network layer. Specifically, for a network layer (nn.Linear layer), its first metric matrix is ​​calculated based on the weight matrix and input data of the layer, reflecting the importance of each part to the final output.

[0081] The pruning rate can be the proportion of parameters planned to be removed during the pruning process. For example, if the set pruning rate is 30%, it means that 30% of the parameters will be considered unimportant and will be removed from the preset large language model.

[0082] In some implementations, the weight matrix of each network layer may be obtained, and the L2 norm of each weight matrix may be calculated row by row, and the obtained L2 norm matrix is ​​the first metric matrix.

[0083] Furthermore, the pruning rate can be calculated based on the statistical characteristics of the first metric matrix (such as mean, standard deviation, etc.). Specifically, based on the calculated first metric matrix, the pruning rate of each network layer can be determined by a specific algorithm (such as calculating the singular value ratio and normalizing it) to determine how many proportions of parameters of each network layer will be removed in the subsequent pruning process.

[0084] Through the above method, the first metric matrix and pruning rate of each network layer can be accurately obtained, so as to more effectively reduce the number of model parameters and computational overhead in the subsequent model pruning and optimization process, while maintaining the performance of the model as much as possible.

[0085] In some embodiments, in order to systematically analyze and quantify the importance of each component in the preset large language model, the layer metric criteria of all linear layers in each network layer can be integrated to generate a first metric matrix corresponding to the network layer to lay the foundation for efficient and accurate pruning operations. For example, in step 101, "obtaining the first metric matrix of each network layer in multiple network layers of the preset large language model" may include:

[0086] (101.a1) Obtain multiple linear layers corresponding to each network layer included in the large language model, and in each network layer, determine multiple weight matrices corresponding to each linear layer;

[0087] (101.a2)Determine the element positions of the matrix elements in each weight matrix;

[0088] (101.a3)Obtain the input data of the linear layer corresponding to each weight matrix, and calculate multiple norm values corresponding to multiple input dimensions of the input data;

[0089] (101.a4)Based on the element positions of each matrix element, determine the target norm value of the corresponding input dimension from multiple norm values;

[0090] (101.a5)Determine the measurement criteria of each matrix element according to each matrix element and the target norm value;

[0091] (101.a6)Generate the layer measurement criteria corresponding to each linear layer based on the multiple measurement criteria of the multiple matrix elements corresponding to the multiple weight matrices;

[0092] (101.a7)Obtain the first measurement matrix corresponding to each network layer based on the multiple layer measurement criteria corresponding to the multiple linear layers.

[0093] Among them, a linear layer can be a basic unit that performs a linear transformation in a neural network. It can transform the input data through a weight matrix and can add a bias term.

[0094] Among them, a weight matrix can be a parameter matrix in a linear layer, used to map the input data from one space to another space, and its size is determined by the number of input features and the number of output features.

[0095] Among them, a matrix element can be a single value in a weight matrix, representing the connection strength between a specific input feature and an output feature.

[0096] Among them, the element position can be the row and column coordinates of the matrix element in its corresponding weight matrix.

[0097] Among them, the input data can be the data input to the linear layer, and it can be a multi-dimensional tensor.

[0098] Among them, the input dimension can be a single feature dimension in the input data, representing a characteristic or attribute of a sample.

[0099] Among them, the norm value can be a measurement method for measuring the size of a vector or a matrix. For an input feature, the norm value can be the L2-norm (Euclidean norm), used to quantify the importance of the input feature.

[0100] Among them, the target norm value can be the norm value corresponding to each matrix element. For example, for the matrix element in the i-th row and j-th column, its target norm value is the norm value of the j-th input dimension.

[0101] Among them, the metric criterion can be a standard used to evaluate the importance of each element in the weight matrix.

[0102] Among them, the layer metric criterion can be an overall evaluation standard obtained by comprehensively considering the metric criteria of all weight matrices within a linear layer, reflecting the importance of this layer.

[0103] Exemplarily, in a pre-set large language model, each network layer (such as each DecoderLayer) may contain multiple linear layers, such as nn.Linear layers. These linear layers can be q_proj, k_proj, v_proj, o_proj in the multi-head attention module, or up_proj, down_proj, etc. in the feed-forward network.

[0104] Furthermore, for each linear layer, the position of each matrix element in its weight matrix can be determined. The weight matrix can be a two-dimensional array, where each matrix element has a specific row and column index. And, the input data of each linear layer can be obtained. The input data is usually a three-dimensional tensor with a shape of [N, L, Cin], where N is the batch size, L is the sequence length, and Cin is the feature dimension size. For each feature of the input data, its corresponding L2-norm (i.e., the norm value) can be calculated.

[0105] Furthermore, for each matrix element in the weight matrix, its corresponding target norm value of the input feature can be determined from multiple pre-computed norm values according to its element position. For example, for the matrix element in the i-th row and j-th column, the norm value of the j-th input dimension can be determined as its target norm value. Then, the metric criterion of each matrix element is calculated using the following formula :

[0106] ;

[0107] Among them, is the element in the i-th row and j-th column of the weight matrix, is the norm value of the matrix element in the j-th dimension of the input matrix corresponding to the input data.

[0108] Furthermore, for each linear layer in the network layer, the metrics of all matrix elements can be aggregated to generate the layer metric of the linear layer. The calculation of the layer metric can be achieved by summarizing the metrics of all matrix elements, or by calculating the average, maximum or other statistics of the metrics of all elements, and the embodiments of the present application do not impose too many restrictions on this.

[0109] Furthermore, for each network layer, the layer metrics of all linear layers can be aggregated to generate the first metric matrix of the network layer. This can be achieved by summarizing all layer metrics, or calculating the average, maximum, or other statistics of all layer metrics. The embodiments of the present application do not impose too many restrictions on this.

[0110] Through the above steps, a first metric matrix can be generated for each network layer in the preset large language model. The first metric matrix can be used for subsequent pruning operations to reduce the number of parameters and computational overhead of the model while maintaining the performance of the model as much as possible.

[0111] In some implementations, in order to achieve efficient and accurate pruning operations, the pruning rate of each network layer may be determined to ensure the effectiveness and rationality of the pruning process. Exemplarily, the "determining the pruning rate of the corresponding network layer based on the first metric matrix" in step 101 may include:

[0112] (101.b1) Calculate the metric average for the first metric matrix;

[0113] (101.b2) For each linear layer in each network layer, obtain the preset threshold constant, and obtain the target threshold based on the product of the metric average and the threshold constant;

[0114] (101.b3) determining, from the plurality of metric elements of the first metric matrix, a second number of a plurality of target metric elements that are greater than a target threshold;

[0115] (101.b4) obtaining a third number of the plurality of metric elements of the first metric matrix, and obtaining a singular value ratio based on a ratio of the second number to the third number;

[0116] (101.b5) Standardize the singular value ratio to obtain the target singular value ratio;

[0117] (101.b6) Determine the pruning rate for each network layer based on the target singular value ratio corresponding to the multiple linear layers corresponding to each network layer.

[0118] The threshold constant may be a preset fixed value (such as Hyper_m=5, etc., which may be set according to actual conditions), which is used to be multiplied by the metric average to generate a target threshold.

[0119] Among them, the target threshold can be a value obtained by multiplying the metric average by a threshold constant, which can be used to distinguish whether a metric element is important, that is, it is a benchmark for judging whether a metric element should be retained or pruned.

[0120] Among them, the metric element can be a single element in the first average metric matrix, representing the importance score of a specific part.

[0121] Among them, the second quantity can be the number of metric elements greater than the target threshold, and these elements are considered important and should be retained.

[0122] Among them, the third quantity can be the total number of all metric elements in the first average metric matrix.

[0123] Among them, the singular value ratio can be the ratio of the second quantity to the third quantity, reflecting the proportion of important metric elements in the total elements.

[0124] Among them, the target singular value ratio can be the result after normalizing the singular value ratio. The normalization process makes the comparison between different network layers or models more fair and reasonable, and helps to determine the pruning rate.

[0125] Exemplarily, first, the average value of all elements in the first metric matrix (layer_wmetric) can be calculated to obtain the metric average. Then, the preset threshold constant Hyper_m is obtained, and the target threshold is obtained according to the product of the threshold constant and the metric average. The specific process is as follows:

[0126] max_shred = torch.mean(layer_wmetric) Hyper_m;

[0127] Furthermore, the number of elements in the first metric matrix greater than the target threshold max_shred can be counted as the second quantity. The specific process is as follows:

[0128] count = (layer_wmetric > max_shred).sum().item();

[0129] Furthermore, the total number of elements in layer_wmetric can be calculated as the third quantity, and then the ratio of the second quantity to the third quantity is calculated to obtain the singular value ratio. The specific process is as follows:

[0130] total_params = layer_wmetric.numel();

[0131] outlier_ratio = float(count) / total_params 100;

[0132] Furthermore, the outlier ratios of all network layers can be normalized, that is, the outlier ratios of all layers are normalized to the range of [0, 1]. Specifically, after normalization using the minimum and maximum values, a scaling process can be performed (for example, multiplying by Lamda 2, and Lamda is usually 0.2) to obtain the target outlier ratio. The specific process is as follows:

[0133] ;

[0134] Furthermore, adjustments can be made according to the target outlier ratios of all network layers to consider the pruning rate. The specific process is as follows:

[0135] ;

[0136] Furthermore, the final pruning rate can be calculated by the following formula:

[0137] ;

[0138] Through the above method, the pruning rate corresponding to each network layer can be dynamically calculated according to the outlier ratio and accurately applied to the pruning of the preset large language model to achieve the purpose of effective compression and accelerated inference of the model.

[0139] Step 102: For multiple attention heads included in each network layer, determine multiple importance scores according to the corresponding first metric matrix, and construct a second metric matrix according to the multiple importance scores.

[0140] In some embodiments, to evaluate the importance of each attention head, for multiple attention heads included in each network layer, their importance scores can be determined according to the corresponding first metric matrix, and a second metric matrix can be constructed according to these scores to facilitate understanding of the key parts of the model internal structure and also provide clear guidance for subsequent pruning operations.

[0141] Among them, an attention head can include a set of weight matrices (such as q_proj, k_proj, v_proj) for calculating queries (Query), keys (Key), and values (Value). In the Transformer architecture of the large language model, each attention module usually contains multiple attention heads, and each attention head independently performs the self-attention mechanism to capture the relationships between different parts of the input sequence.

[0142] Among them, the importance score can be a numerical value calculated for each attention head based on the metric criterion in its corresponding first metric matrix, reflecting the importance of this attention head in the entire pre-set large language model. For example, if the importance score is relatively low, it indicates that the importance of this attention head is relatively low, and vice versa.

[0143] Among them, the second metric matrix can be a newly constructed matrix that contains the importance scores of each attention head, which provides a basis for subsequent pruning operations to help determine which attention heads should be retained or removed.

[0144] In some embodiments, the shape of the first metric matrix can be adjusted to a shape corresponding to the number of attention heads, such as (head_num, 128), where head_num represents the number of attention heads in the first metric matrix, and 128 represents that the dimension of the feature vector processed by each attention head is 128.

[0145] Furthermore, after adjusting the shape, the sum of each row can be calculated, that is, the sum of all feature vectors corresponding to each attention head, and the importance score of each attention head can be obtained. Based on the multiple importance scores corresponding to the multiple attention heads included in the network layer, the second metric matrix can be constructed.

[0146] In some embodiments, after determining the multiple importance scores corresponding to the multiple attention heads, the multiple attention heads can also be grouped, and the second metric matrix can be constructed according to the grouping results, so as to facilitate subsequent pruning of attention heads through group sparsity and improve the pruning efficiency. The basis for grouping can be determined according to the actual situation. For example, grouping can be performed according to the number of attention heads included in each group in the order of the attention heads, or grouping can be performed in other ways. The embodiments of the present application do not make specific limitations on this.

[0147] By calculating the importance score for each attention head and constructing the second metric matrix, more refined and effective pruning is achieved, which helps to improve the pruning efficiency and accuracy without affecting the model performance.

[0148] Please refer to Figure 3, in some embodiments, the present application obtains the first mask matrix and the second mask matrix by setting a grouped sparse pruning mask function (get_wanda_sp_prune_mask function). Specifically, the average of each matrix column in the first metric matrix can be calculated to obtain the first average metric matrix, and the shape of the first metric matrix can be adjusted to obtain the second average metric matrix. Then, according to the pre-calculated pruning rate, the first pruning threshold is calculated, the target group corresponding to each attention head is obtained, and the second average metric matrix is adjusted based on the grouping relationship between multiple attention heads to obtain the second metric matrix, so as to facilitate the subsequent determination of the sparsity of each target group and improve the pruning efficiency. Then, using the second metric matrix and the first pruning threshold, the first mask matrix is determined to determine the channels that need to be pruned and perform efficient pruning, effectively releasing the storage space of the model. Further, the number of groups of the pruned target groups can be calculated, and based on this, the target number of the pruned attention heads can be calculated. The second mask matrix is calculated according to the target number and the second metric matrix, and the first mask matrix and the second mask matrix are returned to achieve accurate pruning. During this process, in order to ensure the effectiveness and rationality of the process, the second metric matrix can be constructed based on multiple importance scores to effectively evaluate and quantify the importance of each attention head in the large language model, and at the same time lay a foundation for efficient and accurate pruning operations. For example, step 102 may include:

[0149] (102.1) For multiple attention heads included in each network layer, the matrix columns corresponding to each linear layer of the first metric matrix are averaged to obtain the first average metric matrix;

[0150] (102.2) Determine the correspondence between multiple matrix features in the first average metric matrix and multiple attention heads, and adjust the first average metric matrix based on the correspondence to obtain the second average metric matrix;

[0151] (102.3) Based on the second average metric matrix, determine the importance score corresponding to each attention head, and construct the second metric matrix based on the multiple importance scores corresponding to multiple attention heads.

[0152] Among them, the first average metric matrix can be a new matrix obtained by averaging the matrix columns corresponding to each linear layer in the first metric matrix of each network layer, which reflects the average importance of each attention head in each linear layer.

[0153] Among them, the second average metric matrix can be a new matrix obtained by adjusting the first average metric matrix considering the correspondence between the attention head and the matrix feature, and is used to reflect the actual contribution of each attention head.

[0154] In some embodiments, the average importance of each attention head can be initially evaluated by averaging the matrix columns corresponding to each linear layer in the first metric matrix of each network layer. Then, based on the correspondence between the attention head and the matrix features, the first average metric matrix is adjusted to more accurately reflect the importance of each attention head. Finally, using the information in the second average metric matrix, the importance score of each attention head is calculated, and these scores are aggregated into a new matrix to guide subsequent pruning operations.

[0155] Exemplarily, if each network layer of the pre-set large language model contains 64 attention heads, i.e., head_num = 64, and the feature dimension processed by each attention head is 128. Suppose a certain network layer corresponds to the first metric matrix W_metric, and its size is the same as the weight matrix corresponding to this network layer, for example, both are 8192×8192. Then, when calculating the importance score corresponding to each attention head, the columns of the first metric matrix can be averaged to obtain the first average metric matrix. Each row of the first average metric matrix represents the average importance score of a feature dimension, and the size of the first average metric matrix is 1×8192.

[0156] Furthermore, according to the correspondence between multiple matrix features in the first average metric matrix and multiple attention heads, the first average metric matrix can be further adjusted to obtain the second average metric matrix to ensure its correct correspondence with the attention heads.

[0157] Furthermore, the rows of the second average metric matrix can be summed separately, that is, the sum of all feature vectors included in each attention head is calculated to obtain the importance score corresponding to each attention head. Finally, multiple attention heads will correspond to a 64×1 matrix (i.e., the second metric matrix), where each element represents the importance score of an attention head for subsequent pruning decisions.

[0158] By calculating the importance score of each attention head, the parameters with the greatest output contribution in the large language model can be accurately evaluated, facilitating subsequent implementation of more fine-grained pruning.

[0159] In some embodiments, in order to more effectively reduce the parameter storage space and accelerate the pruning speed of the model, the attention heads can be further grouped to obtain the second metric matrix, so as to utilize group sparsity for pruning. For example, (102.3) "Constructing the second metric matrix based on multiple importance scores corresponding to multiple attention heads" can include:

[0160] (102.3.1) Obtaining the target group corresponding to each attention head;

[0161] (102.3.2) Based on the grouping relationship between the multiple attention heads, the position of the importance score of each attention head corresponding to the second average metric matrix is ​​adjusted to obtain a second metric matrix arranged according to the target grouping.

[0162] The target grouping may be a plurality of groups obtained by grouping the attention heads according to certain rules or standards. The importance scores (each importance score corresponds to an attention head) within each target group are grouped in order, or the attention heads within each target group have similar importance or functional characteristics.

[0163] The grouping relationship may be a description of the grouping association between different attention heads. For example, attention head 1 and attention head 2 may belong to the same target group, and attention head 1 and attention head 3 may belong to different target groups.

[0164] In some embodiments, the attention heads can be directly grouped evenly according to the predefined parameter num_key_value_group. For example, if the multi-head attention module of the large language model contains head_num=64 attention heads, set num_key_value_group=8, that is, the 64 attention heads need to be divided into 8 groups, each group will contain 8 attention heads, thus, the shape of the matrix can be adjusted based on the attention scores in each target group, specifically, the shape of the matrix can be adjusted to (head_num / num_key_value_group, num_key_value_group), and a second metric matrix of 8×8 is obtained. Alternatively, the position of each attention head can be adjusted after grouping, and the second metric matrix can be obtained directly after grouping and marking.

[0165] By grouping the attention heads, it is easier to use group sparsity for subsequent pruning, which helps to simplify the pruning process while still achieving a certain degree of structured pruning.

[0166] Step 103, determining a first calibration matrix of the second metric matrix according to the size relationship between the pruning rate and multiple importance scores in the second metric matrix, and determining the corresponding first mask position based on the sparse distribution characteristics of the first calibration matrix to obtain a first mask matrix.

[0167] In some embodiments, in order to improve the efficiency and accuracy of pruning decisions, the sparse distribution characteristics of the matrix can be determined based on the size relationship of multiple importance scores, so as to determine the corresponding mask position, thereby ensuring that the pruning operation of the large language model can maintain its performance as much as possible while reducing parameters.

[0168] Among them, the first calibration matrix can be a matrix generated after calibration in the second metric matrix based on the pruning rate and the magnitude of each importance score in the second metric matrix.

[0169] Among them, the sparse distribution characteristic can be the distribution of non-zero elements (i.e., important parts) and zero elements (i.e., unimportant parts) in the first calibration matrix. After grouping according to the attention heads, the sparse distribution characteristic can be the sparse distribution characteristic corresponding to each target group.

[0170] Among them, the first mask position can be the position marked in the first calibration matrix to indicate whether to retain (True or 1) or prune (False or 0), which is used to indicate the attention heads that need to be retained and the attention heads that need to be removed.

[0171] Among them, the first mask matrix can be a binary matrix (True / False, or 1 / 0) generated based on the first mask position for actual pruning operations. True indicates retaining the weight or attention head at that position, and False indicates pruning.

[0172] In some embodiments, the first pruning threshold corresponding to the second metric matrix can be determined according to the pruning rate. In the second metric matrix, the importance scores greater than the first pruning threshold are calibrated as True, and the importance scores less than the first pruning threshold are calibrated as False. After calibrating all the importance scores corresponding to the attention heads in the second metric matrix, the first calibration matrix is obtained.

[0173] Furthermore, according to the group sparsity of each target group in the first calibration matrix, the corresponding first mask position can be determined for each target group as a unit to obtain the first mask matrix. Exemplarily, in the first calibration matrix, the sum of each row of the first calibration matrix (each row corresponds to the calibration values of multiple importance scores included in a target group) can be calculated. When the sum obtained by a target group is greater than a preset threshold, it indicates that the element distribution of this target group is relatively dense and non-sparse, and no pruning should be performed. In the first calibration matrix, all values of this target group can be calibrated as True or 1; when the sum obtained by a target group is less than the preset threshold, it indicates that the element distribution of this target group is relatively sparse and can be pruned. In the first calibration matrix, all values of this target group can be calibrated as False or 0. Thus, the corresponding first mask position can be determined according to the positions of True (or 1) and False (or 0). After calibrating all the values of the target groups, the first mask matrix can be obtained.

[0174] By obtaining the first masking matrix, the specific pruning positions can be determined, thereby reducing the model parameters and computational complexity while maintaining the key performance of the model, achieving the optimization and acceleration of the model, and facilitating the deployment of the model in resource-constrained environments.

[0175] In some embodiments, in order to quickly and effectively evaluate the importance of each attention head in a large language model, a first pruning threshold can be determined according to the pruning rate, and a first calibration matrix corresponding to the second metric matrix can be determined according to the first pruning threshold, so as to quickly make a pruning decision based on the importance scores. For example, "determining the first calibration matrix of the second metric matrix according to the pruning rate and the magnitude relationship of multiple importance scores in the second metric matrix" in step 103 may include:

[0176] (103.a1) Determine a first number of attention heads to be pruned according to the pruning rate;

[0177] (103.a2) Determine the first pruning threshold from the second metric matrix according to the first number and the magnitude relationship of multiple importance scores;

[0178] (103.a3) Compare the first pruning threshold with each importance score in the second metric matrix respectively, and determine the calibration value corresponding to each importance score according to the magnitude relationship between the first pruning threshold and each importance score;

[0179] (103.a4) Determine the first calibration matrix of the second metric matrix based on the multiple calibration values corresponding to the multiple importance scores.

[0180] Among them, the first number can be the number of attention heads to be pruned determined based on the pruning rate.

[0181] Among them, the first pruning threshold can be a value determined according to the first number and the importance scores in the second metric matrix. Importance scores greater than the first pruning threshold are more important, and conversely, importance scores less than the first pruning threshold are less important.

[0182] Among them, the calibration value can be the result obtained by comparing each importance score with the first pruning threshold. The calibration value can be a binary value (True / False or 1 / 0). True (or 1) indicates that the attention head should be retained, and False (or 0) indicates that the attention head should be pruned.

[0183] Exemplarily, the first number of attention heads to be pruned can be calculated according to the pruning rate pruning_ratio. Specifically, the first number can be the product of the pruning rate and the total number head_num of all attention heads corresponding to the second metric matrix, that is, pruning_ratio head_num. After that, all the values in the second metric matrix can be sorted from smallest to largest to obtain the sorted score array scores_sorted. According to the first quantity as the index, the value divided by the first quantity is taken from the score array scores_sorted as the first pruning threshold, that is:

[0184] threshold = score_sorted[pruning_ratio head_num];

[0185] Furthermore, after determining the first pruning threshold, the values greater than the first pruning threshold in the second metric matrix can be set to True, and the values less than or equal to the first pruning threshold can be set to False, so as to determine the calibration value corresponding to each importance score in the second metric matrix. After that, after all the importance scores in the second metric matrix are calibrated, based on multiple calibration values, the first calibration matrix of the second metric matrix can be determined. In the first calibration matrix, all values are calibrated to True or False.

[0186] Through the above method, it can be ensured that the pruning decision is made based on the relative importance of all attention heads, rather than randomly or arbitrarily, making the subsequent pruning process more reasonable and accurate, and effectively avoiding mispruning.

[0187] Please refer to Figure 3 , in some embodiments, in order to effectively evaluate and quantify the importance of each target group in the large language model, so as to quickly evaluate the sparse distribution characteristics of the target group in units of the target group and achieve fast pruning, the corresponding first mask position can be determined based on multiple target calibration values corresponding to multiple target groups to obtain the first mask matrix, so as to improve the pruning efficiency and ensure that the large language model can maintain its performance as much as possible while reducing parameters. For example, "determining the corresponding first mask position based on the sparse distribution characteristics of the first calibration matrix to obtain the first mask matrix" in step 103 may include:

[0188] (103.b1) Obtain multiple calibration values corresponding to multiple attention heads included in each target group in the first calibration matrix;

[0189] (103.b2) Determine the target group value corresponding to the target group based on the multiple calibration values corresponding to each target group;

[0190] (103.b3) Obtain a preset mask threshold, and compare the target group value with the mask threshold to obtain a comparison result; among them, the comparison result of each target group in the first calibration matrix is associated with the sparse distribution characteristics of the corresponding target group;

[0191] (103.b4) Based on the comparison result, adjust the multiple calibration values corresponding to the multiple attention heads included in the target group in the first calibration matrix to obtain multiple target calibration values corresponding to the multiple attention heads;

[0192] (103.b5) Determine the corresponding first mask position based on the multiple target calibration values corresponding to the multiple target groups to obtain a first mask matrix.

[0193] Among them, the target group value can be the statistical or summary result of the calibration values corresponding to all attention heads in each target group. The target group value can be the average value, sum or result of other aggregation functions of all calibration values in the target group.

[0194] Among them, the mask threshold can be a preset fixed value, which is used to compare with the target group value to determine whether the calibration value of this group needs to be adjusted.

[0195] Among them, the comparison result can be the result obtained by comparing the target group value with the mask threshold. The comparison result can be a boolean value (True / False) to indicate whether the target group value is greater than or equal to the mask threshold.

[0196] Among them, the target calibration value can be the calibrated value adjusted based on the comparison result. If the comparison result is True, all the target calibration values corresponding to all the calibration values in the target group are adjusted to True; if the comparison result is False, all the target calibration values corresponding to all the calibration values in the target group are adjusted to False.

[0197] Among them, the first mask position can be the position marked in the first calibration matrix that needs to be retained (True) or pruned (False).

[0198] Exemplarily, in the first calibration matrix, based on the multiple calibration values of each target group, calculate the target group value corresponding to each target group. Exemplarily, if the calibration values corresponding to target group 1 are (False, True, True, True, True, False, True, True), if False corresponds to 0 and True corresponds to 1, add up the numerical values corresponding to all the calibration values, and the target group value corresponding to each target group can be obtained. The target group value of the above example is 6.

[0199] Furthermore, the mask threshold can be set according to the actual situation to reasonably determine the sparse distribution characteristics corresponding to the target group based on the size of the target group value. When the target group value is greater than the mask threshold, it indicates that the sparse distribution characteristic of the target group corresponding to the target group value is non-sparse. For example, if the mask threshold is 0.25 and the target group value is 6, the comparison result indicates that the target group value is greater than the mask threshold, and the sparse distribution characteristic of this target group is sparse; conversely, when the target group value is less than the mask threshold, it indicates that the sparse distribution characteristic of the target group corresponding to the target group value is sparse. For example, if the mask threshold is 0.25 and the target group value is 0, the comparison result indicates that the target group value is less than the mask threshold, and the sparse distribution characteristic of this target group is sparse.

[0200] Furthermore, when the target group value is greater than the mask threshold, the target calibration value of the target group value can be determined as True (or 1) in the first calibration matrix, and all calibration values corresponding to the target group are modified to the target calibration value, that is, the calibration values corresponding to each attention head are modified; conversely, when the target group value is greater than the mask threshold, the target calibration value of the target group value can be determined as False (or 0) in the first calibration matrix, and all calibration values corresponding to the target group are modified to the target calibration value.

[0201] Furthermore, after calculating the target group values corresponding to all target groups and modifying (or keeping unchanged, for example, if the original calibration value is False and the target calibration value is also False, then no modification is required) the calibration values corresponding to all target groups in the first calibration matrix according to the target calibration values, the corresponding first mask positions (that is, the False positions to be pruned or the True positions not to be pruned) can be determined based on the multiple target calibration values corresponding to the multiple target groups, and the corresponding first mask matrix can be generated.

[0202] In some embodiments, in addition to setting the mask threshold, a basic mask threshold can be further set. The basic mask threshold is less than the mask threshold, so that at least some attention heads are retained in extreme cases to avoid the situation where all elements are considered unimportant and pruned. For example, if the mask threshold is 1, then the basic mask threshold can be set to 0.25, and the specific values of the mask threshold and the basic mask threshold can be set according to the actual situation.

[0203] Exemplarily, if the mask threshold is 0.25, during the calculation, the sum of all elements in the first calibration matrix is 0, that is, all target grouping values are 0. This means that according to the current mask threshold, none of the attention heads or target groupings are considered important, that is, all attention heads are below the mask threshold, which will lead to excessive pruning and may damage the model performance because important features or weights are wrongly removed. To avoid this situation, a base mask threshold is introduced so that in the case where no target grouping meets the condition of being greater than the mask threshold, the base mask threshold is used to determine that at least a part of the attention heads should be retained, thereby avoiding a significant decline in the performance of the large language model. For example, if all target grouping values are 0 (or other values less than the mask threshold), then each target grouping value is compared with the base mask threshold again, and the target calibration value corresponding to the target grouping value greater than the base mask threshold is determined to be True (or 1), and vice versa is set to False (or 0). Exemplarily, the mask threshold can be set to 0.25 and the base mask threshold can be set to 0.

[0204] Through the above method, the robustness of the pruning process can be ensured, preventing the loss of model performance caused by overly strict pruning thresholds. By adjusting the mask threshold and the base mask threshold, the degree of pruning can be flexibly controlled to achieve a balance between model compression and performance retention.

[0205] Step 104, prune the key channels and value channels associated with the attention heads indicated by the first mask matrix.

[0206] In some embodiments, in order to effectively reduce redundant parameters in the large language model and optimize the model structure, the key channels and value channels associated with the attention heads indicated by the first mask matrix can be pruned to reduce the computational cost while trying to maintain the performance of the model.

[0207] Among them, the key channels can be the output channels in the key projection layer. Each attention head has a corresponding key projection layer (k_proj), and the key projection layer can map the input features to the key space.

[0208] Among them, the value channels can be the output channels in the value projection layer. Each attention head has a corresponding value projection layer (v_proj), and the value projection layer can map the input features to the value space.

[0209] Exemplarily, when an attention head is in the first mask matrix and the corresponding target calibration value is True (or 1), it indicates that the attention head is important and there is no need to prune the key channels and value channels associated with the attention head at the current network layer; conversely, when an attention head is in the first mask matrix and the corresponding target calibration value is False (or 0), it indicates that the attention head is not important and the key channels and value channels associated with the attention head at the current network layer can be pruned.

[0210] In some embodiments, since the output feature dimensions of the query, key, and value vectors of each attention head are different. For example, the output feature dimensions of the query projection layer and the output projection layer are 8192, and the output dimensions of the key projection layer and the value projection layer are 1024. Therefore, in this application, the first mask matrix is used to prune the key channels and value channels associated with the attention head, and the second mask matrix is used to prune the query channels and output channels associated with the attention head, so as to reduce the model size and computational complexity while maintaining the overall performance and structural integrity of the model and improving the accuracy of the pruning process. At the same time, since this application uses the group pruning method for pruning, the efficiency and accuracy of pruning can be greatly improved.

[0211] Step 105, determine the target number of attention heads indicated by the first mask matrix, determine the second calibration matrix of the second metric matrix according to the size relationship between the target number and multiple importance scores in the second metric matrix, and based on the sparse distribution characteristics of the second calibration matrix, determine the corresponding second mask positions to obtain the second mask matrix.

[0212] In some embodiments, in order to maintain the matching performance of the multi-head attention module (also known as the multi-head attention mechanism) and avoid calculation errors, the second mask matrix can be calculated based on the target number of pruned attention heads indicated by the first mask matrix to prune the query channels and output channels associated with the attention head, so as to improve the accuracy of the pruning decision, simplify the subsequent pruning operation process, and ensure that the large language model can maintain its performance as much as possible while reducing parameters.

[0213] Among them, the target number can be the number of attention heads marked as False or 0 (that is, indicating pruning) in the first mask matrix.

[0214] Among them, the second calibration matrix can be a threshold determined based on the target number after sorting the importance scores in the second metric matrix from small to large, and after comparing each importance score with the threshold, determining the calibrated value corresponding to each importance score according to the comparison result, and generating a matrix based on multiple calibrated values, which is used to intuitively indicate whether the attention head corresponding to each importance score is important.

[0215] Among them, the second mask position can be the position marked in the second calibration matrix that needs to be retained (True) or pruned (False).

[0216] Among them, the second mask matrix can be a binary matrix (True / False or 1 / 0) generated based on the second mask position, which is used for actual pruning operations. True indicates retaining the attention head at this position, and False indicates pruning.

[0217] Exemplarily, in order to more precisely control the pruning process and perform adaptive pruning, the target number of attention heads indicated by the first mask matrix can be determined, and the corresponding threshold can be updated again according to the target number to avoid over-pruning. For example, first, the number of groups of target groups to be pruned, num_pruned_groups, can be calculated. The calculation formula is as follows:

[0218] num_pruned_groups = num_key_value_group - sum(W_metric_kv);

[0219] Among them, num_key_value_group is a predefined parameter indicating the number of groups into which all attention heads are divided; sum(W_metric_kv) indicates the number of groups retained after pruning according to the indication of the first mask matrix, that is, the number of groups not pruned;

[0220] Furthermore, the target number of attention heads to be pruned, pruned_num_heads, can be calculated. The calculation formula is as follows (where " / / " represents integer division):

[0221] group_size = head_num / / num_key_value_group;

[0222] pruned_num_heads = group_size * num_pruned_groups;

[0223] Among them, head_num is the total number of attention heads in the current network layer; group_size is the number of attention heads included in each target group.

[0224] Furthermore, the second metric matrix can be sorted from smallest to largest to obtain the sorted score array sorted_scores2. Using the target number as the index, the value obtained after dividing the score array sorted_scores2 by the target number is used as the second pruning threshold, that is:

[0225] threshold = sorted_scores2[pruned_num_heads];

[0226] Further, after determining the second pruning threshold, values greater than the second pruning threshold in the second metric matrix can be set to True, and values less than or equal to the second pruning threshold can be set to False, so as to determine the second calibration value corresponding to each importance score in the second metric matrix. After that, after all the importance scores in the second metric matrix are calibrated, based on multiple second calibration values, a second calibration matrix of the second metric matrix can be determined. In the second calibration matrix, all values are calibrated to True or False.

[0227] In some embodiments, for the sparse distribution characteristics of the second calibration matrix, determining the corresponding mask positions to obtain the second mask matrix is the same as the process of determining the corresponding first mask positions based on the sparse distribution characteristics of the first calibration matrix described above to obtain the first mask matrix. The embodiments of the present application do not elaborate on the process of obtaining the second mask matrix, and specifically, reference can be made to the aforementioned process of obtaining the first mask matrix.

[0228] By determining the target number of attention heads indicated by the first mask matrix for pruning, the pruning threshold can be dynamically adjusted to maximize the retention of model performance, ensure that the pruned model structure remains symmetric and complete, thereby achieving efficient model compression and acceleration.

[0229] Step 106, prune the query channels and output channels associated with the attention heads indicated by the second mask matrix to obtain the large language model.

[0230] In some embodiments, to ensure the symmetry and balance of the model, it is also necessary to prune the query channels and output channels to avoid potential performance losses and achieve more balanced, effective, and structured model pruning.

[0231] Among them, the query channel can be the output channel of the query projection layer. In the multi-head attention module, each attention head has a corresponding query projection layer, and the query projection layer can map the input features to the query space.

[0232] Among them, the output channel can be the output channel corresponding to the linear transformation layer (output projection layer), that is, the output channel after linear transformation.

[0233] Exemplarily, when the attention head is in the second mask matrix and the corresponding calibration value is True (or 1), it indicates that the attention head is important and there is no need to prune the query channel and output channel associated with the attention head at the current network layer; conversely, when the attention head is in the second mask matrix and the corresponding calibration value is False (or 0), it indicates that the attention head is not important and the query channel and output channel associated with the attention head at the current network layer can be pruned.

[0234] In some embodiments, since the output feature dimensions of the query, key, and value vectors of each attention head are different. For example, the output feature dimensions of the query projection layer and the output projection layer are 8192, and the output dimensions of the key projection layer and the value projection layer are 1024. Therefore, in this application, the second mask matrix is used to prune the key channel and value channel associated with the attention head, and the second mask matrix is used to prune the query channel and output channel associated with the attention head, so as to reduce the model size and computational complexity while maintaining the overall performance and structural integrity of the model and improving the accuracy of the pruning process. At the same time, since this application uses the group pruning method for pruning, the efficiency and accuracy of pruning can be greatly improved.

[0235] Further, the compress function can be called to prune the output channels of the query projection layer, output projection layer, key projection layer, and value projection layer, thereby actually pruning the parameters and effectively reducing the number of parameters of the large language model.

[0236] Further, for the multi-layer perception mechanism (also known as the feed-forward neural network module) included in each network layer, the pruning threshold of the weight matrix corresponding to the multi-layer perception mechanism can also be determined according to the first metric matrix and the pruning rate, so as to obtain the third mask matrix. And according to the third mask matrix, the pruning function (compress function) is called to prune the weight matrix with small weights in the up_proj layer (upper projection layer), or down_proj layer (lower projection layer), or gate_proj layer (gated projection layer).

[0237] Further, after all network layers are pruned, the large language model can be obtained.

[0238] It should be noted that this application can achieve structured pruning for each network layer. Compared with unstructured pruning, this application is more hardware-friendly. Although unstructured pruning can reduce the computational amount of the model, since the sparse matrix after its pruning is difficult to efficiently process on general-purpose hardware, specific software and hardware support is usually required to achieve the acceleration effect. However, the structured pruning method adopted in this application retains the regular structure of the matrix by pruning the corresponding channels, enabling the pruned model to run efficiently on general-purpose hardware without additional hardware support. In addition, this application belongs to a post-training pruning algorithm, which can maintain the performance of the model without additional fine-tuning training, while significantly reducing the storage space and computational overhead of the model.

[0239] In the embodiments of the present application, the pruning rate of each network layer in a preset large language model is determined by obtaining a first metric matrix of each network layer and based on the first metric matrix; for multiple attention heads included in each network layer, multiple importance scores are determined according to the corresponding first metric matrix, and a second metric matrix is constructed according to the multiple importance scores; a first calibration matrix of the second metric matrix is determined according to the size relationship between the pruning rate and the multiple importance scores in the second metric matrix, and based on the sparse distribution characteristics of the first calibration matrix, the corresponding first mask position is determined to obtain a first mask matrix; pruning is performed on the key channels and value channels associated with the attention heads indicated by the first mask matrix; the target number of the attention heads indicated by the first mask matrix is determined, a second calibration matrix of the second metric matrix is determined according to the size relationship between the target number and the multiple importance scores in the second metric matrix, and based on the sparse distribution characteristics of the second calibration matrix, the corresponding second mask position is determined to obtain a second mask matrix; pruning is performed on the query channels and output channels associated with the attention heads indicated by the second mask matrix to obtain a large language model. In this way, the pruning rate of each network layer of the large language model can be first determined through the first metric matrix of each network layer, so as to adaptively adjust the pruning granularity according to different layers of the model and avoid the decline of model performance caused by excessive pruning. Moreover, by determining the contribution degree of each attention head to the model output through the importance scores of each attention head, it can be ensured that the pruning operation will not significantly affect the performance of the model. Further, the key channels and value channels of the attention heads that need to be pruned are determined through the first mask matrix, effectively reducing the number of parameters of the model. After pruning the key channels and value channels, the query channels and output channels of the corresponding attention heads are further pruned according to the second mask matrix. In this way, by pruning based on the structural characteristics of the output channels of the key projection layer, value projection layer, query projection layer and output projection layer of the model, the pruning process of different parts can be controlled more finely, ensuring that important parts are retained, so that the pruned large language model does not require additional fine-tuning training and can still maintain the original performance, thereby accelerating the computational efficiency of the model. In summary, the present application can reduce the storage space of the large language model and accelerate the computational efficiency of the large language model.

[0240] Please refer to Figure 4 , Figure 4 which is the overall flowchart of the parameter pruning method for the large language model. The following will be combined with Figure 4, a general introduction to the technical solution of this application is provided. Specifically, after initiating the parameter pruning process for the large language model, a preset large language model (such as the LLaMA-3 model) and a calibration dataset (such as the C4 dataset) can be loaded to provide input data for subsequent calculation of the metric criteria. Then, the loop variable i = 0 can be initialized, and each network layer of the large language model can be traversed, and for each network layer, the pruning rate can be calculated layer by layer. Specifically, when the current layer number i is less than the total number of network layers of the large language model, the operation of calculating the pruning rate can continue. When calculating the pruning rate of the current network layer, based on the first metric matrix of this network layer (such as can be calculated through the weight matrix and the input data norm), combined with a preset threshold constant and statistical features (such as the singular value ratio), the pruning rate of the current network layer can be dynamically determined. After saving the pruning rate of each network layer into the corresponding pruning rate linked list (all_layer_ratio), the next layer (i + 1) network layer can be processed until all network layers are traversed.

[0241] Furthermore, after calculating the pruning rates of all network layers, the pruning rate linked list of all layers can be normalized by normal distribution to ensure a reasonable distribution of pruning rates for each layer and avoid performance loss caused by extreme pruning. Further, after performing the normal distribution normalization process on all network layers, i = 0 can be initialized again to start the second round of loop, and structured pruning can be performed layer by layer. During the process of performing structured pruning on each network layer one by one, if the current layer number i is less than the total number of network layers of the model, the structured pruning operation can continue.

[0242] Specifically, during the process of performing structured pruning on the current network layer, for multiple attention heads of the current network layer, the importance score of each attention head can be calculated based on the first metric matrix, a second metric matrix can be constructed, and it can be determined whether the current layer is the output projection layer of the attention module. If so, according to the pruning rate and the importance score of the second metric matrix, the pruning threshold can be determined through sorting to generate the first calibration matrix, and based on the group sparsity of the calibration matrix, the first mask matrix (marking the key channels and value channels to be pruned) and then the second mask matrix (marking the query channels and output channels to be pruned) can be determined.

[0243] Furthermore, the first mask matrix can be used to call the pruning function (compress function) to prune the output channels corresponding to the key projection layer (k_proj) and value projection layer (v_proj) of the attention module, and the second mask matrix can be used to prune the output channels corresponding to the query projection layer (q_proj) and output projection layer (o_proj).

[0244] In some embodiments, if the current layer is not the output projection layer of the attention module (i.e., the current network layer is an MLP module), a threshold is calculated using the first metric matrix and the pruning rate, and a third mask matrix is calculated. Based on the third mask matrix, a pruning function is called to prune the weight matrices corresponding to the up_proj, down_proj, etc. of the MLP module.

[0245] Further, after pruning the parameters of the attention module and the MLP module of the current network layer, the next network layer (i + 1) can be processed until all network layers are traversed. After completing the pruning operation for all network layers, the pruned large language model is saved, and the processing flow ends.

[0246] The above is only an embodiment of the parameter pruning method for the large language model proposed in this application. Without departing from the concept of this application, the specific operation process can be increased or decreased.

[0247] Please refer to Figure 5 , this application embodiment also provides a parameter pruning device for a large language model, which can implement the above parameter pruning method for the large language model. The parameter pruning device for the large language model includes:

[0248] An acquisition module 51, configured to acquire the first metric matrix of each network layer in a plurality of network layers of a preset large language model, and determine the pruning rate of the corresponding network layer based on the first metric matrix;

[0249] A construction module 52, configured to, for a plurality of attention heads included in each network layer, determine a plurality of importance scores according to the corresponding first metric matrix, and construct a second metric matrix according to the plurality of importance scores;

[0250] A first determination module 53, configured to determine a first calibration matrix of the second metric matrix according to the size relationship between the pruning rate and the plurality of importance scores in the second metric matrix, and determine the corresponding first mask position based on the sparse distribution characteristic of the first calibration matrix to obtain a first mask matrix;

[0251] A first pruning module 54, configured to prune the key channels and value channels associated with the attention heads indicated by the first mask matrix;

[0252] A second determination module 55, configured to determine the target number of the attention heads indicated by the first mask matrix, determine a second calibration matrix of the second metric matrix according to the size relationship between the target number and the plurality of importance scores in the second metric matrix, and determine the corresponding second mask position based on the sparse distribution characteristic of the second calibration matrix to obtain a second mask matrix;

[0253] The second pruning module 56 is used to prune the query channels and output channels associated with the attention heads indicated by the second mask matrix, so as to obtain a large language model.

[0254] The specific implementation manner of the parameter pruning device of the large language model is basically the same as the specific embodiments of the above-mentioned parameter pruning method of the large language model, and will not be elaborated here. On the premise of meeting the requirements of the embodiments of the present application, the parameter pruning device of the large language model can also set other functional modules to implement the parameter pruning method of the large language model in the above embodiments.

[0255] The embodiments of the present application also provide a computer device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the above-mentioned parameter pruning method of the large language model. The computer device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0256] Please refer to Figure 6 , Figure 6 which schematically shows the hardware structure of a computer device in another embodiment. The computer device includes:

[0257] A processor 61, which can be implemented by using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0258] A memory 62, which can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 62 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 62 and are called by the processor 61 to execute the parameter pruning method of the large language model in the embodiments of the present application;

[0259] An input / output interface 63, which is used to implement information input and output;

[0260] A communication interface 64, which is used to implement communication interaction between this device and other devices, and can implement communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0261] The bus 65 transmits information between various components of the device (such as the processor 61, the memory 62, the input / output interface 63, and the communication interface 64);

[0262] Among them, the processor 61, the memory 62, the input / output interface 63, and the communication interface 64 are communicatively connected to each other inside the device through the bus 65.

[0263] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned parameter pruning method of the large language model.

[0264] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0265] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0266] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0267] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0268] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0269] In the description of the present application and the above-mentioned drawings, the terms "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0270] It should be understood that in the present application, "at least one (item)" and "several" mean one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally means that the associated objects before and after are in an "or" relationship. "At least one (piece) of the following" or its similar expression refers to any combination of these items, including any combination of single item (piece) or plural items (pieces). For example, at least one (piece) of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0271] In several embodiments provided by the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the above-mentioned division of units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0272] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0273] In addition, in each embodiment of the present application, each functional unit can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0274] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present application. The foregoing storage medium includes: various media that can store programs such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0275] The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings, and thus do not limit the scope of the rights of the embodiments of the present application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A parameter pruning method for a large language model, characterized in that: Applied to natural language processing tasks, the method comprises: Obtaining a first metric matrix of each network layer in a plurality of network layers of a preset large language model, and determining a pruning rate corresponding to the network layer based on the first metric matrix; For the multiple attention heads included in each network layer, determine a plurality of importance scores according to the corresponding first metric matrix, and construct a second metric matrix according to the plurality of importance scores; Determining a first calibration matrix of the second metric matrix according to a size relationship between the pruning rate and a plurality of importance scores in the second metric matrix, and determining a corresponding first mask position based on a sparse distribution characteristic of the first calibration matrix to obtain a first mask matrix; Pruning the key channel and the value channel associated with the attention head indicated by the first mask matrix; Determine a target number of attention heads indicated by the first mask matrix, determine a second calibration matrix of the second metric matrix according to a size relationship between the target number and a plurality of importance scores in the second metric matrix, and determine a corresponding second mask position based on a sparse distribution characteristic of the second calibration matrix to obtain a second mask matrix; The query channel and the output channel associated with the attention head indicated by the second mask matrix are pruned to obtain a large language model.

2. The parameter pruning method for a large language model according to claim 1, characterized in that: The step of determining a plurality of importance scores for the plurality of attention heads included in each network layer according to the corresponding first metric matrix, and constructing a second metric matrix according to the plurality of importance scores comprises: For each of the attention heads included in each network layer, averaging the matrix columns corresponding to each linear layer of the first metric matrix to obtain a first average metric matrix; Determine a correspondence between a plurality of matrix features in the first average metric matrix and the plurality of attention heads, and adjust the first average metric matrix based on the correspondence to obtain a second average metric matrix; Based on the second average metric matrix, an importance score corresponding to each attention head is determined, and based on the multiple importance scores corresponding to the multiple attention heads, a second metric matrix is ​​constructed.

3. The parameter pruning method for a large language model according to claim 2, characterized in that: The constructing a second metric matrix based on the multiple importance scores corresponding to the multiple attention heads includes: Get the target group corresponding to each attention head; Based on the grouping relationship between the multiple attention heads, the position of the importance score of each attention head corresponding to the second average metric matrix is ​​adjusted to obtain a second metric matrix arranged according to the target grouping.

4. The parameter pruning method for a large language model according to claim 1, characterized in that: The determining a first calibration matrix of the second metric matrix according to the size relationship between the pruning rate and a plurality of importance scores in the second metric matrix comprises: Determining a first number of attention heads to be pruned according to the pruning rate; Determining a first pruning threshold from the second metric matrix according to the first quantity and the magnitude relationship of the plurality of importance scores; Comparing the first pruning threshold with each importance score in the second metric matrix respectively, and determining a calibration value corresponding to each importance score according to a magnitude relationship between the first pruning threshold and each importance score; A first calibration matrix of the second metric matrix is ​​determined based on a plurality of calibration values ​​corresponding to the plurality of importance scores.

5. The parameter pruning method for a large language model according to claim 1, characterized in that: The determining of the corresponding first mask position based on the sparse distribution characteristic of the first calibration matrix to obtain the first mask matrix includes: Obtaining multiple calibration values ​​corresponding to multiple attention heads included in each target group in the first calibration matrix; Determining a target group value corresponding to the target group based on the multiple calibration values ​​corresponding to each target group; Obtaining a preset mask threshold, and comparing the target group value with the mask threshold to obtain a comparison result; wherein the comparison result of each target group of the first calibration matrix is ​​associated with a sparse distribution characteristic of the corresponding target group; Based on the comparison result, adjusting a plurality of calibration values ​​corresponding to the plurality of attention heads included in the target group in the first calibration matrix to obtain a plurality of target calibration values ​​corresponding to the plurality of attention heads; Based on the multiple target calibration values ​​corresponding to the multiple target groups, corresponding first mask positions are determined to obtain a first mask matrix.

6. The parameter pruning method for a large language model according to claim 1, characterized in that: The step of obtaining a first metric matrix of each network layer in a plurality of network layers of a preset large language model includes: Obtain multiple linear layers corresponding to each network layer included in the large language model, and determine multiple weight matrices corresponding to each linear layer in each network layer; Determining the element position of each matrix element in each weight matrix; Obtain input data of the linear layer corresponding to each weight matrix, and calculate multiple norm values ​​corresponding to the input data in multiple input dimensions; Based on the element position of each matrix element, determining a target norm value of a corresponding input dimension from the plurality of norm values; Determining a measurement criterion for each matrix element according to each matrix element and the target norm value; Generate a layer metric criterion corresponding to each linear layer based on multiple metric criteria of multiple matrix elements corresponding to the multiple weight matrices; Based on the multiple layer measurement criteria corresponding to the multiple linear layers, a first measurement matrix corresponding to each network layer is obtained.

7. The parameter pruning method for a large language model according to claim 1, characterized in that: The determining, based on the first metric matrix, a pruning rate corresponding to the network layer includes: Calculating a metric average for the first metric matrix; For each linear layer in each network layer, a preset threshold constant is obtained, and a target threshold is obtained according to the product of the metric average and the threshold constant; Determining, from among the plurality of metric elements of the first metric matrix, a second number of a plurality of target metric elements greater than the target threshold; Obtaining a third number of the plurality of metric elements of the first metric matrix, and obtaining a singular value ratio based on a ratio of the second number to the third number; Performing standardization on the singular value ratio to obtain a target singular value ratio; Based on the target singular value ratios corresponding to the multiple linear layers corresponding to each network layer, the pruning rate of each network layer is determined.

8. A parameter pruning device for a large language model, characterized in that: Applied to natural language processing tasks, the device comprises: An acquisition module, configured to acquire a first metric matrix of each network layer in a plurality of network layers of a preset large language model, and determine a pruning rate corresponding to the network layer based on the first metric matrix; A construction module, configured to determine a plurality of importance scores for a plurality of attention heads included in each network layer according to the corresponding first metric matrix, and construct a second metric matrix according to the plurality of importance scores; A first determination module is used to determine a first calibration matrix of the second metric matrix according to the pruning rate and the size relationship of multiple importance scores in the second metric matrix, and determine the corresponding first mask position based on the sparse distribution characteristics of the first calibration matrix to obtain a first mask matrix; A first pruning module, configured to prune a key channel and a value channel associated with the attention head indicated by the first mask matrix; A second determination module is used to determine the target number of attention heads indicated by the first mask matrix, determine a second calibration matrix of the second metric matrix according to the size relationship between the target number and multiple importance scores in the second metric matrix, and determine the corresponding second mask position based on the sparse distribution characteristics of the second calibration matrix to obtain a second mask matrix; The second pruning module is used to prune the query channel and the output channel associated with the attention head indicated by the second mask matrix to obtain a large language model.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the parameter pruning method for a large language model according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the parameter pruning method for a large language model according to any one of claims 1 to 7 is implemented.

Images