Parameter pruning method, device and equipment for large language model and readable storage medium

Abstract

The embodiment of the invention provides a parameter pruning method and device for a large language model, equipment and a readable storage medium. The method comprises the following steps: acquiring a first measurement matrix of each network layer in a preset large language model, and determining a pruning rate of the corresponding network layer; determining a plurality of importance scores for a plurality of attention heads contained in each network layer, and constructing a second measurement matrix; determining a first calibration matrix of the second measurement matrix according to the pruning rate and the second measurement matrix, and obtaining a first mask matrix based on the sparse distribution characteristic of the first calibration matrix; pruning a key channel and a value channel; determining a second calibration matrix of the second measurement matrix according to the target number and the second measurement matrix, and obtaining a second mask matrix based on the sparse distribution characteristic of the second calibration matrix; and pruning a query channel and an output channel associated with the second mask matrix to obtain a large language model. Therefore, the storage space of the large language model can be reduced, and the calculation efficiency of the large language model is improved.

Description

Technical Field

[0001] The present application relates to the technical field of network model compression, and in particular to a parameter pruning method, device, equipment and readable storage medium for a large language model. Background Art

[0002] Large Language Model (LLM) is an artificial intelligence technology based on deep learning and one of the core research contents of natural language processing. By training on massive text data, large language models can understand and generate natural language texts and perform a variety of natural language processing tasks including text generation, translation, question answering, etc.

[0003] However, since the number of parameters in large language models is huge, often starting at billions, the requirements for computing and storage overhead are very high. Therefore, it is necessary to compress the weight space of large language models to reduce the complexity of model calculations and improve calculation efficiency.

[0004] In the related art, some elements in the weights of a large language model can be set to zero through unstructured pruning to reduce the computational complexity of the model. However, the large language model after unstructured pruning still needs to save the complete weight matrix, which means that the total storage requirement has not been substantially reduced; and the model still needs to perform the same calculation steps as the original model. Therefore, unstructured pruning not only cannot compress the storage space of the large language model, but also cannot speed up the computational efficiency of the large language model. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a parameter pruning method, device, equipment and readable storage medium for a large language model, which can reduce the storage space of the large language model and accelerate the computing efficiency of the large language model.

[0006] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a parameter pruning method for a large language model, the method comprising: Obtaining a first metric matrix of each network layer in a plurality of network layers of a preset large language model, and determining a pruning rate corresponding to the network layer based on the first metric matrix; For the multiple attention heads included in each network layer, determine a plurality of importance scores according to the corresponding first metric matrix, and construct a second metric matrix according to the plurality of importance scores; Determining a first calibration matrix of the second metric matrix according to a size relationship between the pruning rate and a plurality of importance scores in the second metric matrix, and determining a corresponding first mask position based on a sparse distribution characteristic of the first calibration matrix to obtain a first mask matrix; Pruning the key channel and the value channel associated with the attention head indicated by the first mask matrix; Determine a target number of attention heads indicated by the first mask matrix, determine a second calibration matrix of the second metric matrix according to a size relationship between the target number and a plurality of importance scores in the second metric matrix, and determine a corresponding second mask position based on a sparse distribution characteristic of the second calibration matrix to obtain a second mask matrix; The query channel and the output channel associated with the attention head indicated by the second mask matrix are pruned to obtain a large language model.

[0007] Accordingly, a second aspect of an embodiment of the present application proposes a parameter pruning device for a large language model, the device comprising: An acquisition module, configured to acquire a first metric matrix of each network layer in a plurality of network layers of a preset large language model, and determine a pruning rate corresponding to the network layer based on the first metric matrix; A construction module, configured to determine a plurality of importance scores for a plurality of attention heads included in each network layer according to the corresponding first metric matrix, and construct a second metric matrix according to the plurality of importance scores; A first determination module is used to determine a first calibration matrix of the second metric matrix according to the pruning rate and the size relationship of multiple importance scores in the second metric matrix, and determine the corresponding first mask position based on the sparse distribution characteristics of the first calibration matrix to obtain a first mask matrix; A first pruning module, configured to prune a key channel and a value channel associated with the attention head indicated by the first mask matrix; A second determination module is used to determine the target number of attention heads indicated by the first mask matrix, determine a second calibration matrix of the second metric matrix according to the size relationship between the target number and multiple importance scores in the second metric matrix, and determine the corresponding second mask position based on the sparse distribution characteristics of the second calibration matrix to obtain a second mask matrix; The second pruning module is used to prune the query channel and the output channel associated with the attention head indicated by the second mask matrix to obtain a large language model.

[0008] In some embodiments, the building block is further used to: For each of the attention heads included in each network layer, averaging the matrix columns corresponding to each linear layer of the first metric matrix to obtain a first average metric matrix; Determine a correspondence between a plurality of matrix features in the first average metric matrix and the plurality of attention heads, and adjust the first average metric matrix based on the correspondence to obtain a second average metric matrix; Based on the second average metric matrix, an importance score corresponding to each attention head is determined, and based on the multiple importance scores corresponding to the multiple attention heads, a second metric matrix is ​​constructed.

[0009] In some embodiments, the building block is further used to: Get the target group corresponding to each attention head; Based on the grouping relationship between the multiple attention heads, the position of the importance score of each attention head corresponding to the second average metric matrix is ​​adjusted to obtain a second metric matrix arranged according to the target grouping.

[0010] In some implementations, the first determining module is further configured to: Determining a first number of attention heads to be pruned according to the pruning rate; Determining a first pruning threshold from the second metric matrix according to the first quantity and the magnitude relationship of the plurality of importance scores; Comparing the first pruning threshold with each importance score in the second metric matrix respectively, and determining a calibration value corresponding to each importance score according to a magnitude relationship between the first pruning threshold and each importance score; A first calibration matrix of the second metric matrix is ​​determined based on a plurality of calibration values ​​corresponding to the plurality of importance scores.

[0011] In some implementations, the first determining module is further configured to: Obtaining multiple calibration values ​​corresponding to multiple attention heads included in each target group in the first calibration matrix; Determining a target group value corresponding to the target group based on the multiple calibration values ​​corresponding to each target group; Obtaining a preset mask threshold, and comparing the target group value with the mask threshold to obtain a comparison result; wherein the comparison result of each target group of the first calibration matrix is ​​associated with a sparse distribution characteristic of the corresponding target group; Based on the comparison result, adjusting a plurality of calibration values ​​corresponding to the plurality of attention heads included in the target group in the first calibration matrix to obtain a plurality of target calibration values ​​corresponding to the plurality of attention heads; Based on the multiple target calibration values ​​corresponding to the multiple target groups, corresponding first mask positions are determined to obtain a first mask matrix.

[0012] In some implementations, the acquisition module is further used to: Obtain multiple linear layers corresponding to each network layer included in the large language model, and determine multiple weight matrices corresponding to each linear layer in each network layer; Determining the element position of each matrix element in each weight matrix; Obtain input data of the linear layer corresponding to each weight matrix, and calculate multiple norm values ​​corresponding to the input data in multiple input dimensions; Based on the element position of each matrix element, determining a target norm value of a corresponding input dimension from the plurality of norm values; Determining a measurement criterion for each matrix element according to each matrix element and the target norm value; Generate a layer metric criterion corresponding to each linear layer based on multiple metric criteria of multiple matrix elements corresponding to the multiple weight matrices; Based on the multiple layer measurement criteria corresponding to the multiple linear layers, a first measurement matrix corresponding to each network layer is obtained.

[0013] In some implementations, the acquisition module is further used to: Calculating a metric average for the first metric matrix; For each linear layer in each network layer, a preset threshold constant is obtained, and a target threshold is obtained according to the product of the metric average and the threshold constant; Determining, from among the plurality of metric elements of the first metric matrix, a second number of a plurality of target metric elements greater than the target threshold; Obtaining a third number of the plurality of metric elements of the first metric matrix, and obtaining a singular value ratio based on a ratio of the second number to the third number; Performing standardization on the singular value ratio to obtain a target singular value ratio; Based on the target singular value ratios corresponding to the multiple linear layers corresponding to each network layer, the pruning rate of each network layer is determined.

[0014] Correspondingly, the third aspect of the embodiments of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the parameter pruning method of the large language model described in any one of the embodiments of the first aspect of the present application when executing the computer program.

[0015] Correspondingly, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the parameter pruning method of the large language model described in any one of the embodiments of the first aspect of the present application.

[0016] The embodiment of the present application obtains a first metric matrix of each network layer in a plurality of network layers of a preset large language model, and determines a pruning rate of the corresponding network layer based on the first metric matrix; for a plurality of attention heads contained in each network layer, determines a plurality of importance scores according to the corresponding first metric matrix, and constructs a second metric matrix according to the plurality of importance scores; determines a first calibration matrix of the second metric matrix according to a size relationship between the pruning rate and the plurality of importance scores in the second metric matrix, and determines a corresponding first mask position based on a sparse distribution characteristic of the first calibration matrix to obtain a first mask matrix; prunes key channels and value channels associated with the attention heads indicated by the first mask matrix; determines a target number of attention heads indicated by the first mask matrix, determines a second calibration matrix of the second metric matrix according to a size relationship between the target number and the plurality of importance scores in the second metric matrix, and determines a corresponding second mask position based on a sparse distribution characteristic of the second calibration matrix to obtain a second mask matrix; prunes query channels and output channels associated with the attention heads indicated by the second mask matrix to obtain a large language model. In this way, the pruning rate of each network layer can be determined by the first metric matrix of each network layer of the large language model, so as to adaptively adjust the granularity of pruning according to the different layers of the model, and avoid excessive pruning to cause the performance of the model to deteriorate. In addition, the contribution of each attention head to the model output is determined by the importance score of each attention head, which can ensure that the pruning operation will not significantly affect the performance of the model. Further, the key channel and value channel of the attention head to be pruned are determined by the first mask matrix, which effectively reduces the number of parameters of the model. After pruning the key channel and the value channel, the query channel and the output channel of the corresponding attention head are further pruned according to the second mask matrix. In this way, by pruning based on the structural characteristics of the output channels of the key projection layer and the value projection layer, the query projection layer and the output projection layer of the model, the pruning process of different parts can be controlled more finely, ensuring that the important parts are retained, so that the pruned large language model does not require additional fine-tuning training, and can still maintain the original performance, thereby accelerating the computational efficiency of the model. In summary, the present application can reduce the storage space of the large language model and accelerate the computational efficiency of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the architecture of a parameter pruning system for a large language model provided in an embodiment of the present application; Figure 2 It is a flow chart of a parameter pruning method for a large language model provided in an embodiment of the present application; Figure 3 is a flowchart of calculating a first mask matrix and a second mask matrix provided in an embodiment of the present application; Figure 4 It is an overall flow chart of the parameter pruning method of a large language model provided in an embodiment of the present application; Figure 5 It is a functional module diagram of a parameter pruning device for a large language model provided in an embodiment of the present application; Figure 6 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0021] Large Language Model (LLM) is an artificial intelligence technology based on deep learning and one of the core research contents of natural language processing. By training on massive text data, large language models can understand and generate natural language texts and perform a variety of natural language processing tasks including text generation, translation, question answering, etc.

[0022] However, since the number of parameters in large language models is huge, often starting at billions, the requirements for computing and storage overhead are very high. Therefore, it is necessary to compress the weight space of large language models to reduce the complexity of model calculations and improve calculation efficiency.

[0023] In the related art, some elements in the weights of a large language model can be set to zero through unstructured pruning to reduce the computational complexity of the model. However, the large language model after unstructured pruning still needs to save the complete weight matrix, which means that the total storage requirement has not been substantially reduced; and the model still needs to perform the same calculation steps as the original model. Therefore, unstructured pruning not only cannot compress the storage space of the large language model, but also cannot speed up the computational efficiency of the large language model.

[0024] Based on this, the embodiments of the present application provide a parameter pruning method, apparatus, device and readable storage medium for a large language model, which can reduce the storage space of the large language model and accelerate the computing efficiency of the large language model.

[0025] The parameter pruning method, apparatus, device and readable storage medium of a large language model provided in the embodiments of the present application are specifically described through the following embodiments. First, the parameter pruning system of a large language model in the embodiments of the present application is described.

[0026] Please refer to Figure 1 In some implementations, an embodiment of the present application provides a parameter pruning system for a large language model, including a terminal 11 and a server 12.

[0027] Exemplarily, the terminal 11 may be a smart phone, a personal computer, a workstation, etc. The terminal 11 may provide a graphical interface or a command line tool, allowing the user to load a large language model, select a correction data set, and set pruning parameters (such as pruning rate, pruning threshold, etc.). The terminal 11 may also load the model and preprocess the data, call the pruning algorithm, and display the pruning results to help the user evaluate the pruning effect, etc.

[0028] Furthermore, the server 12 can be a high-performance computing server, a cluster server, a dedicated AI server, etc. The server 12 can receive the model and parameters sent by the terminal 11, load the large language model and data set, execute the parameter pruning method, and allocate computing resources according to task requirements to ensure the efficient operation of the pruning task. After pruning is completed, the server 12 can store the pruned large language model and related results to a disk or a database, and return the results to the terminal 11 so that the terminal 11 can display the results.

[0029] The parameter pruning method of the large language model in the embodiment of the present application can be illustrated by the following embodiment.

[0030] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for enabling the normal operation of the embodiment of the present application will be obtained.

[0031] In the embodiment of the present application, the parameter pruning device of a large language model will be described from the dimension of the parameter pruning device of the large language model, and the parameter pruning device of the large language model can be specifically integrated in a computer device. Figure 2 , Figure 2 The step flow chart of the parameter pruning method of the large language model provided in the embodiment of the present application, the embodiment of the present application takes the parameter pruning device of the large language model as an example, and when the processor on the terminal or server executes the program instructions corresponding to the parameter pruning method of the large language model, the specific process is as follows: Step 101: obtain a first metric matrix of each network layer in a plurality of network layers of a preset large language model, and determine a pruning rate of the corresponding network layer based on the first metric matrix.

[0032] In some embodiments, in order to retain the effectiveness and accuracy of the model as much as possible while reducing the computing cost, the first metric matrix of each network layer in the multiple network layers of the preset large language model can be obtained, and the pruning rate of the corresponding network layer can be determined based on this, so as to effectively reduce the model size in the subsequent pruning process and improve the computing speed of the model.

[0033] Among them, the preset large language model can be an LLaMA-3 model, or other models based on the Transformer architecture. The core components of the preset large language model include a multi-head attention module and a feedforward neural network (Multilayer Perceptron, MLP) module, which are used to process complex natural language processing tasks.

[0034] Among them, the network layer can be the basic building block in a large language model, such as DecoderLayer. Each network layer can contain multiple submodules, such as multi-head self-attention modules and MLP modules. Among them, the multi-head attention module includes linear layers such as q_proj (query projection), k_proj (key projection), v_proj (value projection) and o_proj (output projection), and the MLP module can contain linear layers such as up_proj (upward projection), down_proj (downward projection) and gate_proj (gated projection).

[0035] Among them, the first metric matrix can be used to measure the importance of each part in the network layer. Specifically, for a network layer (nn.Linear layer), its first metric matrix is ​​calculated based on the weight matrix and input data of the layer, reflecting the importance of each part to the final output.

[0036] The pruning rate can be the proportion of parameters planned to be removed during the pruning process. For example, if the set pruning rate is 30%, it means that 30% of the parameters will be considered unimportant and will be removed from the preset large language model.

[0037] In some implementations, the weight matrix of each network layer may be obtained, and the L2 norm of each weight matrix may be calculated row by row, and the obtained L2 norm matrix is ​​the first metric matrix.

[0038] Furthermore, the pruning rate can be calculated based on the statistical characteristics of the first metric matrix (such as mean, standard deviation, etc.). Specifically, based on the calculated first metric matrix, the pruning rate of each network layer can be determined by a specific algorithm (such as calculating the singular value ratio and normalizing it) to determine how many proportions of parameters of each network layer will be removed in the subsequent pruning process.

[0039] Through the above method, the first metric matrix and pruning rate of each network layer can be accurately obtained, so as to more effectively reduce the number of model parameters and computational overhead in the subsequent model pruning and optimization process, while maintaining the performance of the model as much as possible.

[0040] In some embodiments, in order to systematically analyze and quantify the importance of each component in the preset large language model, the layer metric criteria of all linear layers in each network layer can be integrated to generate a first metric matrix corresponding to the network layer to lay the foundation for efficient and accurate pruning operations. For example, in step 101, "obtaining the first metric matrix of each network layer in multiple network layers of the preset large language model" may include: (101.a1) Obtain multiple linear layers corresponding to each network layer contained in the large language model, and in each network layer, determine multiple weight matrices corresponding to each linear layer; (101.a2) determining the element positions of each matrix element in each weight matrix; (101.a3) Get the input data of the linear layer corresponding to each weight matrix, and calculate multiple norm values ​​corresponding to the input data in multiple input dimensions; (101.a4) determining a target norm value for the corresponding input dimension from a plurality of norm values ​​based on the element position of each matrix element; (101.a5) Determine the measurement criteria for each matrix element based on the value of each matrix element and the target norm; (101.a6) generating a layer metric corresponding to each linear layer based on multiple metric criteria of multiple matrix elements corresponding to multiple weight matrices; (101.a7) Based on multiple layer measurement criteria corresponding to multiple linear layers, a first measurement matrix corresponding to each network layer is obtained.

[0041] Among them, the linear layer can be a basic unit for performing linear transformation in a neural network, which can transform the input data through a weight matrix and add a bias term.

[0042] Among them, the weight matrix can be a parameter matrix in the linear layer, which is used to map the input data from one space to another space, and its size is determined by the number of input features and the number of output features.

[0043] Here, a matrix element can be a single value in the weight matrix, representing the strength of the connection between a specific input feature and the output feature.

[0044] The element position may be the row and column coordinates of the matrix element in the weight matrix to which it belongs.

[0045] The input data may be data input to a linear layer, which may be a multi-dimensional tensor.

[0046] The input dimension can be a single feature dimension in the input data, representing a characteristic or attribute of the sample.

[0047] The norm value may be a measure of the size of a vector or matrix. For an input feature, the norm value may be an L2-norm (Euclidean norm), which is used to quantify the importance of the input feature.

[0048] The target norm value may be the norm value corresponding to each matrix element. For example, for the matrix element in the i-th row and j-th column, the target norm value is the norm value of the j-th input dimension.

[0049] The metric may be a standard for evaluating the importance of each element in the weight matrix.

[0050] The layer metric criterion may be an overall evaluation criterion obtained by comprehensively considering the metric criteria of all weight matrices in a linear layer, reflecting the importance of the layer.

[0051] For example, in a preset large language model, each network layer (e.g., each DecoderLayer) may contain multiple linear layers, such as nn.Linear layers. These linear layers can be q_proj, k_proj, v_proj, o_proj in a multi-head attention module, or up_proj, down_proj, etc. in a feedforward network.

[0052] Furthermore, for each linear layer, the position of each matrix element in its weight matrix can be determined. The weight matrix can be a two-dimensional array in which each matrix element has a specific row and column index. In addition, the input data of each linear layer can be obtained. The input data is usually a three-dimensional tensor with a shape of [N, L, Cin], where N is the batch size, L is the sequence length, and Cin is the feature dimension size. For each feature of the input data, its corresponding L2-norm (i.e., norm value) can be calculated.

[0053] Furthermore, for each matrix element in the weight matrix, the target norm value of the corresponding input feature can be determined from multiple pre-calculated norm values ​​according to its element position. For example, for the matrix element in the i-th row and j-th column, the norm value of the j-th input dimension can be determined as its target norm value. Then, the metric criterion for each matrix element is calculated using the following formula: : ; in, is the element in the i-th row and j-th column of the weight matrix, It is the norm value of the j-th dimension matrix element in the input matrix corresponding to the input data.

[0054] Furthermore, for each linear layer in the network layer, the metrics of all matrix elements can be aggregated to generate the layer metric of the linear layer. The calculation of the layer metric can be achieved by summarizing the metrics of all matrix elements, or by calculating the average, maximum or other statistics of the metrics of all elements, and the embodiments of the present application do not impose too many restrictions on this.

[0055] Furthermore, for each network layer, the layer metrics of all linear layers can be aggregated to generate the first metric matrix of the network layer. This can be achieved by summarizing all layer metrics, or calculating the average, maximum, or other statistics of all layer metrics. The embodiments of the present application do not impose too many restrictions on this.

[0056] Through the above steps, a first metric matrix can be generated for each network layer in the preset large language model. The first metric matrix can be used for subsequent pruning operations to reduce the number of parameters and computational overhead of the model while maintaining the performance of the model as much as possible.

[0057] In some implementations, in order to achieve efficient and accurate pruning operations, the pruning rate of each network layer may be determined to ensure the effectiveness and rationality of the pruning process. Exemplarily, the "determining the pruning rate of the corresponding network layer based on the first metric matrix" in step 101 may include: (101.b1) Calculate the metric average for the first metric matrix; (101.b2) For each linear layer in each network layer, obtain the preset threshold constant, and obtain the target threshold based on the product of the metric average and the threshold constant; (101.b3) determining, from the plurality of metric elements of the first metric matrix, a second number of a plurality of target metric elements that are greater than a target threshold; (101.b4) obtaining a third number of the plurality of metric elements of the first metric matrix, and obtaining a singular value ratio based on a ratio of the second number to the third number; (101.b5) Standardize the singular value ratio to obtain the target singular value ratio; (101.b6) Determine the pruning rate for each network layer based on the target singular value ratio corresponding to the multiple linear layers corresponding to each network layer.

[0058] The threshold constant may be a preset fixed value (such as Hyper_m=5, etc., which may be set according to actual conditions), which is used to be multiplied by the metric average to generate a target threshold.

[0059] The target threshold may be a value obtained by multiplying the metric average value by a threshold constant, and may be used to distinguish whether a metric element is important, that is, it is a benchmark for determining whether a metric element should be retained or pruned.

[0060] The metric element may be a single element in the first average metric matrix, representing an importance score of a specific part.

[0061] The second number may be the number of metric elements that are greater than a target threshold, and these elements are considered important and should be retained.

[0062] The third number may be the total number of all metric elements in the first average metric matrix.

[0063] The singular value ratio may be a ratio of the second quantity to the third quantity, reflecting the proportion of important metric elements in the total elements.

[0064] The target singular value ratio may be a result of normalizing the singular value ratio, which makes the comparison between different network layers or models fairer and more reasonable, and helps determine the pruning rate.

[0065] Exemplarily, first, the average value of all elements in the first metric matrix (layer_wmetric) can be calculated to obtain the metric average value. Then, a preset threshold constant Hyper_m is obtained, and the target threshold is obtained according to the product of the threshold constant and the metric average value. The specific process is as follows: max_shred=torch.mean(layer_wmetric) Hyper_m; Furthermore, the number of elements in the first metric matrix that are greater than the target threshold max_shred may be counted as the second number. The specific process is as follows: count=(layer_wmetric>max_shred).sum().item(); Furthermore, the total number of elements in layer_wmetric can be calculated as the third number, and then the ratio of the second number to the third number can be calculated to obtain the singular value ratio. The specific process is as follows: total_params=layer_wmetric.numel(); outlier_ratio=float(count) / total_params 100; Furthermore, the outlier ratios of all network layers can be normalized, that is, the singular value ratios of all layers can be normalized to the range [0,1]. Specifically, the minimum and maximum values ​​can be used for normalization and then scaled (for example, multiplied by Lambda 2, Lamda is usually 0.2), and the target singular value ratio is obtained. The specific process is as follows: ; Furthermore, the target singular value ratio of all network layers can be adjusted to take into account the pruning rate. The specific process is as follows: ; Furthermore, the final pruning rate can be calculated by the following formula: ; Through the above method, the pruning rate corresponding to each network layer can be dynamically calculated according to the singular value ratio, and accurately applied to the pruning of the preset large language model to achieve the purpose of effective compression of the model and accelerated reasoning.

[0066] Step 102: for each of the attention heads included in each network layer, determine a plurality of importance scores according to the corresponding first metric matrix, and construct a second metric matrix according to the plurality of importance scores.

[0067] In some embodiments, in order to evaluate the importance of each attention head, the importance scores of the multiple attention heads included in each network layer can be determined according to the corresponding first metric matrix, and a second metric matrix can be constructed based on these scores, so as to understand the key parts of the internal structure of the model and provide clear guidance for subsequent pruning operations.

[0068] Among them, the attention head may include a set of weight matrices (such as q_proj, k_proj, v_proj) for calculating the query (Query), key (Key) and value (Value). In the Transformer architecture of the large language model, each attention module usually contains multiple attention heads, and each attention head independently executes the self-attention mechanism to capture the relationship between different parts of the input sequence.

[0069] The importance score may be a value calculated for each attention head based on the metric criteria in the first metric matrix corresponding to the attention head, reflecting the importance of the attention head in the entire preset large language model. For example, if the importance score is low, it indicates that the importance of the attention head is low, otherwise it is high.

[0070] The second metric matrix may be a newly constructed matrix, which contains the importance scores of each attention head, and provides a basis for subsequent pruning operations to help determine which attention heads should be retained or removed.

[0071] In some embodiments, the shape of the first metric matrix can be adjusted to a shape corresponding to the number of attention heads, for example (head_num, 128), where head_num represents the number of attention heads in the first metric matrix, and 128 represents that the dimension of the feature vector processed by each attention head is 128.

[0072] Furthermore, after adjusting the shape, each row can be summed, that is, all eigenvectors corresponding to each attention head can be summed to obtain the importance score of each attention head. According to the multiple importance scores corresponding to the multiple attention heads contained in the network layer, the second metric matrix can be constructed.

[0073] In some embodiments, after determining multiple importance scores corresponding to multiple attention heads, the multiple attention heads can be grouped, and a second metric matrix can be constructed based on the grouping results to facilitate subsequent pruning of the attention heads through grouping sparsity to improve the efficiency of pruning. The basis for grouping can be determined according to actual conditions, for example, according to the number of attention heads contained in each group, the attention heads can be grouped in order, or they can be grouped in other ways, and the embodiments of the present application do not impose specific restrictions on this.

[0074] By calculating the importance score for each attention head and constructing a second metric matrix, more refined and more effective pruning is achieved, which helps to improve the efficiency and accuracy of pruning without affecting the performance of the model.

[0075] Please refer to Figure 3 In some embodiments, the present application obtains the first mask matrix and the second mask matrix by setting a group sparse pruning mask function (get_wanda_sp_prune_mask function). Specifically, the first average metric matrix can be obtained by calculating the average of each matrix column in the first metric matrix, and the shape of the first metric matrix can be adjusted to obtain the second average metric matrix. After that, the first pruning threshold is calculated according to the pre-calculated pruning rate, the target group corresponding to each attention head is obtained, and the second average metric matrix is ​​adjusted based on the grouping relationship between multiple attention heads to obtain the second metric matrix, so as to facilitate the subsequent determination of the sparsity of each target group and improve the efficiency of pruning. After that, the first mask matrix is ​​determined using the second metric matrix and the first pruning threshold to determine the channels that need to be pruned, and efficient pruning is performed to effectively release the storage space of the model. Furthermore, the number of groups of pruned target groups can be calculated, and the target number of pruned attention heads can be calculated accordingly, the second mask matrix is ​​calculated according to the target number and the second metric matrix, and the first mask matrix and the second mask matrix are returned to achieve accurate pruning. In this process, in order to ensure the effectiveness and rationality of the process, a second metric matrix can be constructed based on multiple importance scores to effectively evaluate and quantify the importance of each attention head in the large language model, while laying the foundation for efficient and accurate pruning operations. For example, step 102 may include: (102.1) for each of the multiple attention heads included in each network layer, average the matrix columns corresponding to each linear layer of the first metric matrix to obtain a first average metric matrix; (102.2) determining a correspondence between a plurality of matrix features in the first average metric matrix and a plurality of attention heads, and adjusting the first average metric matrix based on the correspondence to obtain a second average metric matrix; (102.3) Determine an importance score corresponding to each attention head based on the second average metric matrix, and construct a second metric matrix based on the multiple importance scores corresponding to the multiple attention heads.

[0076] The first average metric matrix may be a new matrix obtained by averaging the matrix columns corresponding to each linear layer in the first metric matrix of each network layer, which reflects the average importance of each attention head in each linear layer.

[0077] The second average metric matrix may be a new matrix obtained by adjusting the first average metric matrix based on the correspondence between the attention head and the matrix features, and is used to reflect the actual contribution of each attention head.

[0078] In some embodiments, the average importance of each attention head can be preliminarily evaluated by averaging the matrix columns corresponding to each linear layer in the first metric matrix of each network layer. Then, based on the correspondence between the attention head and the matrix features, the first average metric matrix is ​​adjusted to more accurately reflect the importance of each attention head. Finally, the importance score of each attention head is calculated using the information in the second average metric matrix, and these scores are summarized into a new matrix to guide subsequent pruning operations.

[0079] For example, if each network layer of the preset large language model contains 64 attention heads, that is, head_num=64, the feature dimension processed by each attention head is 128. Assume that a certain network layer corresponds to a first metric matrix W_metric, and its size is the same as the weight matrix corresponding to the network layer, for example, both are 8192×8192. Then, when calculating the importance score corresponding to each attention head, the columns of the first metric matrix can be averaged to obtain a first average metric matrix. Each row of the first average metric matrix represents the average importance score of a feature dimension, and the size of the first average metric matrix is ​​1×8192.

[0080] Furthermore, the first average metric matrix can be further adjusted according to the correspondence between multiple matrix features in the first average metric matrix and multiple attention heads to obtain a second average metric matrix to ensure that its correspondence with the attention heads is correct.

[0081] Furthermore, the rows of the second average metric matrix can be summed separately, that is, all the eigenvectors contained in each attention head can be summed to obtain the importance score corresponding to each attention head. Finally, multiple attention heads will correspond to a 64×1 matrix (that is, the second metric matrix), in which each element represents the importance score of an attention head for subsequent pruning decisions.

[0082] By calculating the importance score of each attention head, we can accurately evaluate the parameters that contribute most to the output of the large language model, making it easier to implement more fine-grained pruning later.

[0083] In some implementations, in order to more effectively reduce parameter storage space and accelerate the pruning speed of the model, the attention heads may be further grouped to obtain a second metric matrix, so as to facilitate pruning by utilizing group sparsity. For example, "constructing a second metric matrix based on multiple importance scores corresponding to multiple attention heads" in (102.3) may include: (102.3.1) Get the target group corresponding to each attention head; (102.3.2) Based on the grouping relationship between the multiple attention heads, the position of the importance score of each attention head corresponding to the second average metric matrix is ​​adjusted to obtain a second metric matrix arranged according to the target grouping.

[0084] The target grouping may be a plurality of groups obtained by grouping the attention heads according to certain rules or standards. The importance scores (each importance score corresponds to an attention head) within each target group are grouped in order, or the attention heads within each target group have similar importance or functional characteristics.

[0085] The grouping relationship may be a description of the grouping association between different attention heads. For example, attention head 1 and attention head 2 may belong to the same target group, and attention head 1 and attention head 3 may belong to different target groups.

[0086] In some embodiments, the attention heads can be directly grouped evenly according to the predefined parameter num_key_value_group. For example, if the multi-head attention module of the large language model contains head_num=64 attention heads, set num_key_value_group=8, that is, the 64 attention heads need to be divided into 8 groups, each group will contain 8 attention heads, thus, the shape of the matrix can be adjusted based on the attention scores in each target group, specifically, the shape of the matrix can be adjusted to (head_num / num_key_value_group, num_key_value_group), and a second metric matrix of 8×8 is obtained. Alternatively, the position of each attention head can be adjusted after grouping, and the second metric matrix can be obtained directly after grouping and marking.

[0087] By grouping the attention heads, it is easier to use group sparsity for subsequent pruning, which helps to simplify the pruning process while still achieving a certain degree of structured pruning.

[0088] Step 103, determining a first calibration matrix of the second metric matrix according to the size relationship between the pruning rate and multiple importance scores in the second metric matrix, and determining the corresponding first mask position based on the sparse distribution characteristics of the first calibration matrix to obtain a first mask matrix.

[0089] In some embodiments, in order to improve the efficiency and accuracy of pruning decisions, the sparse distribution characteristics of the matrix can be determined based on the size relationship of multiple importance scores, so as to determine the corresponding mask position, thereby ensuring that the pruning operation of the large language model can maintain its performance as much as possible while reducing parameters.

[0090] The first calibration matrix may be a matrix generated after calibration in the second metric matrix based on the pruning rate and the size of each importance score in the second metric matrix.

[0091] Among them, the sparse distribution characteristic can be the distribution of non-zero elements (i.e., important parts) and zero elements (i.e., unimportant parts) in the first calibration matrix. After grouping according to the attention heads, the sparse distribution characteristic can be the sparse distribution characteristic corresponding to each target group.

[0092] The first mask position may be a position in the first calibration matrix that is marked as needing to be retained (True or 1) or pruned (False or 0), and is used to indicate the attention heads that need to be retained and the attention heads that need to be removed.

[0093] The first mask matrix can be a binary matrix (True / False, or 1 / 0) generated based on the first mask position, which is used for the actual pruning operation. True means retaining the weight or attention head at this position, and False means pruning.

[0094] In some embodiments, a first pruning threshold corresponding to the second metric matrix can be determined based on the pruning rate, and in the second metric matrix, importance scores greater than the first pruning threshold are calibrated as True, and importance scores less than the first pruning threshold are calibrated as False. After the importance scores corresponding to all attention heads in the second metric matrix are calibrated, a first calibration matrix is ​​obtained.

[0095] Furthermore, the first mask position can be determined based on the group sparsity of each target group in the first calibration matrix, with the target group as the unit, to obtain the first mask matrix. Exemplarily, in the first calibration matrix, each row of the first calibration matrix (each row corresponds to the calibration value of multiple importance scores contained in a target group) can be summed. When the sum obtained by the target group is greater than a preset threshold, it indicates that the target group elements are densely distributed and non-sparse, and pruning should not be performed. In the first calibration matrix, all values ​​of the target group can be calibrated to True or 1; and when the sum obtained by the target group is less than the preset threshold, it indicates that the target group elements are sparsely distributed and can be pruned. In the first calibration matrix, all values ​​of the target group can be calibrated to False or 0. At this point, the corresponding first mask position can be determined based on the position of True (or 1) and False (or 0). After the values ​​of all target groups are calibrated, the first mask matrix can be obtained.

[0096] By obtaining the first mask matrix, the specific location of pruning can be clarified, thereby reducing model parameters and computational complexity while maintaining the key performance of the model, achieving model optimization and acceleration, and facilitating model deployment in resource-constrained environments.

[0097] In some embodiments, in order to quickly and effectively evaluate the importance of each attention head in a large language model, a first pruning threshold may be determined according to the pruning rate, and a first calibration matrix corresponding to the second metric matrix may be determined according to the first pruning threshold, so as to quickly implement a pruning decision based on importance scores. For example, "determining the first calibration matrix of the second metric matrix according to the size relationship between the pruning rate and multiple importance scores in the second metric matrix" in step 103 may include: (103.a1) Determine the first number of attention heads to be pruned according to the pruning rate; (103.a2) determining a first pruning threshold from a second metric matrix according to the first quantity and a magnitude relationship of a plurality of importance scores; (103.a3) Compare the first pruning threshold with each importance score in the second metric matrix respectively, and determine the calibration value corresponding to each importance score according to the relationship between the first pruning threshold and each importance score; (103.a4) Determine a first calibration matrix for the second metric matrix based on the multiple calibration values ​​corresponding to the multiple importance scores.

[0098] The first number may be the number of attention heads that need to be pruned based on the pruning rate.

[0099] The first pruning threshold may be a value determined according to the first quantity and the importance score in the second metric matrix. An importance score greater than the first pruning threshold is more important, whereas an importance score less than the first pruning threshold is less important.

[0100] The calibration value may be the result obtained by comparing each importance score with the first pruning threshold, and the calibration value may be a binary value (True / False or 1 / 0). True (or 1) indicates that the attention head should be retained, and False (or 0) indicates that the attention head should be pruned.

[0101] For example, the first number of attention heads to be pruned can be calculated according to the pruning ratio pruning_ratio. Specifically, the first number can be obtained by multiplying the pruning ratio by the total number of all attention heads head_num corresponding to the second metric matrix, that is, pruning_ratio After that, all the values ​​in the second metric matrix can be sorted from small to large to obtain the sorted score array scores_sorted. According to the first number as the index, the value divided by the first number is taken from the score array scores_sorted as the first pruning threshold, that is: threshold=score_sorted[pruning_ratio head_num]; Furthermore, after determining the first pruning threshold, the values ​​in the second metric matrix that are greater than the first pruning threshold can be set to True, and the values ​​that are less than or equal to the first pruning threshold can be set to False, so as to determine the calibration value corresponding to each importance score in the second metric matrix. Afterwards, after all importance scores in the second metric matrix are calibrated, a first calibration matrix of the second metric matrix can be determined based on multiple calibration values, in which all values ​​are calibrated to True or False.

[0102] Through the above method, it can be ensured that the pruning decision is made based on the relative importance of all attention heads, rather than randomly or arbitrarily, making the subsequent pruning process more reasonable and accurate, and effectively avoiding wrong pruning.

[0103] Please refer to Figure 3In some implementations, in order to effectively evaluate and quantify the importance of each target group in the large language model, so as to quickly evaluate the sparse distribution characteristics of the target group in units of the target group and implement fast pruning, the corresponding first mask position can be determined based on multiple target calibration values ​​corresponding to the multiple target groups to obtain a first mask matrix to improve the efficiency of pruning and ensure that the large language model maintains its performance as much as possible while reducing parameters. For example, "determine the corresponding first mask position based on the sparse distribution characteristics of the first calibration matrix to obtain the first mask matrix" in step 103 may include: (103.b1) Obtain multiple calibration values ​​corresponding to multiple attention heads contained in each target group in the first calibration matrix; (103.b2) determining a target group value corresponding to the target group based on a plurality of calibration values ​​corresponding to each target group; (103.b3) Obtaining a preset mask threshold, and comparing the target group value with the mask threshold to obtain a comparison result; wherein the comparison result of each target group of the first calibration matrix is ​​associated with the sparse distribution characteristics of the corresponding target group; (103.b4) Based on the comparison result, multiple calibration values ​​corresponding to the multiple attention heads included in the target group in the first calibration matrix are adjusted to obtain multiple target calibration values ​​corresponding to the multiple attention heads; (103.b5) Determine corresponding first mask positions based on a plurality of target calibration values ​​corresponding to a plurality of target groups, and obtain a first mask matrix.

[0104] The target group value may be a statistical or aggregated result of the calibration values ​​corresponding to all attention heads in each target group. The target group value may be the average value, sum, or other aggregation function result of all calibration values ​​in the target group.

[0105] The mask threshold may be a preset fixed value, which is used to compare with the target group value to determine whether the group needs to adjust its calibration value.

[0106] The comparison result may be a result obtained by comparing the target group value with the mask threshold value. The comparison result may be a Boolean value (True / False) to indicate whether the target group value is greater than or equal to the mask threshold value.

[0107] Among them, the target calibration value can be a calibration value adjusted based on the comparison result. If the comparison result is True, the target calibration values ​​corresponding to all calibration values ​​in the target group are all adjusted to True; if the comparison result is False, the target calibration values ​​corresponding to all calibration values ​​in the target group are all adjusted to False.

[0108] The first mask position may be a position marked in the first calibration matrix as needing to be retained (True) or pruned (False).

[0109] Exemplarily, in the first calibration matrix, based on multiple calibration values ​​of each target group, the target group value corresponding to each target group can be calculated. Exemplarily, if the calibration value corresponding to target group 1 is (False, True, True, True, True, False, True, True), if False corresponds to 0 and True corresponds to 1, the numerical values ​​corresponding to all calibration values ​​are added together to obtain the target group value corresponding to each target group. The target group value of the above example is 6.

[0110] Furthermore, the mask threshold can be set according to the actual situation, and is used to reasonably determine the sparse distribution characteristics corresponding to the target group according to the size of the target group value. When the target group value is greater than the mask threshold, it indicates that the sparse distribution characteristics of the target group corresponding to the target group value are non-sparse. For example, if the mask threshold is 0.25 and the target group value is 6, the comparison result indicates that the target group value is greater than the mask threshold, and the sparse distribution characteristics of the target group are sparse; conversely, when the target group value is less than the mask threshold, it indicates that the sparse distribution characteristics of the target group corresponding to the target group value are sparse. For example, if the mask threshold is 0.25 and the target group value is 0, the comparison result indicates that the target group value is less than the mask threshold, and the sparse distribution characteristics of the target group are sparse.

[0111] Furthermore, when the target group value is greater than the mask threshold, the target calibration value of the target group value can be determined to be True (or 1) in the first calibration matrix, and all calibration values ​​corresponding to the target group are modified to the target calibration value, that is, the calibration value corresponding to each attention head is modified; conversely, when the target group value is greater than the mask threshold, the target calibration value of the target group value can be determined to be False (or 0) in the first calibration matrix, and all calibration values ​​corresponding to the target group are modified to the target calibration value.

[0112] Furthermore, after calculating the target group values ​​corresponding to all target groups and modifying the calibration values ​​corresponding to all target groups according to the target calibration values ​​in the first calibration matrix (or keeping them unchanged, for example, if the original calibration value is False and the target calibration value is also False, no modification is required), the corresponding first mask positions (that is, the False positions that need to be pruned or the True positions that do not need to be pruned) can be determined according to the multiple target calibration values ​​corresponding to the multiple target groups, and the corresponding first mask matrix can be generated.

[0113] In some embodiments, in addition to setting the mask threshold, a basic mask threshold may be further set. The basic mask threshold is smaller than the mask threshold, so that at least some attention heads are retained in extreme cases to avoid the situation where all elements are considered unimportant and pruned. For example, if the mask threshold is 1, the basic mask threshold may be set to 0.25. The specific values ​​of the mask threshold and the basic mask threshold may be set according to actual conditions.

[0114] Exemplarily, if the mask threshold is 0.25, during the calculation process, the sum of all elements in the first calibration matrix is ​​0, that is, all target group values ​​are 0, which means that according to the current mask threshold, no attention head or target group is considered important, that is, all attention heads are below the mask threshold, which will lead to over-pruning and may damage the model performance because important features or weights are mistakenly removed. In order to avoid this situation, a basic mask threshold is introduced so that when no target group satisfies a value greater than the mask threshold, the basic mask threshold is used to determine that at least part of the attention heads should be retained, thereby avoiding a significant drop in the performance of the large language model. For example, if all target group values ​​are 0 (or other values ​​less than the mask threshold), each target group value is re-compared with the basic mask threshold, and the target calibration value corresponding to the target group value greater than the basic mask threshold is determined to be True (or 1), otherwise it is set to False (or 0). Exemplarily, the mask threshold can be set to 0.25 and the basic mask threshold can be set to 0.

[0115] Through the above methods, the robustness of the pruning process can be ensured, and the model performance loss caused by overly strict pruning thresholds can be prevented. By adjusting the mask threshold and the base mask threshold, the degree of pruning can be flexibly controlled to achieve a balance between model compression and performance preservation.

[0116] Step 104, pruning the key channel and value channel associated with the attention head indicated by the first mask matrix.

[0117] In some embodiments, in order to effectively reduce redundant parameters in a large language model and optimize the model structure, the key channel and value channel associated with the attention head indicated by the first mask matrix can be pruned to reduce the computational cost while maintaining the performance of the model as much as possible.

[0118] Among them, the key channel can be the output channel in the key projection layer. Each attention head has a corresponding key projection layer (k_proj), which can map the input features to the key space.

[0119] Among them, the value channel can be the output channel in the value projection layer. Each attention head has a corresponding value projection layer (v_proj), which can map the input features to the value space.

[0120] Exemplarily, when the attention head is in the first mask matrix and the corresponding target calibration value is True (or 1), it indicates that the attention head is important and there is no need to prune the key channel and value channel associated with the attention head in the current network layer; conversely, when the attention head is in the first mask matrix and the corresponding target calibration value is False (or 0), it indicates that the attention head is not important and the key channel and value channel associated with the attention head in the current network layer can be pruned.

[0121] In some embodiments, since the output feature dimensions of the query, key, and value vectors of each attention head are different, for example, the output feature dimension of the query projection layer and the output projection layer is 8192, and the output dimension of the key projection layer and the value projection layer is 1024, the present application uses a first mask matrix to prune the key channel and the value channel associated with the attention head, and uses a second mask matrix to prune the query channel and the output channel associated with the attention head, so as to reduce the model size and the amount of computation while maintaining the overall performance and structural integrity of the model and improving the accuracy of the pruning process. At the same time, since the present application uses group pruning for pruning, the efficiency and accuracy of pruning can be greatly improved.

[0122] Step 105, determine the target number of attention heads indicated by the first mask matrix, determine the second calibration matrix of the second metric matrix according to the size relationship between the target number and multiple importance scores in the second metric matrix, and determine the corresponding second mask position based on the sparse distribution characteristics of the second calibration matrix to obtain the second mask matrix.

[0123] In some embodiments, in order to maintain the matching performance of the multi-head attention module (also known as the multi-head attention mechanism) and avoid calculation errors, a second mask matrix can be calculated based on the target number of pruned attention heads indicated by the first mask matrix to prune the query channel and output channel associated with the attention head, so as to improve the accuracy of the pruning decision, simplify the subsequent pruning operation process, and ensure that the large language model maintains its performance as much as possible while reducing parameters.

[0124] The target number may be based on the number of attention heads marked as False or 0 (i.e., indicating pruning) in the first mask matrix.

[0125] Among them, the second calibration matrix can be based on the number of targets. After sorting the importance scores in the second metric matrix from small to large, a threshold is determined based on the number of targets, and after comparing each importance score with the threshold, the calibrated value corresponding to each importance score is determined according to the comparison result, and a matrix generated based on multiple calibrated values ​​is used to intuitively indicate whether the attention head corresponding to each importance score is important.

[0126] The second mask position may be a position marked in the second calibration matrix as needing to be retained (True) or pruned (False).

[0127] The second mask matrix can be a binary matrix (True / False or 1 / 0) generated based on the second mask position, which is used for the actual pruning operation. True means retaining the attention head at this position, and False means pruning.

[0128] For example, in order to more accurately control the pruning process and perform adaptive pruning, the target number of attention heads indicated by the first mask matrix can be determined, and the corresponding threshold can be updated according to the target number to avoid over-pruning. For example, first, the number of groups num_pruned_groups of the pruned target groups can be calculated, and the calculation formula is as follows: num_pruned_groups=num_key_value_group-sum(W_metric_kv); Among them, num_key_value_group is a predefined parameter, which indicates the number of groups into which all attention heads are divided; sum(W_metric_kv) indicates the number of groups retained after pruning according to the instructions of the first mask matrix, that is, the number of groups that are not pruned; Furthermore, the target number of pruned attention heads, pruned_num_heads, can be calculated. The calculation formula is as follows (where “ / / ” indicates integer division): group_size=head_num / / num_key_value_group; pruned_num_heads=group_size num_pruned_groups; Among them, head_num is the total number of attention heads in the current network layer; group_size is the number of attention heads contained in each target group.

[0129] Furthermore, the second metric matrix can be sorted from small to large to obtain the sorted score array sorted_scores2. Based on the number of targets as the index, the value after the target number is divided is taken from the score array sorted_scores2 as the second pruning threshold, that is: threshold=sorted_scores2[pruned_num_heads]; Furthermore, after determining the second pruning threshold, the values ​​in the second metric matrix that are greater than the second pruning threshold can be set to True, and the values ​​that are less than or equal to the second pruning threshold can be set to False, so as to determine the second calibration value corresponding to each importance score in the second metric matrix. Afterwards, after all importance scores in the second metric matrix are calibrated, a second calibration matrix of the second metric matrix can be determined based on multiple second calibration values, in which all values ​​are calibrated to True or False.

[0130] In some embodiments, for the sparse distribution characteristics based on the second calibration matrix, the corresponding mask positions are determined to obtain the second mask matrix, which is the same as the process of determining the corresponding first mask positions based on the sparse distribution characteristics of the first calibration matrix and obtaining the first mask matrix described above. The embodiment of the present application does not elaborate on the processing process of obtaining the second mask matrix, and specific reference may be made to the aforementioned processing process of obtaining the first mask matrix.

[0131] By determining the target number of attention heads to be pruned as indicated by the first mask matrix, the pruning threshold can be dynamically adjusted to maximize model performance and ensure that the pruned model structure remains symmetrical and complete, thereby achieving efficient model compression and acceleration.

[0132] Step 106, pruning the query channel and output channel associated with the attention head indicated by the second mask matrix to obtain a large language model.

[0133] In some implementations, in order to ensure the alignment and balance of the model, the query channel and the output channel need to be pruned to avoid potential performance loss and achieve a more balanced, effective, and structured model pruning.

[0134] The query channel can be the output channel of the query projection layer. In the multi-head attention module, each attention head has a corresponding query projection layer, which can map the input features to the query space.

[0135] Among them, the output channel can be the output channel corresponding to the linear transformation layer (output projection layer), that is, the output channel after linear transformation.

[0136] Exemplarily, when the corresponding calibration value of the attention head in the second mask matrix is ​​True (or 1), it indicates that the attention head is important, and there is no need to prune the query channel and output channel associated with the attention head in the current network layer; conversely, when the corresponding calibration value of the attention head in the second mask matrix is ​​False (or 0), it indicates that the attention head is not important, and the query channel and output channel associated with the attention head in the current network layer can be pruned.

[0137] In some embodiments, since the output feature dimensions of the query, key, and value vectors of each attention head are different, for example, the output feature dimension of the query projection layer and the output projection layer is 8192, and the output dimension of the key projection layer and the value projection layer is 1024, the present application uses a second mask matrix to prune the key channel and the value channel associated with the attention head, and uses a second mask matrix to prune the query channel and the output channel associated with the attention head, so as to reduce the model size and the amount of calculation while maintaining the overall performance and structural integrity of the model and improving the accuracy of the pruning process. At the same time, since the present application uses group pruning for pruning, the efficiency and accuracy of pruning can be greatly improved.

[0138] Furthermore, the compress function can be called to prune the output channels of the query projection layer, output projection layer, key projection layer, and value projection layer, thereby actually pruning the parameters and effectively reducing the number of parameters of the large language model.

[0139] Furthermore, for the multi-layer perception mechanism (also called feedforward neural network module) contained in each network layer, the pruning threshold of the weight matrix corresponding to the multi-layer perception mechanism can also be determined according to the first metric matrix and the pruning rate, so as to obtain the third mask matrix, and according to the third mask matrix, the pruning function (compress function) is called to prune the weight matrix with small weight in the up_proj layer (upper projection layer), down_proj layer (lower projection layer), or gate_proj layer (gated projection layer).

[0140] Furthermore, after all network layers are pruned, a large language model can be obtained.

[0141] It should be noted that the present application can implement structured pruning of each network layer, which is more hardware-friendly than unstructured pruning. Although unstructured pruning can reduce the amount of computation of the model, the sparse matrix after pruning is difficult to process efficiently on general hardware, and specific software and hardware support is usually required to achieve the acceleration effect. The structured pruning method adopted in the present application retains the regular structure of the matrix by pruning the corresponding channels, so that the pruned model can run efficiently on general hardware without the need for additional hardware support. In addition, the present application belongs to a post-training pruning algorithm, which can maintain the performance of the model without the need for additional fine-tuning training, while significantly reducing the storage space and computational overhead of the model.

[0142] The embodiment of the present application obtains a first metric matrix of each network layer in a plurality of network layers of a preset large language model, and determines a pruning rate of the corresponding network layer based on the first metric matrix; for a plurality of attention heads contained in each network layer, determines a plurality of importance scores according to the corresponding first metric matrix, and constructs a second metric matrix according to the plurality of importance scores; determines a first calibration matrix of the second metric matrix according to a size relationship between the pruning rate and the plurality of importance scores in the second metric matrix, and determines a corresponding first mask position based on a sparse distribution characteristic of the first calibration matrix to obtain a first mask matrix; prunes key channels and value channels associated with the attention heads indicated by the first mask matrix; determines a target number of attention heads indicated by the first mask matrix, determines a second calibration matrix of the second metric matrix according to a size relationship between the target number and the plurality of importance scores in the second metric matrix, and determines a corresponding second mask position based on a sparse distribution characteristic of the second calibration matrix to obtain a second mask matrix; prunes query channels and output channels associated with the attention heads indicated by the second mask matrix to obtain a large language model. In this way, the pruning rate of each network layer can be determined first by the first metric matrix of each network layer of the large language model, so as to adaptively adjust the pruning granularity according to the different layers of the model, and avoid excessive pruning to cause the performance of the model to deteriorate. In addition, the contribution of each attention head to the model output is determined by the importance score of each attention head, which can ensure that the pruning operation will not significantly affect the performance of the model. Further, the key channel and value channel of the attention head to be pruned are determined by the first mask matrix, which effectively reduces the number of parameters of the model. After pruning the key channel and the value channel, the query channel and the output channel of the corresponding attention head are further pruned according to the second mask matrix. In this way, by pruning based on the structural characteristics of the output channels of the key projection layer and the value projection layer, the query projection layer and the output projection layer of the model, the pruning process of different parts can be controlled more finely, ensuring that the important parts are retained, so that the pruned large language model does not require additional fine-tuning training, and can still maintain the original performance, thereby accelerating the computational efficiency of the model. In summary, the present application can reduce the storage space of the large language model and accelerate the computational efficiency of the large language model.

[0143] Please refer to Figure 4 , Figure 4 This is the overall flow chart of the parameter pruning method for a large language model. Figure 4, the technical solution of the present application is generally introduced. Specifically, after starting the parameter pruning process of the large language model, the preset large language model (such as LLaMA-3 model) and the correction data set (such as C4 data set) can be loaded to provide input data for the subsequent calculation of the metric criterion. After that, the loop variable i=0 can be initialized to start traversing each network layer of the large language model, and the pruning rate is calculated layer by layer for each network layer. Specifically, when the current layer sequence number i is less than the total number of network layers of the large language model, the operation of calculating the pruning rate can be continued. When calculating the pruning rate of the current network layer, based on the first metric matrix of the network layer (such as can be calculated by the weight matrix and the input data norm), combined with the preset threshold constant and statistical features (such as the singular value ratio), the pruning rate of the current network layer is dynamically determined, and after the pruning rate of each network layer is stored in the corresponding pruning rate linked list (all_layer_ratio), the next layer (i+1) of the network layer is processed until all network layers are traversed.

[0144] Furthermore, after calculating the pruning rates of all network layers, the pruning rate linked lists of all layers can be normalized to ensure that the pruning rates of each layer are reasonably distributed and to avoid performance loss caused by extreme pruning. Furthermore, after normalization of all network layers, i=0 can be initialized again to start the second cycle and perform structured pruning layer by layer. In the process of structured pruning of each network layer, if the current layer number i is less than the total number of network layers in the model, the structured pruning operation will continue.

[0145] Specifically, in the process of performing a structured pruning operation on the current network layer, the importance score of each attention head can be calculated based on the first metric matrix for multiple attention heads of the current network layer, a second metric matrix can be constructed, and it can be determined whether the current layer is the output projection layer of the attention module. If so, the pruning threshold is determined by sorting according to the pruning rate and the importance score of the second metric matrix, a first calibration matrix is ​​generated, and based on the grouping sparsity of the calibration matrix, a first mask matrix (marking the key channel and value channel to be pruned) and a second mask matrix (marking the query channel and output channel to be pruned) are determined.

[0146] Furthermore, the first mask matrix can be used to call the pruning function (compress function) to prune the output channels corresponding to the key projection layer (k_proj) and the value projection layer (v_proj) of the attention module, and the second mask matrix can be used to prune the output channels corresponding to the query projection layer (q_proj) and the output projection layer (o_proj).

[0147] In some implementations, if the current layer is not the output projection layer of the attention module (i.e., the current network layer is an MLP module), the first metric matrix and the pruning rate are used to calculate the threshold and calculate the third mask matrix. Based on the third mask matrix, the pruning function is called to prune the weight matrices corresponding to the upper projection layer (up_proj), the lower projection layer (down_proj), etc. of the MLP module.

[0148] Furthermore, after the parameters of the attention module and the MLP module of the current network layer are pruned, the next network layer (i+1) can be processed until all network layers are traversed. After the pruning operation of all network layers is completed, the pruned large language model is saved and the processing flow ends.

[0149] The above is only one embodiment of the parameter pruning method for a large language model proposed in this application. The specific operation flow can be increased or decreased without departing from the concept of this application.

[0150] See also Figure 5 The embodiment of the present application further provides a parameter pruning device for a large language model, which can implement the parameter pruning method for the large language model. The parameter pruning device for the large language model includes: An acquisition module 51 is used to acquire a first metric matrix of each network layer in a plurality of network layers of a preset large language model, and determine a pruning rate of the corresponding network layer based on the first metric matrix; A construction module 52 is used to determine a plurality of importance scores for a plurality of attention heads included in each network layer according to the corresponding first metric matrix, and to construct a second metric matrix according to the plurality of importance scores; A first determination module 53 is used to determine a first calibration matrix of the second metric matrix according to the size relationship between the pruning rate and the multiple importance scores in the second metric matrix, and determine the corresponding first mask position based on the sparse distribution characteristics of the first calibration matrix to obtain a first mask matrix; A first pruning module 54, configured to prune the key channel and the value channel associated with the attention head indicated by the first mask matrix; A second determination module 55 is used to determine the target number of attention heads indicated by the first mask matrix, determine a second calibration matrix of the second metric matrix according to the size relationship between the target number and a plurality of importance scores in the second metric matrix, and determine the corresponding second mask position based on the sparse distribution characteristics of the second calibration matrix to obtain a second mask matrix; The second pruning module 56 is used to prune the query channel and the output channel associated with the attention head indicated by the second mask matrix to obtain a large language model.

[0151] The specific implementation of the parameter pruning device of the large language model is basically the same as the specific implementation of the parameter pruning method of the large language model described above, and will not be repeated here. On the premise of meeting the requirements of the embodiment of the present application, the parameter pruning device of the large language model can also be provided with other functional modules to implement the parameter pruning method of the large language model in the above embodiment.

[0152] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the parameter pruning method of the large language model when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0153] See also Figure 6 , Figure 6 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes: The processor 61 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 62 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 62 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 62, and the processor 61 calls and executes the parameter pruning method of the large language model in the embodiment of the present application; Input / output interface 63, used to implement information input and output; Communication interface 64, used to realize communication interaction between the device and other devices, which can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); A bus 65 that transmits information between the various components of the device (e.g., the processor 61, the memory 62, the input / output interface 63, and the communication interface 64); The processor 61 , the memory 62 , the input / output interface 63 and the communication interface 64 are connected to each other in communication within the device via a bus 65 .

[0154] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the parameter pruning method of the above-mentioned large language model.

[0155] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0156] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0157] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0158] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0160] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0161] It should be understood that in the present application, "at least one (item)" and "several" refer to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0162] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0163] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0165] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0166] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A parameter pruning method for a large language model, characterized in that: The method comprises: Obtaining a first metric matrix of each network layer in a plurality of network layers of a preset large language model, and determining a pruning rate corresponding to the network layer based on the first metric matrix; For the multiple attention heads included in each network layer, determine a plurality of importance scores according to the corresponding first metric matrix, and construct a second metric matrix according to the plurality of importance scores; Determining a first calibration matrix of the second metric matrix according to a size relationship between the pruning rate and a plurality of importance scores in the second metric matrix, and determining a corresponding first mask position based on a sparse distribution characteristic of the first calibration matrix to obtain a first mask matrix; Pruning the key channel and the value channel associated with the attention head indicated by the first mask matrix; Determine a target number of attention heads indicated by the first mask matrix, determine a second calibration matrix of the second metric matrix according to a size relationship between the target number and a plurality of importance scores in the second metric matrix, and determine a corresponding second mask position based on a sparse distribution characteristic of the second calibration matrix to obtain a second mask matrix; The query channel and the output channel associated with the attention head indicated by the second mask matrix are pruned to obtain a large language model.

2. The parameter pruning method for a large language model according to claim 1, characterized in that: The step of determining a plurality of importance scores for the plurality of attention heads included in each network layer according to the corresponding first metric matrix, and constructing a second metric matrix according to the plurality of importance scores comprises: For each of the attention heads included in each network layer, averaging the matrix columns corresponding to each linear layer of the first metric matrix to obtain a first average metric matrix; Determine a correspondence between a plurality of matrix features in the first average metric matrix and the plurality of attention heads, and adjust the first average metric matrix based on the correspondence to obtain a second average metric matrix; Based on the second average metric matrix, an importance score corresponding to each attention head is determined, and based on the multiple importance scores corresponding to the multiple attention heads, a second metric matrix is ​​constructed.

3. The parameter pruning method for a large language model according to claim 2, characterized in that: The constructing a second metric matrix based on the multiple importance scores corresponding to the multiple attention heads includes: Get the target group corresponding to each attention head; Based on the grouping relationship between the multiple attention heads, the position of the importance score of each attention head corresponding to the second average metric matrix is ​​adjusted to obtain a second metric matrix arranged according to the target grouping.

4. The parameter pruning method for a large language model according to claim 1, characterized in that: The determining a first calibration matrix of the second metric matrix according to the size relationship between the pruning rate and a plurality of importance scores in the second metric matrix comprises: Determining a first number of attention heads to be pruned according to the pruning rate; Determining a first pruning threshold from the second metric matrix according to the first quantity and the magnitude relationship of the plurality of importance scores; Comparing the first pruning threshold with each importance score in the second metric matrix respectively, and determining a calibration value corresponding to each importance score according to a magnitude relationship between the first pruning threshold and each importance score; A first calibration matrix of the second metric matrix is ​​determined based on a plurality of calibration values ​​corresponding to the plurality of importance scores.

5. The parameter pruning method for a large language model according to claim 1, characterized in that: The determining of the corresponding first mask position based on the sparse distribution characteristic of the first calibration matrix to obtain the first mask matrix includes: Obtaining multiple calibration values ​​corresponding to multiple attention heads included in each target group in the first calibration matrix; Determining a target group value corresponding to the target group based on the multiple calibration values ​​corresponding to each target group; Obtaining a preset mask threshold, and comparing the target group value with the mask threshold to obtain a comparison result; wherein the comparison result of each target group of the first calibration matrix is ​​associated with a sparse distribution characteristic of the corresponding target group; Based on the comparison result, adjusting a plurality of calibration values ​​corresponding to the plurality of attention heads included in the target group in the first calibration matrix to obtain a plurality of target calibration values ​​corresponding to the plurality of attention heads; Based on the multiple target calibration values ​​corresponding to the multiple target groups, corresponding first mask positions are determined to obtain a first mask matrix.

6. The parameter pruning method for a large language model according to claim 1, characterized in that: The step of obtaining a first metric matrix of each network layer in a plurality of network layers of a preset large language model includes: Obtain multiple linear layers corresponding to each network layer included in the large language model, and determine multiple weight matrices corresponding to each linear layer in each network layer; Determining the element position of each matrix element in each weight matrix; Obtain input data of the linear layer corresponding to each weight matrix, and calculate multiple norm values ​​corresponding to the input data in multiple input dimensions; Based on the element position of each matrix element, determining a target norm value of a corresponding input dimension from the plurality of norm values; Determining a measurement criterion for each matrix element according to each matrix element and the target norm value; Generate a layer metric criterion corresponding to each linear layer based on multiple metric criteria of multiple matrix elements corresponding to the multiple weight matrices; Based on the multiple layer measurement criteria corresponding to the multiple linear layers, a first measurement matrix corresponding to each network layer is obtained.

7. The parameter pruning method for a large language model according to claim 1, characterized in that: The determining, based on the first metric matrix, a pruning rate corresponding to the network layer includes: Calculating a metric average for the first metric matrix; For each linear layer in each network layer, a preset threshold constant is obtained, and a target threshold is obtained according to the product of the metric average and the threshold constant; Determining, from among the plurality of metric elements of the first metric matrix, a second number of a plurality of target metric elements greater than the target threshold; Obtaining a third number of the plurality of metric elements of the first metric matrix, and obtaining a singular value ratio based on a ratio of the second number to the third number; Performing standardization on the singular value ratio to obtain a target singular value ratio; Based on the target singular value ratios corresponding to the multiple linear layers corresponding to each network layer, the pruning rate of each network layer is determined.

8. A parameter pruning device for a large language model, characterized in that: The device comprises: An acquisition module, configured to acquire a first metric matrix of each network layer in a plurality of network layers of a preset large language model, and determine a pruning rate corresponding to the network layer based on the first metric matrix; A construction module, configured to determine a plurality of importance scores for a plurality of attention heads included in each network layer according to the corresponding first metric matrix, and construct a second metric matrix according to the plurality of importance scores; A first determination module is used to determine a first calibration matrix of the second metric matrix according to the pruning rate and the size relationship of multiple importance scores in the second metric matrix, and determine the corresponding first mask position based on the sparse distribution characteristics of the first calibration matrix to obtain a first mask matrix; A first pruning module, configured to prune a key channel and a value channel associated with the attention head indicated by the first mask matrix; A second determination module is used to determine the target number of attention heads indicated by the first mask matrix, determine a second calibration matrix of the second metric matrix according to the size relationship between the target number and multiple importance scores in the second metric matrix, and determine the corresponding second mask position based on the sparse distribution characteristics of the second calibration matrix to obtain a second mask matrix; The second pruning module is used to prune the query channel and the output channel associated with the attention head indicated by the second mask matrix to obtain a large language model.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the parameter pruning method for a large language model according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the parameter pruning method for a large language model according to any one of claims 1 to 7 is implemented.

Images