Structured pruning method for large language model and related equipment

Abstract

The embodiment of the invention provides a structured pruning method for a large language model and related equipment, and belongs to the technical field of neural network lightweight. The method comprises the following steps: acquiring a first weight matrix of a plurality of attention modules of a to-be-pruned initial large language model and a second weight matrix of a plurality of sensing modules; determining a global pruning threshold based on a first fluctuation metric matrix corresponding to the first weight matrix and a second fluctuation metric matrix corresponding to the second weight matrix; determining a key value mask matrix and a query mask matrix corresponding to each attention module and a sensing mask matrix corresponding to each sensing module based on a global pruning threshold; performing pruning processing on the corresponding first weight matrix by using the key value mask matrix and the query mask matrix, and performing pruning processing on the corresponding second weight matrix by using the perception mask matrix, so as to determine the pruned large language model of the initial large language model. According to the method, the model compression accuracy of the large language model can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of lightweight neural networks, and in particular to a structured pruning method for large language models and related equipment. Background Art

[0002] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data. In order to reduce the model's computing resource requirements and storage space without significantly reducing model performance, large language models need to be compressed.

[0003] In the related art, model compression is achieved by uniformly pruning all network layers of a large language model. However, for some large language models with special structures, such as LLaMA-3, the dimensions of the network layers they contain are different, so traditional model compression methods are no longer applicable, which leads to poor model compression accuracy of large language models. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a structured pruning method and related equipment for a large language model, aiming to improve the model compression accuracy of the large language model.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a structured pruning method for a large language model, comprising: Obtain an initial large language model to be pruned, and obtain corresponding first weight matrices from multiple attention modules of the initial large language model, and obtain corresponding second weight matrices from multiple perception modules of the initial large language model; Determine a first volatility measurement matrix based on the first weight matrix, determine a second volatility measurement matrix based on the second weight matrix, and determine a global pruning threshold according to the first volatility measurement matrix and the second volatility measurement matrix; Based on the global pruning threshold and the first fluctuation metric matrix, determine the key value mask matrix and the query mask matrix corresponding to each attention module, and based on the global pruning threshold and the second fluctuation metric matrix, determine the perception mask matrix corresponding to each perception module; Using the key-value mask matrix and the query mask matrix of each attention module, the corresponding first weight matrix is ​​pruned to obtain a first target matrix, and using the perception mask matrix of each perception module, the corresponding second weight matrix is ​​pruned to obtain a second target matrix; Based on the first target matrix and the second target matrix, a large language model after pruning the initial large language model is determined.

[0006] In an embodiment of the present application, each attention module includes a plurality of first weight matrices, each of which includes an initial key matrix, an initial query matrix, an initial value matrix, and a first output matrix, and the output dimensions of the initial key matrix, the initial query matrix, the initial value matrix, and the first output matrix are not completely the same; Determining a first volatility measurement matrix based on the first weight matrix includes: Obtain an initial sample and input the initial sample into an initial large language model; For each first weight matrix, embedding processing is performed on the initial sample based on the initial key matrix, the initial query matrix, the initial value matrix and the first output matrix in sequence to obtain a corresponding first output sample; Determine a first original matrix based on the first output sample, and perform mean processing on multiple first original matrices belonging to the same attention module to obtain a first mean matrix; The first mean matrices corresponding to all attention modules are stacked to obtain the first fluctuation measure matrix.

[0007] In the embodiment of the present application, a plurality of first original matrices belonging to the same attention module are averaged to obtain a first average matrix, including: Performing mean processing on the matrix elements in each first original matrix respectively to obtain the corresponding metric mean of each first original matrix; Based on the matrix elements in each first original matrix, a metric standard difference value corresponding to each first original matrix is ​​calculated; Based on the metric mean and the metric standard deviation, update the matrix elements in the corresponding first original matrix to obtain a first normalized matrix; Perform mean processing on multiple first normalized matrices belonging to the same attention module to obtain a first mean matrix.

[0008] In an embodiment of the present application, each attention module is connected to a corresponding perception module, each perception module includes a plurality of second weight matrices, and each second weight matrix includes a mapping matrix and a second output matrix; Determining a second volatility measurement matrix based on the second weight matrix includes: For each first weight matrix, embedding the first output sample based on the mapping matrix and the second output matrix in turn to obtain a corresponding second output sample; Determine a second original matrix based on the second output sample, and perform mean processing on multiple second original matrices belonging to the same perception module to obtain a second mean matrix; The second mean matrices corresponding to all perception modules are stacked to obtain the second fluctuation measure matrix.

[0009] In an embodiment of the present application, determining a global pruning threshold according to the first fluctuation metric matrix and the second fluctuation metric matrix includes: Combining the first volatility metric matrix and the second volatility metric matrix to obtain a global metric matrix; Constructing an initial matrix having the same matrix size as that of the global metric matrix, the initial matrix including a plurality of initial elements; Based on a preset pruning rate, a preset cumulative function and a global metric matrix, a plurality of initial elements are updated to obtain a corresponding plurality of updated elements; Determine a target update element with the smallest element value from the multiple update elements, and determine a first target position where the target update element is located; A global pruning threshold is determined based on a second target position that matches the first target position with the global metric matrix.

[0010] In an embodiment of the present application, based on the global pruning threshold and the first fluctuation metric matrix, a key value mask matrix and a query mask matrix corresponding to each attention module are determined, including: Based on the grouping parameters preset by the initial large language model, the matrix shape of each first mean matrix is ​​adjusted to obtain an initial mask matrix; According to the global pruning threshold, the matrix elements of the initial mask matrix are updated to obtain the initial key-value mask matrix; According to the preset local pruning threshold, the matrix elements of the initial key-value mask matrix are masked to obtain the key-value mask matrix corresponding to each attention module; Determine the number of pruned attention heads of any attention module based on the grouping parameter, and update the global pruning threshold based on the number of attention heads to obtain an updated global pruning threshold; Based on the updated global pruning threshold, the matrix elements of the initial mask matrix are masked to obtain the corresponding query mask matrix of each attention module.

[0011] In the embodiment of the present application, the local pruning threshold includes a first local pruning threshold and a second local pruning threshold; According to the preset local pruning threshold, the matrix elements of the initial key-value mask matrix are masked to obtain the key-value mask matrix corresponding to each attention module, including: Sum all matrix elements of the initial key-value mask matrix to get the actual total value; If the actual total value does not meet the preset total value, the initial key-value mask matrix is ​​masked using the first local pruning threshold to obtain the key-value mask matrix corresponding to each attention module; If the actual total value meets the preset total value, the initial key-value mask matrix is ​​masked using the second local pruning threshold to obtain the key-value mask matrix corresponding to each attention module.

[0012] In an embodiment of the present application, the global pruning threshold is updated based on the number of attention heads to obtain an updated global pruning threshold, including: Sorting each matrix element in the first mean matrix to obtain an updated first mean matrix; Determine that the matrix element at the third target position in the updated first mean matrix that matches the number of attention heads is used to update the global pruning threshold.

[0013] In the embodiment of the present application, after determining the large language model after the initial large language model is pruned, the method further includes: Obtain evaluation samples and validation samples that represent the expected output; Input the evaluation sample into the large language model, embed the evaluation sample based on the first target matrix and the second target matrix to obtain an actual sample representing the actual output; The sample difference between the actual sample and the verification sample is calculated, and the large language model is pruned and adjusted based on the sample difference to obtain the pruned and adjusted large language model.

[0014] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a structured pruning device for a large language model, comprising: An acquisition module, used to acquire an initial large language model to be pruned, and acquire a corresponding first weight matrix from multiple attention modules of the initial large language model, and acquire a corresponding second weight matrix from multiple perception modules of the initial large language model; A fluctuation metric matrix determination module, used to determine a first fluctuation metric matrix based on a first weight matrix, determine a second fluctuation metric matrix based on a second weight matrix, and determine a global pruning threshold according to the first fluctuation metric matrix and the second fluctuation metric matrix; A global pruning threshold determination module, used to determine a key value mask matrix and a query mask matrix corresponding to each attention module based on the global pruning threshold and the first fluctuation metric matrix, and to determine a perception mask matrix corresponding to each perception module based on the global pruning threshold and the second fluctuation metric matrix; A mask matrix determination module is used to prune the corresponding first weight matrix using the key value mask matrix and the query mask matrix of each attention module to obtain a first target matrix, and to prune the corresponding second weight matrix using the perception mask matrix of each perception module to obtain a second target matrix; The target module determines the large language model after the initial large language model is pruned based on the first target matrix and the second target matrix.

[0015] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the structured pruning method for a large language model of the above-mentioned first aspect when executing the computer program.

[0016] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the structured pruning method for a large language model of the first aspect is implemented.

[0017] The present application proposes a structured pruning method for a large language model and related equipment, the method comprising: obtaining an initial large language model to be pruned, and obtaining a corresponding first weight matrix from multiple attention modules of the initial large language model, and obtaining a corresponding second weight matrix from multiple perception modules of the initial large language model; determining a first fluctuation measurement matrix based on the first weight matrix, determining a second fluctuation measurement matrix based on the second weight matrix, and determining a global pruning threshold based on the first fluctuation measurement matrix and the second fluctuation measurement matrix; determining a key value mask matrix and a query mask matrix corresponding to each attention module based on the global pruning threshold and the first fluctuation measurement matrix, and ... The second fluctuation measurement matrix determines the corresponding perceptual mask matrix of each perception module; the mask matrices of different output dimensions can ensure that the pruning operation is adapted to all processing layers of the initial large language model, so as to flexibly cope with the different output dimensions of each layer in each attention module of the initial large language model; then, the key value mask matrix and the query mask matrix of each attention module are used to prune the corresponding first weight matrix to obtain the first target matrix, and the perceptual mask matrix of each perception module is used to prune the corresponding second weight matrix to obtain the second target matrix; based on the first target matrix and the second target matrix, the large language model after the initial large language model is pruned is determined. In this way, the corresponding mask matrix is ​​generated in a targeted manner according to the actual conditions of each attention module and each perception module in each processing layer, which can improve the model compression accuracy of the large language model while adapting to the special structure of the initial large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of an optional implementation environment of a structured pruning device for a large language model provided in an embodiment of the present application; Figure 2 This is an optional flowchart of a structured pruning method for a large language model provided in an embodiment of the present application; Figure 3 is another optional flow chart of the structured pruning method for a large language model provided in an embodiment of the present application; Figure 4 This is another optional flowchart of the structured pruning method for a large language model provided in an embodiment of the present application; Figure 5 It is an optional module schematic diagram of a structured pruning device for a large language model provided in an embodiment of the present application; Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0020] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0022] First, some nouns involved in this application are analyzed: ‌Deep Learning is a machine learning method based on neural networks. It simulates the working mode of human brain neurons by building a multi-layer neural network model, so that computers can autonomously learn and extract high-level features from data.‌ The core idea of ​​deep learning is to gradually abstract data into more and more advanced feature representations through layer-by-layer nonlinear transformations, thereby showing excellent performance in complex tasks.‌

[0023] Natural Language Processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0024] Artificial Intelligence (AI): is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0025] A large language model refers to a deep learning model trained with a large amount of text data, which enables the model to generate natural language text or understand the meaning of language text. A large language model can provide in-depth knowledge and language production on various topics by training on a huge data set, learning the patterns and structures of natural language through large-scale unsupervised training, and simulating the human language cognition and generation process to a certain extent.

[0026] The number of parameters in a large language model is huge, often starting from billions, so it requires huge computing and storage overhead. In order to reduce the computing resource requirements and storage space of the model without significantly reducing the performance of the model, it is usually necessary to compress the large language model in actual application.

[0027] Among them, structured pruning is a model compression technology that aims to reduce the number of parameters and computational complexity of the model by removing some specific structures in the neural network. In traditional large language models such as LLaMA and LLaMA2, the output dimensions of the query layer, key-value layer, and output layer of the attention module are usually the same. Therefore, in related technologies, all network layers of the large language model are uniformly pruned using the same mask matrix to achieve model compression, where the network layer includes the query layer, key-value layer, and output layer. However, for some special large language models that are different from traditional model structures, such as LLaMA-3 (LLaMA3), the output dimensions of the key layer / value layer are different from the output dimensions of the query layer, so the traditional model compression method is no longer applicable, which leads to poor model compression accuracy of large language models.

[0028] Based on this, an embodiment of the present application provides a structured pruning method and related equipment for a large language model, aiming to improve the model compression accuracy of the large language model.

[0029] For example, Figure 1 As shown, Figure 1 It is a schematic diagram of an optional implementation environment of a structured pruning device for a large language model provided in an embodiment of the present application, wherein the implementation environment includes a terminal 11 and a server 12, wherein at least one terminal 11 and the server 12 are connected via a communication network.

[0030] Further, the server end 12 is deployed with a structured pruning device for a large language model proposed in an embodiment of the present application (for the sake of ease of description, it may also be referred to as a "structured pruning device" below). After obtaining the initial large language model to be pruned, and obtaining the corresponding first weight matrix from multiple attention modules of the initial large language model, and obtaining the corresponding second weight matrix from multiple perception modules of the initial large language model; the server end 12 determines a first fluctuation measurement matrix based on the first weight matrix, determines a second fluctuation measurement matrix based on the second weight matrix, and determines a global pruning threshold according to the first fluctuation measurement matrix and the second fluctuation measurement matrix; then, Based on the global pruning threshold and the first fluctuation measure matrix, the key-value mask matrix and the query mask matrix corresponding to each attention module are determined, and based on the global pruning threshold and the second fluctuation measure matrix, the perceptual mask matrix corresponding to each perception module is determined; then, the key-value mask matrix and the query mask matrix of each attention module are used to prune the corresponding first weight matrix to obtain the first target matrix, and the perceptual mask matrix of each perception module is used to prune the corresponding second weight matrix to obtain the second target matrix; finally, based on the first target matrix and the second target matrix, the large language model obtained after pruning the initial large language model is determined.

[0031] It is understandable that the amount of computation and storage space required for the pruned large language model is small, so after determining the pruned large language model, the server 12 can send the large language model to the terminal 11, so that the pruned large language model can run on the terminal 11. The initial large language model obtained by the server 12 can be sent by the terminal 11 that receives the large language model last, or can be sent by other terminals or the server.

[0032] In addition, the above structured pruning method can also be performed only on the terminal 11 side.

[0033] Among them, the server end 12 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. In addition, the server end 12 can also be a node server in the blockchain network. The terminal 11 can be a mobile phone, a computer, an intelligent voice interaction device, an intelligent wearable device, a smart home appliance, a car terminal, etc., but is not limited to this. The terminal 11 and the server end 12 can be directly or indirectly connected by wired or wireless communication, and the embodiments of the present application are not limited here.

[0034] It should be noted that in the embodiments of the present application, when it comes to information related to user characteristics such as user basic information or user identity, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the user's separate permission or consent will be obtained first. After the user's separate permission or consent is clearly obtained, the necessary data for the normal operation of the embodiments of the present application will be obtained. For example, when the embodiments of the present application obtain the initial large language model to be pruned, the authorization or consent of the relevant personnel will be obtained first, otherwise the initial large language model and other relevant data that cannot be applied to the embodiments of the present application will be obtained. In addition, the other relevant data obtained by the structured pruning device of the present application are all authorized data, which will not be repeated here.

[0035] In the embodiment of the present application, the structured pruning device will be described from the perspective of the structured pruning device, which can be integrated into a computer device, such as a server. Figure 2 As shown, Figure 2 is an optional flowchart of a structured pruning method for a large language model provided in an embodiment of the present application. Figure 2The flowchart shown may include but is not limited to the following steps 101 to 105. The structured pruning device executes a structured pruning method for a large language model (for the sake of convenience, it may also be referred to as a "structured pruning method" below). The specific process is as follows. It should be noted that the present embodiment performs Figure 2 The order of step 101 to step 105 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0036] Step 101, obtain an initial large language model to be pruned, and obtain a corresponding first weight matrix from multiple attention modules of the initial large language model, and obtain a corresponding second weight matrix from multiple perception modules of the initial large language model.

[0037] The following is a detailed description of step 101.

[0038] Among them, the large language model includes multiple processing layers. In order to overcome the limitations of traditional models in processing long-distance dependencies and improve the performance of large language models in various natural language processing tasks, many large language models have introduced multi-head self-attention mechanisms. In this case, each processing layer of the large language model includes at least one set of interconnected attention modules and perception modules; each attention module includes multiple sets of interconnected key layers (Key, K), query layers (Query, Q), value layers (Value, V) and the first output layer. The multi-head self-attention mechanism dynamically calculates the correlation of different positions in the input sequence through the collaboration between the key layer, query layer, value layer and the first output layer, so that the large language model can capture complex dependencies and improve its performance in various natural language processing tasks.

[0039] Among them, in order to realize different degrees of processing of input data, the key layer, query layer, value layer and first output layer are each provided with a corresponding weight parameter matrix. In the embodiment of the present application, the initial large language model refers to the original large-scale deep learning model without any compression or optimization processing, and the key layer of the initial large language model is provided with an initial key matrix, the query layer is provided with an initial query matrix, the value layer is provided with an initial value matrix, and the first output layer is provided with a first output matrix. Each group of interconnected initial value matrices, initial query matrices, initial value matrices and first output matrices is called a first weight matrix; the output dimensions between the initial key matrix, the initial query matrix, the initial value matrix and the first output matrix are not exactly the same. In addition, each perception module includes multiple groups of interconnected mapping layers and a second output layer, the mapping layer is provided with a mapping matrix, and the second output layer is provided with a second output matrix.

[0040] Among them, the attention module is used to capture the dependencies between different positions in the input data and dynamically focus on different parts of the input data; the perception module is used to further analyze and process the output vector of the attention module to capture more complex data relationships, thereby enhancing the expressive power of the large language model.

[0041] Exemplarily, the initial large language model obtained in the embodiment of the present application is LLaMA3, which is a large language model released by Meta in April 2024. Among them, the output dimension of the key layer (k_proj) / value layer (v_proj) of LLaMA3 is different from the output dimension of the query layer (q_proj). It should be noted that the special structure of LLaMA3 makes the traditional structured pruning method not applicable to it.

[0042] The embodiment of the present application will take LLaMA3 as the initial large language model as an example to explain the structured pruning method proposed in the embodiment of the present application, and the beneficial effects of the embodiment of the present application will also gradually appear in the subsequent description.

[0043] Step 102, determining a first volatility measurement matrix based on the first weight matrix, determining a second volatility measurement matrix based on the second weight matrix, and determining a global pruning threshold based on the first volatility measurement matrix and the second volatility measurement matrix.

[0044] The following is a detailed description of step 102.

[0045] Among them, the first fluctuation measurement matrix is ​​a measurement matrix calculated based on multiple first weight matrices of each attention module, and the first fluctuation measurement matrix is ​​used to evaluate the influence of each first weight matrix on the performance of the initial large language model; the second fluctuation measurement matrix is ​​a measurement matrix calculated based on multiple second weight matrices of each perception module, and the second fluctuation measurement matrix is ​​used to evaluate the influence of each second weight matrix on the performance of the initial large language model.

[0046] Among them, the global pruning threshold is used to subsequently generate the key-value mask matrix and query mask matrix corresponding to each attention module, as well as the perception mask matrix corresponding to each perception module.

[0047] In an embodiment of the present application, determining a first volatility measurement matrix based on a first weight matrix includes the following steps: (102.a.1) Obtain an initial sample and input the initial sample into an initial large language model.

[0048] (102.a.2) For each of the first weight matrices, embedding processing is performed on the initial samples based on the initial key matrix, the initial query matrix, the initial value matrix and the first output matrix in sequence to obtain corresponding first output samples.

[0049] (102.a.3) Determine a first original matrix based on the first output sample, and perform mean processing on multiple first original matrices belonging to the same attention module to obtain a first mean matrix.

[0050] (102.a.4) Stack the first mean matrices corresponding to all attention modules to obtain the first fluctuation measure matrix.

[0051] Steps (102.a.1) to (102.a.4) are described in detail below.

[0052] Among them, the initial sample refers to the original data input into the large language model, which can be a text sequence, an image, or other types of data, depending on the actual application scenario of LLaMA3. For example, in a natural language processing task, the initial sample is usually one or more paragraphs of text, such as a sentence or a paragraph. In addition, the initial sample can be obtained from an open source database, or it can be obtained by real-time input by relevant personnel. The embodiment of the present application does not limit the source and quantity of the initial sample.

[0053] Furthermore, the attention module on any processing layer of LLaMA3 usually includes multiple attention heads, each of which is used to independently calculate the attention score, and each attention head corresponds to a first weight matrix. Then, for each attention head, the initial sample is embedded based on the corresponding initial key matrix, initial query matrix, initial value matrix and first output matrix in turn to obtain the corresponding first output sample. Among them, the attention head is the basic component unit of the multi-head attention mechanism in the initial large language model architecture.

[0054] Furthermore, by the following formula <1> , calculate the corresponding fluctuation measure (first original matrix) of any attention head in each attention module: <1> Where N is the number of initial samples input; is the layer number, representing the processing layers; Indicates that the nth sample is The vector value output by the j-th attention head of the layer (the first output sample); Indicates The current average value of the j-th column of the layer; Indicates The historical average of the jth column of the layer input; Indicates The square of the 2-norm of the first weight matrix of the j-th column of the layer input.

[0055] Among them, the 2-norm, also known as the Euclidean norm or L2 norm, is a measurement method of a vector or matrix, which is used to measure the "length" of a vector or the "size" of a matrix.

[0056] In an embodiment of the present application, performing mean processing on multiple first original matrices belonging to the same attention module to obtain a first mean matrix includes the following steps: (A.1) Perform mean processing on the matrix elements in each first original matrix respectively to obtain the corresponding metric mean of each first original matrix.

[0057] (A.2) Based on the matrix elements in each first original matrix, the metric standard difference value corresponding to each first original matrix is ​​calculated.

[0058] (A.3) Based on the metric mean and the metric standard deviation, update the matrix elements in the corresponding first original matrix to obtain a first normalized matrix.

[0059] (A.4) Perform mean processing on multiple first normalized matrices belonging to the same attention module to obtain a first mean matrix.

[0060] Steps (A.1) to (A.4) are described in detail below.

[0061] In the embodiment of the present application, in order to make different features comparable, improve model performance, and accelerate the training process, the implementation of the present application will also normalize the first original matrix. Specifically, the following formula is used: <2> Calculate the first normalized matrix after normalization :

[0062] <2> in, represents the i-th first original matrix; represents the ith metric mean, which is obtained by averaging the matrix elements in the ith first original matrix; std is the standard deviation function, is the standard deviation of the metric.

[0063] Furthermore, multiple first normalized matrices belonging to the same attention module are averaged to obtain a first mean matrix; then, the first mean matrices corresponding to all attention modules are stacked to obtain a first fluctuation measure matrix.

[0064] In an embodiment of the present application, determining the second volatility measurement matrix based on the second weight matrix includes the following steps: (102.b.1) For each first weight matrix, embedding processing is performed on the first output sample based on the mapping matrix and the second output matrix in turn to obtain a corresponding second output sample.

[0065] (102.b.2) Determine a second original matrix based on the second output sample, and perform mean processing on multiple second original matrices belonging to the same perception module to obtain a second mean matrix.

[0066] (102.b.3) Stack the corresponding second mean matrices of all perception modules to obtain the second fluctuation measure matrix.

[0067] Steps (102.b.1) to (102.b.3) are described in detail below.

[0068] In the embodiment of the present application, each attention module is connected to the corresponding perception module, each perception module includes multiple second weight matrices, and each second weight matrix includes a mapping matrix and a second output matrix. Among them, the mapping matrix of LLaMA3 includes a control gate matrix (gate_proj) and an ascending matrix (up_proj), and the second output matrix is ​​a descending matrix (down_proj).

[0069] Among them, through the following formula <3> , calculate the corresponding fluctuation measure of each perception module (the second original matrix): <3> Where N comes from the first output sample of the attention module; is the layer number, representing the processing layers; Indicates the nth first output sample, after the The vector value (second output sample) output after processing the j-th column mapping matrix or the second output matrix of the layer; Indicates The current average value of the j-th column of the layer; Indicates The historical average of the jth column of the layer input; Indicates The square of the 2-norm of the first weight matrix of the j-th column of the layer input.

[0070] Further, the formula <2> The calculation is the square of the product of the input data sample variance and the weight matrix 2-norm, <3> The product of the input data sample variance and the weight matrix 2-norm is calculated. <3> With <2> The difference is that the calculation result is not squared so that the attention module can control the value range through square calculation and make outliers more prominent. Outliers refer to data points in the data set that are significantly different from other data points.

[0071] Furthermore, a plurality of second original matrices belonging to the same perception module are averaged to obtain a second mean matrix; then, the second mean matrices corresponding to all perception modules are stacked to obtain a second fluctuation measurement matrix.

[0072] Alternatively, after obtaining the second original matrix, the second original matrix is ​​normalized by a method similar to steps (A.1) to (A.4) to obtain a second normalized matrix; then, multiple second normalized matrices belonging to the same perception module are averaged to obtain a second mean matrix; then, the second mean matrices corresponding to all perception modules are stacked to obtain a second fluctuation measurement matrix.

[0073] In this way, the importance of each attention module and perception module in the initial large language model is accurately evaluated through the first fluctuation measure matrix and the second fluctuation measure matrix, and the parts that contribute less to the performance of the initial large language model are identified from a global perspective, which facilitates setting the global pruning threshold for the initial large language model based on the first fluctuation measure matrix and the second fluctuation measure matrix, and then implementing subsequent model pruning processing.

[0074] In an embodiment of the present application, determining a global pruning threshold according to a first fluctuation metric matrix and a second fluctuation metric matrix includes the following steps: (102.c.1) Combine the first volatility metric matrix and the second volatility metric matrix to obtain a global metric matrix.

[0075] (102.c.2) Construct an initial matrix having the same matrix size as the global metric matrix, the initial matrix including a plurality of initial elements.

[0076] (102.c.3) Based on a preset pruning rate, a preset cumulative function, and a global metric matrix, multiple initial elements are updated to obtain corresponding multiple updated elements.

[0077] (102.c.4) Determine a target update element with the smallest element value from the multiple update elements, and determine a first target position where the target update element is located.

[0078] (102.c.5) Determine a global pruning threshold based on a second target position that matches the first target position with the global metric matrix.

[0079] Steps (102.c.1) to (102.c.5) are described in detail below.

[0080] The global metric matrix may be obtained by first tiling and concatenating the matrix elements in the first volatility metric matrix and the second volatility metric matrix. A matrix element refers to a specific value at each position in the matrix. Of course, the manner of merging the first volatility metric matrix and the second volatility metric matrix may be specifically set according to actual conditions, and the embodiments of the present application do not limit this.

[0081] Furthermore, in order to determine which weight parameters of the initial large language model should be retained or pruned, it is necessary to construct an intermediate variable, namely the initial matrix, to help determine the global pruning threshold later. The initial matrix is ​​usually initialized to zero or other constant values ​​(initial elements), indicating that all weight parameters have not yet been evaluated or decided whether to prune. It should be noted that the essence of the initial element is also a matrix element, which is called the initial element here for the sake of distinction.

[0082] Furthermore, the matrix elements in the global metric matrix are sorted from small to large to obtain the second target element in the sorted global metric matrix and the second target position corresponding to the second target element. Next, the positions in the initial matrix that are less than the number of elements in the first fluctuation metric matrix are set to True, and the others are set to False, and the initial element value of the True position is set to 512.0 / 3 to obtain the first fluctuation metric matrix after the matrix elements are updated.

[0083] Furthermore, for the updated first volatility measure matrix, the following formula is used: <4> to <5> , update multiple initial elements and obtain corresponding multiple updated elements cw: <4> <5> Among them, pruning_ratio is the preset pruning rate; cumsum is the preset cumulative function; sum is the summation function; compression_weight is the initial element.

[0084] Furthermore, by the following formula <6> , determine the target update element with the smallest element value from multiple update elements, and determine the first target position pos where the target update element is located: <6> Among them, argmin is the minimum function, which is used to return the index of the minimum element in the array.

[0085] Furthermore, by the following formula <7> , determine the global pruning threshold: <7> Among them, sorted_prune is the matrix element in the updated first volatility metric matrix, that is, the first target position pos where the minimum target update element is located is determined from the multiple update elements of the initial matrix, and the matrix element at the corresponding second target position of the global metric matrix is ​​determined, which is the global pruning threshold.

[0086] Among them, the global pruning threshold is used to unify the pruning standard, which can lay the foundation for the subsequent determination of the mask matrices of the attention module and the perception module, respectively, and avoid the occurrence of local optimal solutions or inconsistent pruning decisions when performing pruning based on the mask matrix.

[0087] Step 103, based on the global pruning threshold and the first fluctuation metric matrix, determine the key value mask matrix and query mask matrix corresponding to each attention module, and based on the global pruning threshold and the second fluctuation metric matrix, determine the perception mask matrix corresponding to each perception module.

[0088] The following is a detailed description of step 103.

[0089] Among them, since the output dimensions of the matrices corresponding to each layer in the initial large language model attention module are not exactly the same, it is impossible to directly prune the matrices of the attention module and the perception module according to the global pruning threshold. Based on this, after obtaining the global pruning threshold, the embodiment of the present application determines the mask matrices of each attention module and each perception module based on the global pruning threshold, and each attention module includes two mask matrices with different output dimensions.

[0090] Furthermore, mask matrices of different output dimensions can ensure that the pruning operation is adapted to all processing layers of the initial large language model, so as to flexibly cope with the different output dimension requirements between layers in the attention modules of the initial large language model; the corresponding mask matrices are generated in a targeted manner according to the actual conditions of each attention module and each perception module in each processing layer, which can improve the pruning accuracy of the large language model while adapting to the special structure of the initial large language model, thereby improving the model compression accuracy of the large language model.

[0091] In an embodiment of the present application, based on the global pruning threshold and the first fluctuation metric matrix, determining the key value mask matrix and the query mask matrix corresponding to each attention module includes the following steps: (103.a.1) Based on the grouping parameters preset by the initial large language model, adjust the matrix shape of each first mean matrix to obtain an initial mask matrix.

[0092] (103.a.2) Update the matrix elements of the initial mask matrix according to the global pruning threshold to obtain the initial key-value mask matrix.

[0093] (103.a.3) Mask the matrix elements of the initial key-value mask matrix according to the preset local pruning threshold to obtain the key-value mask matrix corresponding to each attention module.

[0094] (103.a.4) Determine the number of attention heads to be pruned in any attention module based on the grouping parameter, and update the global pruning threshold based on the number of attention heads to obtain an updated global pruning threshold.

[0095] (103.a.5) Based on the updated global pruning threshold, the matrix elements of the initial mask matrix are masked to obtain the query mask matrix corresponding to each attention module.

[0096] Steps (103.a.1) to (103.a.5) are described in detail below.

[0097] In the embodiment of the present application, the attention mechanism of the initial large language model groups each query layer so that each group of query layers shares the same key weight matrix and value weight matrix, thereby reducing the amount of computation while maintaining more diversity. The parameter used to control the number of groups is the grouping parameter, which determines the actual output dimension size of each query weight matrix, key weight matrix, value weight matrix, and output weight matrix after grouping.

[0098] Furthermore, for the first mean matrix corresponding to each attention module, the grouping parameter is used to adjust its shape. Specifically, the shape of the first mean matrix (attn_metric_i) is adjusted to (num_head / num_key_value_group, num_key_value_group) to obtain the initial mask matrix (attn_metric_kv), where num_head is the number of attention heads and num_key_value_group is the grouping parameter.

[0099] Furthermore, the matrix elements in the initial mask matrix that are equal to or greater than the global pruning threshold are set to True, and the matrix elements in the initial mask matrix that are less than the global pruning threshold are set to False, to obtain an initial key-value mask matrix. In order to further refine the pruning operation of each attention module based on the global pruning strategy, it is also necessary to further update the initial key-value mask matrix according to the preset local pruning threshold. Among them, the local pruning threshold is user-defined, or it is determined autonomously by the initial large language model or other related models based on historical data or current data. The specific value of the local pruning threshold can be adaptively adjusted according to the actual situation, and the embodiment of the present application does not limit this.

[0100] Specifically, each row of the initial mask matrix is ​​summed up to obtain a one-dimensional array of size num_key_value_group, which represents its importance measure; then, the matrix elements in the one-dimensional array are masked based on the local pruning threshold to obtain a key-value mask matrix.

[0101] Furthermore, since the output dimensions of the query layer and the key-value layer are different, the global pruning threshold is updated by dynamically adjusting the global pruning threshold, and the query mask matrix is ​​determined by updating the global pruning threshold. In this way, although the dimensions of each layer in the attention module of the initial large language model are different, the structured pruning method proposed in the embodiment of the present application generates respective independent mask matrices respectively, ensuring that each module performs consistent pruning operations under its respective dimensions, so that the pruning strategy is better adapted to the model structure of LLaMA3, thereby improving the pruning refinement of the initial large language model, and then improving the pruning accuracy of the initial large language model.

[0102] Among them, the key-value mask matrix represents whether the binary matrix in the initial key matrix and initial value matrix to be pruned in each attention module needs to be pruned (0 means pruning, 1 means keeping). The query mask matrix represents whether the binary matrix in the initial query matrix to be pruned in each attention module needs to be pruned. The number of pruned attention heads represents the number of attention heads removed after pruning, which is used to dynamically adjust the global pruning threshold to obtain the adjusted updated global pruning threshold.

[0103] In an embodiment of the present application, according to a preset local pruning threshold, the matrix elements of the initial key-value mask matrix are masked to obtain the key-value mask matrix corresponding to each attention module, including the following steps: (B.1) Sum all matrix elements of the initial key-value mask matrix to obtain the actual total value.

[0104] (B.2) If the actual total value does not meet the preset total value, the initial key-value mask matrix is ​​masked using the first local pruning threshold to obtain the key-value mask matrix corresponding to each attention module.

[0105] (B.3) If the actual total value meets the preset total value, the initial key-value mask matrix is ​​masked using the second local pruning threshold to obtain the key-value mask matrix corresponding to each attention module.

[0106] Steps (B.1) to (B.3) are described in detail below.

[0107] In an embodiment of the present application, the local pruning threshold includes a first local pruning threshold and a second local pruning threshold. The preset total value can be set to 0, so that for the current attention module, if the sum of all matrix elements in the initial key-value mask matrix is ​​not 0, the matrix elements equal to or greater than the first local pruning threshold are set to 1, and the matrix elements less than the first local pruning threshold are set to 0, and the key-value mask matrix is ​​obtained.

[0108] Alternatively, if the sum of all matrix elements in the initial key-value mask matrix is ​​0, the matrix elements equal to or greater than the second local pruning threshold are set to 1, and the matrix elements less than the second local pruning threshold are set to 0, to obtain the key-value mask matrix.

[0109] It should be noted that the preset total value, the first local pruning threshold and the second local pruning threshold can all be set according to actual conditions. This is only an example and does not limit the embodiments of the present application.

[0110] In an embodiment of the present application, updating the global pruning threshold based on the number of attention heads to obtain an updated global pruning threshold includes the following steps: (C.1) Sort the matrix elements in the first mean matrix to obtain an updated first mean matrix.

[0111] (C.2) Determine the matrix element at the third target position in the updated first mean matrix that matches the number of attention heads as the updated global pruning threshold.

[0112] Steps (C.1) to (C.2) are described in detail below.

[0113] In the present embodiment, firstly, the following formula <8> Determine the number of attention groups num_pruned_groups that are pruned for any attention module: num_pruned_groups=num_key_value_group-sum(attn_metric_kv) <8> Among them, attn_metric_kv is the initial key-value mask matrix; num_key_value_group is the grouping parameter.

[0114] Furthermore, by the following formula <9> to <10> Determine the number of pruned attention heads for any attention module, pruned_num_heads: group_size=head_num / / num_key_value_group <9> <10> Among them, head_num represents the total number of attention heads in the current attention module; / / represents integer divisibility.

[0115] Furthermore, after determining the number of attention heads, the matrix elements in the first mean matrix can be sorted from small to large to obtain the updated first mean matrix. Then, by the following formula <11> Determine and update the global pruning threshold threshold': threshold'=sort[pruned_num_heads] <11> Among them, sort is a sorting function; that is, it is determined that in the first mean matrix after sorting and updating, the matrix element at the third target position that matches the number of pruned attention heads, pruned_num_heads, is the updated global pruning threshold.

[0116] Step 104, using the key-value mask matrix and the query mask matrix of each attention module, pruning the corresponding first weight matrix to obtain the first target matrix, and using the perception mask matrix of each perception module, pruning the corresponding second weight matrix to obtain the second target matrix.

[0117] Step 104 is described in detail below.

[0118] In an embodiment of the present application, for each attention module of the initial large language model, the number of output channels of the initial key matrix and the initial value matrix is ​​pruned using a key-value mask matrix; the number of output channels of the initial query matrix is ​​pruned using a query mask matrix, and the number of input channels of the first output matrix is ​​pruned using the query mask matrix; and then the first target matrix is ​​obtained after the first weight matrix is ​​pruned.

[0119] Furthermore, for each perception module of the initial large language model, the number of input channels of the mapping matrix is ​​pruned using the perception mask matrix, and the number of output channels of the second output matrix is ​​pruned using the perception mask matrix; thereby obtaining a second target matrix after the second weight matrix is ​​pruned.

[0120] Step 105: Determine a large language model after pruning the initial large language model based on the first target matrix and the second target matrix.

[0121] The following is a detailed description of step 105.

[0122] Among them, the large language model after pruning the initial large language model can be obtained by replacing the original first weight matrix with the first target matrix and replacing the original second weight matrix with the second target matrix, thereby achieving high-accuracy model compression of the initial large language model.

[0123] In the embodiment of the present application, after determining the large language model after the initial large language model is pruned, the following steps are also included: (105.a.1) Obtain evaluation samples and validation samples that represent the desired output.

[0124] (105.a.2) Input the evaluation sample into the large language model, embed the evaluation sample based on the first target matrix and the second target matrix, and obtain the actual sample representing the actual output.

[0125] (105.a.3) Calculate the sample difference between the actual sample and the verification sample, and prune the large language model based on the sample difference to obtain the pruned and adjusted large language model.

[0126] Steps (105.a.1) to (105.a.3) are described in detail below.

[0127] In an embodiment of the present application, in order to verify the model performance of the large language model obtained after pruning, the output result after processing the input evaluation sample based on the large language model can be compared with the preset expected value; if the difference between the two is within the expected range, it means that the pruned large language model meets the model compression requirements, otherwise, it is necessary to continue to prune and adjust the large language model until a large language model that meets the requirements is obtained.

[0128] The evaluation samples are samples used to test and evaluate the performance of the large language model. The evaluation samples can be used as input data to verify the performance of the pruned large language model in practical applications. The specific forms of the evaluation samples include but are not limited to text, images, etc., depending on the task type of the large language model. The verification samples represent the expected output results of the evaluation samples after being processed by the large language model.

[0129] The sample difference can be determined by calculating the mean square error, cross entropy loss or cosine similarity between the actual sample and the verification sample, and the specific method of determining the sample difference can be set according to the actual situation. In addition, the threshold used for comparison with the sample difference to determine whether the large language model needs further pruning and adjustment can also be set according to the actual situation, and the embodiment of the present application does not limit this.

[0130] The pruning adjustment may be to adjust the pruning rate, or the specific manner of the pruning adjustment may be set according to actual conditions, which is not limited in the embodiments of the present application.

[0131] like Figure 3 , Figure 4 As shown, Figure 3 is another optional flowchart of the structured pruning method for a large language model provided in an embodiment of the present application. Figure 4 This is another optional flow chart of the structured pruning method for a large language model provided in the embodiment of the present application. In order to make the reader more clearly understand the structured pruning method proposed in the embodiment of the present application, the following will take LLaMA3 as an example, combined with Figure 3 , Figure 4 A complete example is given below.

[0132] (1) Load the LLaMA3 large model, where LLaMA3 includes multiple processing layers, each of which includes a set of interconnected attention (Attention) modules and perception (MLP) modules; the Attention module includes multiple attention heads, each of which includes a key layer (k_proj), a value layer (v_proj), a query layer (q_proj), the first output layer (o_proj) and its corresponding weight matrix.

[0133] (2) Obtain the initial sample and perform the following operations on each processing layer of the LLaMA3 model: Process the initial sample based on the Attention module and MLP module of the LLaMA3 model, and then calculate the fluctuation metric attn_metric of the attention module (Attention) of the current processing layer and the fluctuation metric mlp_metric of the perception module (MLP) of the current processing layer.

[0134] (3) The fluctuation metric attn_metric of the Attention module is stored in the attn_metric_list linked list, and the fluctuation metric mlp_metric of the MLP module is stored in the mlp_metric_list linked list; and the input sample mean attn_mean obtained after the initial sample is processed by the Attention module is stored in the attn_mean_list linked list, and the input sample mean mlp_mean obtained after the initial sample is processed by the MLP module is stored in the mlp_mean_list linked list, so that the relevant matrix elements can be updated based on the attn_mean_list linked list and the mlp_mean_list linked list.

[0135] (4) After the fluctuation metrics of all processing layers are calculated, all the metric vectors in attn_metric_list are stacked into the first fluctuation metric matrix attn_metric; the metric vectors of all layers in mlp_metric_list are stacked into the second fluctuation metric matrix mlp_metric.

[0136] (5) Perform normalization processing on the first volatility metric matrix attn_metric and the second volatility metric matrix mlp_metric respectively; and adjust the shape of the attn_metric matrix after normalization processing to (nlayers, head_num, 128), where nlayers represents the number of processing layers, head_num is the number of attention heads, for example, for the LLaMA3-70B model, head_num=64; 128 represents the dimension of each attention head (head); it should be noted that the specific shape of the attn_metric matrix after adjustment can be set according to actual conditions.

[0137] (6) Average the second dimension of the attn_metric matrix (counting from 0) to obtain the importance score of each head (still named attn_metric), whose shape is (nlayers, head_num). This new attn_metric will be used as the input of the get_layer_mask function.

[0138] (7) Call the get_layer_mask function to calculate three mask matrices: the key-value mask matrix (attn_mask_kv), the query mask matrix (attn_mask_q), and the perception mask matrix (mlp_mask). Compared with the traditional structured pruning method that uses the same mask matrix, the embodiment of the present application determines the independent mask matrix of each processing layer separately to better adapt to the special structure of LLaMA3, thereby refining the pruning precision and improving the pruning accuracy of LLaMA3.

[0139] (8) For each processing layer, the compress function is called based on attn_mask_q and attn_mask_kv to prune the weight matrices of the q_proj layer, k_proj layer, v_proj layer, and o_proj layer: For the Attention module of the current processing layer, use attn_mask_q to prune the weight matrices of the q_proj layer and the o_proj layer, and use attn_mask_kv to prune the weight matrices of the k_proj layer and the v_proj layer. Specifically, use attn_mask_q to prune the number of output channels of the weight matrix of the q_proj layer, and use attn_mask_kv to prune the number of output channels of the k_proj layer and the v_proj layer: update the number of output channels out_features of the q_proj layer to the number of non-zero elements of attn_mask_q; and update the number of output channels out_features of the k_proj layer and the v_proj layer to the number of non-zero elements of attn_mask_kv. Use attn_mask_q to prune the number of input channels of the o_proj layer, update the number of input channels of the o_proj layer to the number of non-zero elements of attn_mask_q, and update the compensation value vector of the bias of the o_proj layer.

[0140] For the MLP module of the current processing layer, use mlp_mask to prune the number of input channels of the up_proj layer and gate_proj layer weight matrices of the MLP module, and prune the number of output channels of the down_proj layer weight matrix; update the compensation value vector of the down_proj layer bias, synchronously update the number of output channels out_features of the up_proj layer and gate_proj layer, and update the parameters of the intermediate hidden layer of the MLP module to the number of non-zero elements of mlp_mask.

[0141] (9) Based on the obtained evaluation samples, the large language model obtained after pruning is verified. When the verification passes, the corresponding large language model is saved.

[0142] Furthermore, if Figure 4 As shown, step (7) specifically also includes the following steps: (7.1) Flatten the attn_metric and mlp_metric matrices and concatenate them to obtain the global metric matrix prune_metric.

[0143] (7.2) Sort the matrix vectors in prune_metric from large to small to obtain the sorted vector sorted_prune and the corresponding index vector indices of each vector.

[0144] (7.3) Calculate the global pruning threshold threshold: Initialize a vector compression_weight, which is the same size as indices and all values ​​are 1. Set the positions in indices that are less than the number of attn_metric elements to True, and the others to False, to obtain a bool vector indices_b; set the value of compression_weight in the True position in indices_b to 512.0 / 3. Then, calculate the threshold based on compression_weight and the preset pruning rate pruning_ratio.

[0145] (7.4) Calculate the mask matrix mlp_mask of the MLP module: set the positions where the value of mlp_metric is greater than threshold to True, and set the others to False. At this point, the mask matrix mlp_mask corresponding to the MLP module is determined.

[0146] (7.5) Calculate the mask matrices attn_mask_q and attn_mask_kv of the Attention module based on attn_metric and threshold. The shape of attn_metric is [nlayers, head_num], which stores the head importance scores of all processing layers. For each layer of attn_metric, perform the following processing on attn_metric_i to obtain attn_mask_q and attn_mask_kv: Furthermore, step (7.5) specifically includes the following steps: (7.5.1) Adjust the shape of attn_metric_i to (num_head / num_key_value_group, num_key_value_group) to get the new matrix attn_metric_kv; here, multiple heads are divided into num_key_value_group groups, and the num_key_value_group value is a parameter of the Attention module.

[0147] (7.5.2) Set attn_metric_kv to True where it is greater than threshold, and set attn_metric_kv to False where it is less than or equal to threshold.

[0148] (7.5.3) Sum each row of attn_metric_kv to obtain a one-dimensional array of size num_key_value_group that represents the importance metric; if the sum of all elements of attn_mask_kv is not 0, set the values ​​in the one-dimensional array that are greater than or equal to the first local pruning threshold (group_prune_thr1) to 1, and set the values ​​that are less than group_prune_thr1 to 0, to obtain the mask matrix attn_mask_kv.

[0149] If the sum of all elements of attn_mask_kv is 0, attn_mask_kv is recalculated using the second local pruning threshold (group_prune_thr2) to avoid pruning all attention groups. In an optional embodiment, group_prune_thr1 and group_prune_thr2 can be set to 0.25 and 0, respectively.

[0150] (7.5.4) Determine the updated global pruning threshold threshold' based on threshold.

[0151] (7.5.5) Calculate the mask vector attn_mask_q: Set the values ​​greater than threshold in attn_metric_i (referring to the initial attn_metric_i, unsorted) to True, and the others to False, to obtain the mask vector attn_mask_q.

[0152] (7.5.6) Return mlp_mask, attn_mask_kv, and attn_mask_q.

[0153] It should be noted that the part involving formula calculation in this example will not be described in detail, and specific reference may be made to steps 101 to 105 .

[0154] like Figure 5 As shown, Figure 5 This is an optional module diagram of a structured pruning device for a large language model provided in an embodiment of the present application, including: The acquisition module 201 is used to obtain the initial large language model to be pruned, and obtain the corresponding first weight matrix from multiple attention modules of the initial large language model, and obtain the corresponding second weight matrix from multiple perception modules of the initial large language model.

[0155] The volatility metric matrix determination module 202 is used to determine a first volatility metric matrix based on a first weight matrix, determine a second volatility metric matrix based on a second weight matrix, and determine a global pruning threshold based on the first volatility metric matrix and the second volatility metric matrix.

[0156] The global pruning threshold determination module 203 is used to determine the key value mask matrix and query mask matrix corresponding to each attention module based on the global pruning threshold and the first fluctuation measurement matrix, and to determine the perception mask matrix corresponding to each perception module based on the global pruning threshold and the second fluctuation measurement matrix.

[0157] The mask matrix determination module 204 is used to use the key-value mask matrix and the query mask matrix of each attention module to prune the corresponding first weight matrix to obtain the first target matrix, and use the perception mask matrix of each perception module to prune the corresponding second weight matrix to obtain the second target matrix.

[0158] The target module 205 determines a large language model after pruning the initial large language model based on the first target matrix and the second target matrix.

[0159] The present application proposes a structured pruning method for a large language model and related equipment, the method comprising: obtaining an initial large language model to be pruned, and obtaining a corresponding first weight matrix from multiple attention modules of the initial large language model, and obtaining a corresponding second weight matrix from multiple perception modules of the initial large language model; determining a first fluctuation measurement matrix based on the first weight matrix, determining a second fluctuation measurement matrix based on the second weight matrix, and determining a global pruning threshold based on the first fluctuation measurement matrix and the second fluctuation measurement matrix; determining a key value mask matrix and a query mask matrix corresponding to each attention module based on the global pruning threshold and the first fluctuation measurement matrix, and ... The second fluctuation measurement matrix determines the corresponding perceptual mask matrix of each perception module; the mask matrices of different output dimensions can ensure that the pruning operation is adapted to all processing layers of the initial large language model, so as to flexibly cope with the different output dimensions of each layer in each attention module of the initial large language model; then, the key value mask matrix and the query mask matrix of each attention module are used to prune the corresponding first weight matrix to obtain the first target matrix, and the perceptual mask matrix of each perception module is used to prune the corresponding second weight matrix to obtain the second target matrix; based on the first target matrix and the second target matrix, the large language model after the initial large language model is pruned is determined. In this way, the corresponding mask matrix is ​​generated in a targeted manner according to the actual conditions of each attention module and each perception module in each processing layer, which can improve the model compression accuracy of the large language model while adapting to the special structure of the initial large language model.

[0160] The specific implementation of the structured pruning device is basically the same as the specific embodiment of the structured pruning method described above, and will not be repeated here.

[0161] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above structured pruning method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0162] like Figure 6 As shown, Figure 6 : is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application, the electronic device includes: The processor 301 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 302 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 302 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 302, and the processor 301 calls and executes the structured pruning method of the embodiment of this application; Input / output interface 303, used to implement information input and output; Communication interface 304, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.); A bus 305 that transmits information between the various components of the device (e.g., the processor 301, the memory 302, the input / output interface 303, and the communication interface 304); The processor 301 , the memory 302 , the input / output interface 303 and the communication interface 304 are connected to each other in communication within the device via the bus 305 .

[0163] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned structured pruning method is implemented.

[0164] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0165] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0166] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0167] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0168] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0169] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0170] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0171] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0172] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0174] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0175] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A structured pruning method for a large language model, characterized in that: include: Obtain an initial large language model to be pruned, and obtain a corresponding first weight matrix from multiple attention modules of the initial large language model, and obtain a corresponding second weight matrix from multiple perception modules of the initial large language model; Determine a first volatility measurement matrix based on the first weight matrix, determine a second volatility measurement matrix based on the second weight matrix, and determine a global pruning threshold according to the first volatility measurement matrix and the second volatility measurement matrix; Based on the global pruning threshold and the first fluctuation metric matrix, determining a key value mask matrix and a query mask matrix corresponding to each of the attention modules, and based on the global pruning threshold and the second fluctuation metric matrix, determining a perception mask matrix corresponding to each of the perception modules; Using the key-value mask matrix and the query mask matrix of each of the attention modules, pruning the corresponding first weight matrix to obtain a first target matrix, and using the perception mask matrix of each of the perception modules, pruning the corresponding second weight matrix to obtain a second target matrix; A large language model after pruning the initial large language model is determined based on the first target matrix and the second target matrix.

2. The structured pruning method for a large language model according to claim 1, characterized in that: Each of the attention modules includes a plurality of first weight matrices, each of the first weight matrices includes an initial key matrix, an initial query matrix, an initial value matrix and a first output matrix, and output dimensions of the initial key matrix, the initial query matrix, the initial value matrix and the first output matrix are not completely the same; The determining a first volatility measurement matrix based on the first weight matrix comprises: Acquire an initial sample, and input the initial sample into the initial large language model; For each of the first weight matrices, embedding processing is performed on the initial samples based on the initial key matrix, the initial query matrix, the initial value matrix and the first output matrix in sequence to obtain corresponding first output samples; Determine a first original matrix based on the first output sample, and perform mean processing on multiple first original matrices belonging to the same attention module to obtain a first mean matrix; The first mean matrices corresponding to all the attention modules are stacked to obtain a first fluctuation measure matrix.

3. The structured pruning method for a large language model according to claim 2, characterized in that: The step of performing mean processing on a plurality of the first original matrices belonging to the same attention module to obtain a first mean matrix includes: Performing mean processing on matrix elements in each of the first original matrices respectively to obtain a corresponding metric mean of each of the first original matrices; Based on the matrix elements in each of the first original matrices, calculate and obtain the metric standard difference value corresponding to each of the first original matrices; Based on the metric mean and the metric standard deviation, update the matrix elements in the corresponding first original matrix to obtain a first normalized matrix; Perform mean processing on multiple first normalized matrices belonging to the same attention module to obtain a first mean matrix.

4. The structured pruning method for a large language model according to claim 2, characterized in that: Each of the attention modules is connected to the corresponding perception module, each of the perception modules includes a plurality of the second weight matrices, and each of the second weight matrices includes a mapping matrix and a second output matrix; The determining a second volatility measurement matrix based on the second weight matrix comprises: For each of the first weight matrices, embedding processing is performed on the first output samples based on the mapping matrix and the second output matrix in sequence to obtain corresponding second output samples; Determine a second original matrix based on the second output sample, and perform mean processing on a plurality of the second original matrices belonging to the same perception module to obtain a second mean matrix; The second mean matrices corresponding to all the perception modules are stacked to obtain a second fluctuation measurement matrix.

5. The structured pruning method for a large language model according to claim 1, characterized in that: The determining of a global pruning threshold according to the first fluctuation metric matrix and the second fluctuation metric matrix comprises: Combining the first volatility metric matrix and the second volatility metric matrix to obtain a global metric matrix; Constructing an initial matrix having the same matrix size as the global metric matrix, wherein the initial matrix includes a plurality of initial elements; Based on a preset pruning rate, a preset accumulation function and the global metric matrix, updating the plurality of initial elements to obtain a corresponding plurality of updated elements; Determine a target update element with the smallest element value from the multiple update elements, and determine a first target position where the target update element is located; The global pruning threshold is determined based on a second target position that matches the global metric matrix with the first target position.

6. The structured pruning method for a large language model according to claim 2, characterized in that: The determining, based on the global pruning threshold and the first fluctuation metric matrix, a key value mask matrix and a query mask matrix corresponding to each of the attention modules comprises: Based on the grouping parameters preset in the initial large language model, adjusting the matrix shape of each of the first mean matrices to obtain an initial mask matrix; According to the global pruning threshold, updating the matrix elements of the initial mask matrix to obtain an initial key-value mask matrix; According to a preset local pruning threshold, masking is performed on the matrix elements of the initial key-value mask matrix to obtain the key-value mask matrix corresponding to each of the attention modules; Determine the number of pruned attention heads of any of the attention modules based on the grouping parameters, and update the global pruning threshold based on the number of attention heads to obtain an updated global pruning threshold; Based on the updated global pruning threshold, the matrix elements of the initial mask matrix are masked to obtain the query mask matrix corresponding to each attention module.

7. The structured pruning method for a large language model according to claim 6, characterized in that: The local pruning threshold includes a first local pruning threshold and a second local pruning threshold; The masking process is performed on the matrix elements of the initial key-value mask matrix according to the preset local pruning threshold to obtain the key-value mask matrix corresponding to each attention module, including: Summing all matrix elements of the initial key value mask matrix to obtain an actual total value; If the actual total value does not meet the preset total value, masking the initial key-value mask matrix using the first local pruning threshold to obtain the key-value mask matrix corresponding to each attention module; If the actual total value meets the preset total value, the initial key-value mask matrix is ​​masked using the second local pruning threshold to obtain the key-value mask matrix corresponding to each attention module.

8. The structured pruning method for a large language model according to claim 6, characterized in that: The updating of the global pruning threshold based on the number of attention heads to obtain an updated global pruning threshold includes: Sorting each matrix element in the first mean matrix to obtain an updated first mean matrix; Determine in the updated first mean matrix that the matrix element at the third target position that matches the number of attention heads is the updated global pruning threshold.

9. The structured pruning method for a large language model according to claim 7, characterized in that: After determining the large language model after pruning the initial large language model, the method further includes: Obtain evaluation samples and validation samples that represent the expected output; Inputting the evaluation sample into the large language model, and performing embedding processing on the evaluation sample based on the first target matrix and the second target matrix to obtain an actual sample representing an actual output; A sample difference between the actual sample and the verification sample is calculated, and the large language model is pruned and adjusted based on the sample difference to obtain the pruned and adjusted large language model.

10. A structured pruning device for a large language model, characterized in that: include: An acquisition module, used to acquire an initial large language model to be pruned, and acquire a corresponding first weight matrix from multiple attention modules of the initial large language model, and acquire a corresponding second weight matrix from multiple perception modules of the initial large language model; a fluctuation metric matrix determination module, configured to determine a first fluctuation metric matrix based on the first weight matrix, determine a second fluctuation metric matrix based on the second weight matrix, and determine a global pruning threshold according to the first fluctuation metric matrix and the second fluctuation metric matrix; A global pruning threshold determination module, used to determine a key value mask matrix and a query mask matrix corresponding to each of the attention modules based on the global pruning threshold and the first fluctuation metric matrix, and to determine a perception mask matrix corresponding to each of the perception modules based on the global pruning threshold and the second fluctuation metric matrix; A mask matrix determination module, configured to use the key-value mask matrix and the query mask matrix of each of the attention modules to perform pruning processing on the corresponding first weight matrix to obtain a first target matrix, and use the perception mask matrix of each of the perception modules to perform pruning processing on the corresponding second weight matrix to obtain a second target matrix; The target module determines a large language model after pruning the initial large language model based on the first target matrix and the second target matrix.

11. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the structured pruning method for a large language model according to any one of claims 1 to 9 when executing the computer program.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the structured pruning method for a large language model according to any one of claims 1 to 9 is implemented.

Images