Model compression method and device and electronic equipment

The large language model is quantized and optimized through pseudo-quantization nodes and pruning technology, and combined with the SwiGLU activation function, the parameter quantity and calculation complexity problems during compression of the large language model are solved, and the model performance and accuracy are maintained.

CN119089963APending Publication Date: 2024-12-06CHINA TELECOM CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411117996.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The prior art is difficult to reasonably determine the compression methods and compression parameters of large language models, resulting in poor model compression effect.

Method used

The floating-point weight of the initial model is quantized by using a pseudo-quantitative node, and the attention heads in the multi-head attention module that are below the preset threshold are discarded. Combined with quantization and pruning technology, the ReLU activation function and the full connection layer are replaced. SwiGLU activates the function to enhance nonlinear expression.

Benefits of technology

It realizes the reduction of model parameters and calculation complexity, while maintaining the performance and accuracy of the model, solving the problem that the compression method and parameters cannot be reasonably determined during compression processing of large language models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089963A_ABST
    Figure CN119089963A_ABST
Patent Text Reader

Abstract

The invention discloses a model compression method and device and electronic equipment. The method comprises the steps that an initial model is obtained, the initial model comprises a plurality of Transform modules, and the initial model is a large language model with the weight existing in a floating point mode; performing quantization operation on the floating point type weight of the initial model by adopting a pseudo quantization node to obtain a first model, the pseudo quantization node being used for simulating the quantization effect of the initial model; and determining a multi-head attention module in the first model Transform module, and discarding attention heads lower than a preset threshold in the multi-head attention module to obtain a compression model. According to the method and the device, the technical problem that a compression method and compression parameters cannot be reasonably determined when a large language model is compressed in the prior art is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of model compression, and in particular to a model compression method, device and electronic equipment. Background Art

[0002] With the iterative optimization of neural network models, the number of model parameters is also increasing. Taking chatGPT as an example, the number of parameters of this neural network model has jumped from 170 million in the initial GPT1 version to 175 billion in GPT3 released in 2020. The surge in model parameters has put great pressure on the computing, storage, network transmission and other resources required for large neural network models. Therefore, it is necessary to compress large neural network models and optimize and accelerate the computing and storage resources on the end side.

[0003] For a branch of large neural network models, the Large Language Model (LLM), researchers have tried to compress the Transformer module in it using the corresponding model compression method after it was open sourced. The reason why the neural network large language model can be compressed is that it can use the redundancy of network parameters and network structure to reduce the scale of parameters, reconstruct, simplify and accelerate the deep learning model, and obtain a model with fewer parameters and a more streamlined structure without affecting the completion of the original task.

[0004] In related technologies, commonly used model compression methods include pruning, quantization, knowledge distillation, low-rank decomposition, etc. However, there is currently no reasonable and definite way to select the corresponding compression method and compression parameters when compressing large neural network language models.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present application provide a model compression method, device and electronic device to at least solve the technical problem that the related technology cannot reasonably determine the compression method and compression parameters when compressing a large language model.

[0007] According to one aspect of an embodiment of the present application, a model compression method is provided, including: obtaining an initial model, wherein the initial model includes multiple Transformer modules and the initial model is a large language model whose weights are in floating-point form; using a pseudo-quantization node to quantize the floating-point weights of the initial model to obtain a first model, wherein the pseudo-quantization node is used to simulate the quantization effect of the initial model; determining a multi-head attention module in the Transformer module of the first model, and discarding the attention heads in the multi-head attention module that are lower than a preset threshold to obtain a compressed model.

[0008] Optionally, the method further includes: replacing the ReLU activation function of the forward feedback layer and the fully connected layer in the Transformer module with the SwiGLU activation function, wherein the SwiGLU activation function is used to enhance the nonlinear expression capability of the initial model.

[0009] Optionally, before using the pseudo-quantization node to perform a quantization operation on the floating-point weights of the initial model, the method also includes: determining the output distribution corresponding to the SwiGLU activation function, and discarding the output values ​​in the output distribution that exceed a preset distribution threshold; and determining the quantization threshold of the initial model based on the output distribution, wherein the quantization threshold is used to represent the quantization range when performing a quantization operation on the initial model.

[0010] Optionally, before using the pseudo-quantization node to perform a quantization operation on the floating-point weights of the initial model, the method also includes: determining the input data of the batch normalization layer in the Transformer module, and dividing the input data into multiple batches to obtain batch data; determining a first mean and a first variance of the batch data, wherein the first mean and the first variance are used to maintain the stability of the input data in the batch normalization layer; updating the first mean and the first variance to obtain a target mean and a target variance; and determining the parameters of the batch normalization layer based on the target mean and the target variance.

[0011] Optionally, the quantization operation includes quantization and inverse quantization; a pseudo-quantization node is used to perform a quantization operation on the floating-point weights of the initial model, including: determining a floating-point number corresponding to the floating-point weight, and determining a quantization scaling factor and an inverse quantization approximation value when performing a quantization operation on the initial model; and a quantization operation is performed on the floating-point weight based on the pseudo-quantization node, the floating-point number, the quantization scaling factor and the inverse quantization approximation value.

[0012] Optionally, attention heads in a multi-head attention module that are below a preset threshold are discarded, including: obtaining a weight matrix of all attention heads in the multi-head attention module; determining an importance score of each attention head based on the weight matrix, wherein the importance score is used to represent the contribution of each attention head in the first model; sorting the importance scores according to the scores to obtain an importance score sequence; and discarding attention heads in the importance score sequence that have importance scores that are below the preset threshold.

[0013] Optionally, determining the importance score of each attention head based on the weight matrix includes: obtaining a query weight matrix, a key weight matrix, and a value weight matrix in the weight matrix; determining a first Euclidean norm corresponding to the query weight matrix, a second Euclidean norm corresponding to the key weight matrix, and a third Euclidean norm corresponding to the value weight matrix, wherein the first Euclidean norm is used to represent the size of the query weight matrix, the second Euclidean norm is used to represent the size of the key weight matrix, and the third Euclidean norm is used to represent the size of the value weight matrix; determining the importance score based on the first Euclidean norm, the second Euclidean norm, and the third Euclidean norm.

[0014] According to another aspect of an embodiment of the present application, a model compression device is also provided, including: an acquisition module, used to acquire an initial model, wherein the initial model includes multiple Transformer modules, and the initial model is a large language model whose weights are in floating-point form; a quantization module, used to use a pseudo-quantization node to perform a quantization operation on the floating-point weights of the initial model to obtain a first model, wherein the pseudo-quantization node is used to simulate the quantization effect of the initial model; a discarding module, used to determine a multi-head attention module in the Transformer module of the first model, and discard the attention heads in the multi-head attention module that are lower than a preset threshold to obtain a compressed model.

[0015] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned model compression method.

[0016] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned model compression method by running the computer program.

[0017] According to another aspect of the embodiments of the present application, a computer program product is also provided, including computer instructions, which implement the above-mentioned model compression method when executed by a processor.

[0018] In an embodiment of the present application, an initial model is obtained, wherein the initial model includes multiple Transformer modules, and the initial model is a large language model whose weights are in floating-point form; a pseudo-quantization node is used to quantize the floating-point weights of the initial model to obtain a first model, wherein the pseudo-quantization node is used to simulate the quantization effect of the initial model; a multi-head attention module in the Transformer module of the first model is determined, and the attention heads in the multi-head attention module that are lower than a preset threshold are discarded to obtain a compressed model, thereby achieving the purpose of compressing and optimizing the large language model by combining quantization and pruning technology, thereby achieving the technical effect of reducing the number of parameters and computational complexity of the model while maintaining the performance and accuracy of the model, thereby solving the technical problem that the compression method and compression parameters cannot be reasonably determined when the related technology compresses the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a hardware structure diagram of a computer terminal for implementing a model compression method according to an embodiment of the present application;

[0021] Figure 2 is a flow chart of a model compression method according to an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a forward feedback network structure according to an embodiment of the present application;

[0023] Figure 4 It is a structural diagram of a model compression device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] First, some nouns or terms that appear in the process of explaining the embodiments of the present application are subject to the following explanations:

[0027] Large models: Machine learning models with huge parameter scale and complex computational structure paradigm.

[0028] Large language models: usually natural language processing models with large-scale parameters and computing power, such as OpenAI's GPT-3 model.

[0029] Model compression: Reduce the size of the model and the number of parameters without changing the model effect as much as possible, so that the model can have a faster reasoning speed when used.

[0030] Model quantization: A model compression technology that can convert floating-point operations in neural network inference operations into fixed-point operations, which can reduce the model size but will introduce quantization errors.

[0031] Neural network pruning: A model compression technology that aims to remove redundant information in the model, reduce the number of model parameters and computing requirements, and is achieved by deleting weights, convolution kernels or neurons in the neural network model.

[0032] In order to solve the problem of poor model compression capability in the related art, the present application embodiment provides a model compression method, which can be run on Figure 1 In the computer terminal shown, the computer terminal is described below.

[0033] The model compression method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a model compression method. Figure 1As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0034] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0035] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the model compression method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned model compression method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0036] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0037] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0038] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0039] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a model compression method. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] Figure 2 is a flow chart of a model compression method according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0041] Step S202, obtaining an initial model, wherein the initial model includes multiple Transformer modules, and the initial model is a large language model whose weights are in floating point form.

[0042] In the above step S202, the starting point for large language model compression is to construct an initial model including multiple Transformer modules. The initial model is a large-scale language model whose weights are in the form of floating-point numbers, indicating that the initial model has a high-precision mathematical representation and is suitable for various complex computing tasks.

[0043] In the embodiment of the present application, before building the initial model, it is necessary to collect and obtain a high-dimensional, large-scale data set to provide rich language samples for training the initial model. For example, the Natural Questions open source data set is selected, which is a real-world question data set containing 300,000 questions asked by real users in the Google search engine, and the questions and answers are manually annotated.

[0044] Secondly, the collected data set is cleaned, for example: redundant data is removed, dirty (abnormal or erroneous) data is preliminarily counted and cleaned; missing value data is filled, features with a missing rate higher than 30% are deleted, and data with a missing rate lower than 30% are filled with the median; outliers are deleted or modified to normalized data, etc.

[0045] It should be noted that when applying the initial model, the user's query data (such as the aforementioned questions, source language, and input content, etc.) is generally required to be input into the initial model, and the user's query data generally involves user privacy data. The data needs to be processed accordingly here, such as anonymization or de-identification, to protect user privacy.

[0046] Step S204, using a pseudo quantization node to perform a quantization operation on the floating-point weights of the initial model to obtain a first model, wherein the pseudo quantization node is used to simulate the quantization effect of the initial model.

[0047] In the above step S204, quantization processing is a key step in model optimization, which aims to convert the floating-point weights of the model into low-precision representations to reduce the memory usage of the model and accelerate the reasoning process. By introducing pseudo-quantization nodes, the initial model can simulate the effect of quantization operations without actually changing the weight data type, allowing the initial model to adapt to the possible impact of quantization during training, and prepare for subsequent actual quantization. Among them, the pseudo-quantization node is used to quantize and dequantize the weights and features of the convolutional layer before performing convolution calculations on the weights of the convolutional layer and the features of the convolutional layer input, thereby simulating the loss of quantization and allowing the neural network model to learn part of the quantization loss. For example, the weights are quantized from floating point to 8-bit integers, and then dequantized from 8-bit integers to floating point numbers.

[0048] Step S206, determine the multi-head attention module in the Transformer module of the first model, and discard the attention heads in the multi-head attention module that are below a preset threshold to obtain a compressed model.

[0049] In the above step S206, the focus is on the multi-head attention mechanism in the Transformer module, and by evaluating the importance of each attention head, those attention heads that have little impact on model performance are removed. This process is pruning, which is another model compression technology in the embodiment of the present application that is different from quantization. It reduces the complexity and size of the model by removing unimportant parts of the model. This process can also be implemented through a refined scoring system to ensure that the pruning operation is selective and minimizes the negative impact on model performance.

[0050] Through the above steps S202 to S206, the purpose of combining quantization and pruning technology to compress and optimize the large language model is achieved, thereby achieving the technical effect of reducing the number of model parameters and computational complexity while maintaining the performance and accuracy of the model, thereby solving the technical problem that the compression method and compression parameters cannot be reasonably determined when the large language model is compressed due to the related technology. The following is a detailed description.

[0051] Optionally, in the process of constructing the above initial model, the focus is on: replacing the ReLU activation function of the forward feedback layer and the fully connected layer in the Transformer module with the SwiGLU activation function, wherein the SwiGLU activation function is used to enhance the nonlinear expression ability of the initial model.

[0052] In the embodiment of the present application, the SwigLU activation function is used to replace the traditional ReLU activation function and the fully connected layer when constructing the initial model in order to enhance the nonlinear expression ability of the model. The overall construction process can be as follows:

[0053] 1. Select or build a Transformer module:

[0054] The initial model consists of about 12 Transformer network modules, which are cascaded in the form of decoders to form the basic structure of a large language model.

[0055] 2. Decoder stacking structure:

[0056] The decoder is composed of multiple identical layers stacked together, each of which contains two sublayers: the first sublayer is the encoder-decoder attention layer, which uses a masked multi-head self-attention mechanism, where the query value (Q) comes from the output of the previous decoder layer, and the key (K) and value (V) come from the output of the encoder; the second sublayer is a multi-head self-attention convergence; the third sublayer is a bit-by-bit feedforward network, i.e., a forward feedback layer, which is responsible for further nonlinear transformation of the output at each position. Among them, the output of each sublayer is returned to the input of the sublayer through a residual connection to solve the gradient vanishing problem in deep networks. The residual connection is usually followed by a normalization layer (such as layer normalization).

[0057] 3. Replace the activation function:

[0058] In the last layer of the Transformer module (the forward feedback layer), the traditional ReLU activation function and the fully connected layer are replaced with the SwiGLU activation function, which combines the linear transformation of the two weight matrices with the nonlinear transformation of the Swish activation function.

[0059] Figure 3 Schematic diagram of a forward feedback network structure according to an embodiment of the present application. Figure 3 As shown, x represents the input data, which is processed by the self-attention module (Self Attention) and the residual connection. The part replaced by SwiGLU means that in the forward feedback layer of Transformer, the traditional ReLU activation function and the fully connected layer are replaced by the SwiGLU activation function. Subsequently, the input data x is mapped to a higher-dimensional space through a dimensionality increase operation, that is, the first part of the linear transformation in the SwiGLU module, which usually involves the weight matrix W1 and the value matrix V. Then, through a dimensionality reduction operation, that is, the second part of the linear transformation in the SwiGLU module, which usually involves the weight matrix W2, the data x is mapped back to the original dimension; and the Hadamard product is used to represent the gating mechanism in the SwiGLU module, which controls the information flow through the Hadamard product (i.e., element product). Among them, the residual connection involved in the entire module indicates that the output of the self-attention module can be added to the input through the residual connection to alleviate the gradient disappearance problem in deep network training.

[0060] In the above process, by replacing the traditional ReLU activation function and fully connected layer with the SwiGLU activation function, the initial model can obtain stronger nonlinear expression ability. At the same time, the parameters and structure of the SwiGLU activation function can be adjusted and optimized according to different task requirements, thereby improving the performance and generalization ability of the model.

[0061] Optionally, before using the pseudo-quantization node to perform a quantization operation on the floating-point weights of the initial model, the method also includes: determining the output distribution corresponding to the SwiGLU activation function, and discarding the output values ​​in the output distribution that exceed a preset distribution threshold; and determining the quantization threshold of the initial model based on the output distribution, wherein the quantization threshold is used to represent the quantization range when performing a quantization operation on the initial model.

[0062] In the embodiment of the present application, the quantization process can also be optimized according to the data distribution characteristics output by the SwiGLU activation function, as follows:

[0063] Evaluating the distribution of the output values ​​generated by the SwiGLU activation function may include, for example, identifying the mean, variance, and shape of the distribution (e.g., normal distribution, uniform distribution, etc.) of the data. In the above output distribution, there may be some outliers far away from most data points. These outliers may have an adverse effect on the quantization process, so a preset distribution threshold (e.g., data points exceeding ±30% of the mean) may be set before quantization to identify and discard these outliers.

[0064] Furthermore, the range of the quantization operation, i.e., the above-mentioned quantization threshold, can also be determined according to the output distribution of the SwiGLU activation function. The quantization threshold determines the number of bits (such as 8 bits, 16 bits, etc.) used in the quantization process, as well as the quantization accuracy. It should be noted that during the quantization process, the output value of the activation function and the corresponding gradient may need to be truncated to adapt to the quantization threshold.

[0065] Optionally, before using the pseudo-quantization node to perform a quantization operation on the floating-point weights of the initial model, the method also includes: determining the input data of the batch normalization layer in the Transformer module, and dividing the input data into multiple batches to obtain batch data; determining a first mean and a first variance of the batch data, wherein the first mean and the first variance are used to maintain the stability of the input data in the batch normalization layer; updating the first mean and the first variance to obtain a target mean and a target variance; and determining the parameters of the batch normalization layer based on the target mean and the target variance.

[0066] In the embodiment of the present application, batch normalization is a commonly used technique in deep learning, which is used to accelerate model training and improve model stability. Before quantization, the BN layer is specifically processed as follows:

[0067] The input data of the batch normalization BN layer in the Transformer module is divided into multiple batches, and each batch is used as a small training data set; the mean and variance of each batch of data are calculated to obtain the first mean and first variance, which are used to adjust the distribution of the data in the BN layer to maintain the stability of the data distribution in the BN layer; as the model training progresses, the first mean and first variance are updated to obtain the above-mentioned target mean and target variance, which are used to more accurately reflect the overall characteristics of the training data; the parameters of the BN layer, such as the mean, variance, scaling factor, and offset, are adjusted according to the target mean and target variance to ensure that the BN layer can effectively normalize the data.

[0068] Furthermore, the SwiGLU activation function can be normalized before quantization to ensure that the quantized model can maintain similar performance to the initial model.

[0069] In the above step S204, the quantization operation includes quantization and inverse quantization; a pseudo-quantization node is used to perform a quantization operation on the floating-point weights of the initial model, including: determining the floating-point number corresponding to the floating-point weight, and determining the quantization scaling factor and inverse quantization approximation value when performing a quantization operation on the initial model; and a quantization operation is performed on the floating-point weight based on the pseudo-quantization node, the floating-point number, the quantization scaling factor and the inverse quantization approximation value.

[0070] In an embodiment of the present application, the floating-point method can be used to train the initial model, and then the generated weights can be quantized. In some periods, the model needs to be fine-tuned after quantization to improve accuracy. Specifically, a pseudo-quantization node is inserted into the Transformer module of the initial model. The pseudo-quantization node is used to quantize and dequantize the weights and features of the convolutional layer before performing convolution calculations on the weights of the convolutional layer and the features of the convolutional layer input, thereby simulating the quantization loss and allowing the neural network model to learn part of the quantization loss. By introducing the pseudo-quantization node, there is no need to develop a new model architecture for the original initial model, nor is there any need to retrain the initial model. It will only cause a slight loss in precision and accuracy to the initial model, and has a good pseudo-quantization effect.

[0071] In deep learning, if the differences between channels are large, the use of tensor-level quantization may have a certain impact on accuracy. Therefore, channel-level quantization is particularly important to achieve lower precision loss. Specifically, for all parameters of the BN layer, a scale value can be specified for each different channel quantization. For example, when the initial model is running, the data passes through multiple convolutional layers and has multiple channels such as C1, C2, C3... When in the quantization channel C1, the parameter values ​​obtained after the convolution contained in channel C1 need to be quantized. Due to quantization reasons, some precision values ​​may be lost, while in the C2 channel, it is another quantized value after convolution.

[0072] Among them, the quantization operation includes quantization and dequantization. The formulas involved in the quantization and dequantization process through the pseudo quantization node are as follows:

[0073]

[0074] In the formula, v - represents the quantized weight; v^ represents the inverse quantization approximation value; s represents the quantization factor scaling parameter; v represents the input of the pseudo-quantization node, that is, the original floating-point weight to be quantized in the initial model; q represents the parameter of the pseudo-quantization node, q min Indicates the minimum value of the parameter q of the pseudo-quantization node; q max Indicates the maximum value of the parameter q of the pseudo-quantization node; round indicates the rounding operation, which is used to round the value after dividing by the scaling factor to adapt to the discreteness of quantization; the weight v is initialized maxThe maximum floating point value is 1, and the floating point value v of the initial weight min The minimum value is 0; Indicates that According to [q min ,q max ] to truncate (i.e. retain the values ​​within the interval).

[0075] It should be noted that in the process of quantization, weights are converted from floating point numbers to low-precision representations. This process involves non-differentiable operations, such as rounding operations. Since rounding operations are not mathematically differentiable, they will cause problems with gradient updates during back propagation. Therefore, the STE (Straight Through Estimator) technique can be used to set the gradient of the pseudo-quantization operator to 1, so that the gradient of the input is equal to the gradient of the output. Among them, STE is a technology for handling non-differentiable operations in back propagation. Its basic idea is that during back propagation, even if non-differentiable operations are used in forward propagation, the gradient is allowed to "pass through" these operations unimpeded.

[0076] In the above step S206, attention heads in the multi-head attention module that are below a preset threshold are discarded, including: obtaining a weight matrix of all attention heads in the multi-head attention module; determining an importance score of each attention head based on the weight matrix, wherein the importance score is used to represent the contribution of each attention head in the first model; sorting the importance scores according to the scores to obtain an importance score sequence; and discarding attention heads corresponding to importance scores in the importance score sequence that are below the preset threshold.

[0077] Furthermore, the importance score of each attention head is determined based on the weight matrix, including: obtaining a query weight matrix, a key weight matrix and a value weight matrix in the weight matrix; determining a first Euclidean norm corresponding to the query weight matrix, a second Euclidean norm corresponding to the key weight matrix and a third Euclidean norm corresponding to the value weight matrix, wherein the first Euclidean norm is used to represent the size of the query weight matrix, the second Euclidean norm is used to represent the size of the key weight matrix, and the third Euclidean norm is used to represent the size of the value weight matrix; the importance score is determined based on the first Euclidean norm, the second Euclidean norm and the third Euclidean norm.

[0078] In the embodiment of the present application, the purpose of the pruning processing unit is to compress the model by removing the parts that contribute less to the performance of the quantized first model, while maintaining the accuracy of the model as much as possible. The following is an analysis of the key steps and parameters involved in the pruning process:

[0079] 1. Get the weight matrix:

[0080] Get the weight matrices of all attention heads in the multi-head attention module, which may include the query weight matrix (Q), the key weight matrix (K), and the value weight matrix (V).

[0081] Among them, the role of the Q matrix is ​​to perform a linear transformation on the input data and map it to the vector representation of the query, that is, the Q matrix includes the conversion function required to project the input data into the query vector space. For example, the input data is first multiplied by the Q matrix to obtain the query vector Query. The dimension of the Q matrix is ​​di*dk, where di is the dimension of the input vector and dk is the dimension of the projected vector. The role of the K matrix is ​​also to perform a linear transformation, projecting the input data onto the space of the key vector Key. The K matrix contains the conversion function that converts the input into the key vector representation. The input data is multiplied by the K matrix to obtain Key. The dimension of the K matrix is ​​also di*dk. The V matrix is ​​used to generate the value vector Value. The V matrix contains the conversion function that converts the input into the value vector representation. The input data is multiplied by the V matrix to obtain Value. The dimension of the V matrix is ​​di*dv, where dv is the dimension of the value vector.

[0082] 2. Determine the importance score:

[0083] The importance score of each attention head is determined by the Euclidean norm of its weight matrix, which reflects the contribution of each attention head in the first model.

[0084] For each attention head, the Euclidean norm of the Q matrix is ​​calculated, that is, the first Euclidean norm mentioned above; and the Euclidean norm of the K matrix is ​​calculated, that is, the second Euclidean norm mentioned above; and the Euclidean norm of the V matrix is ​​calculated, that is, the third Euclidean norm mentioned above. These Euclidean norms are the square roots of the sum of the squares of the matrix elements and are used to measure the size of the weight matrix.

[0085] Furthermore, the importance score is determined according to the first Euclidean norm, the second Euclidean norm and the third Euclidean norm. The specific formula is as follows:

[0086] Score=α1||Q|| l2 +α2||K|| l2 +α3||V|| l2

[0087] Among them, Score represents the importance score, ||Q|| l2 represents the first Euclidean norm, ||K|| l2 represents the second Euclidean norm, ||V|| l2 represents the third Euclidean norm, α1 is the importance coefficient of the Q matrix, α2 is the importance coefficient of the K matrix, and α3 is the importance coefficient of the V matrix.

[0088] 3. Sorting and pruning:

[0089] According to the calculated importance scores, a hash table is used to sort all attention heads from high to low according to the scores to form an importance score sequence. Specifically, all attention heads are traversed and the importance scores of all attention heads are inserted into the hash table; then the sorting function is called to sort the hash table so that all attention heads in the hash table are sorted from high to low according to the importance scores. In a specific embodiment, it can be set to delete the attention heads that account for p% of the total number of attention heads in the multi-head attention component; the attention heads that account for p% are selected at the end of the hash table, and the selected attention heads are deleted to obtain an updated Transformer model. Compared with random selection and deletion, the head selection method based on evaluation sorting can ensure that the deletion of the attention heads that have the least impact on the model is simple and efficient.

[0090] Furthermore, a preset threshold is set to perform pruning operations, that is, attention heads below the preset threshold are discarded to achieve the deletion and pruning of attention heads with lower importance. It should be noted that after deleting and pruning some attention heads, the parameters of the model may be changed, resulting in a decrease in model accuracy. In order to restore the performance of the model, fine-tune can be used to fine-tune the updated Transformer model training. Specifically, use the pre-trained complete Transformer model as the initialization model; then, load the initialization model, and delete the low-importance attention heads according to the above pruning process; then, continue to train the model after deleting the attention heads on the training data, perform gradient updates at a lower learning rate, and train a certain number of iterations. Take the GPT2 model integrated with 12 layers of Transformer as an example, in which multi-head attention parameters account for about 85%. If 6 heads are deleted, the number of parameters can be reduced by about 10%, and the reduction in the number of parameters can directly lead to a reduction in the amount of calculation. Measured in GB, the Transformer module requires about 1.5GB of storage space, and if 10% of the parameters are compressed, it can be reduced by 0.15GB.

[0091] In the embodiment of the present application, by replacing the ReLU activation function of the last layer (forward feedback layer) of the Transformer module in the large language model and the fully connected layer with the SwiGLU activation function, the model can have stronger nonlinear expression ability, which helps to further improve the performance and generalization ability of the large language model. Furthermore, by combining model quantization processing and model pruning technology, the efficiency and practicality of the model can be improved without significantly losing the performance of the model, making it more suitable for deployment in various environments, achieving the compression effect of the large language model, and it has been verified that the compression of the large language model can delete 30% of the model attention heads to reduce the number of parameters by 16%, while achieving a good accuracy reduction effect, and the accuracy is only reduced by 1.1%.

[0092] According to an embodiment of the present application, a model compression device is provided. It should be noted that the model compression device of the embodiment of the present application can be used to execute the model compression method provided in the embodiment of the present application. The model compression device provided in the embodiment of the present application is introduced below.

[0093] Figure 4 is a structural diagram of a model compression device provided according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0094] An acquisition module 40 is used to acquire an initial model, wherein the initial model includes a plurality of Transformer modules, and the initial model is a large language model whose weights are in floating point form;

[0095] A quantization module 42, configured to perform a quantization operation on the floating-point weights of the initial model using a pseudo quantization node to obtain a first model, wherein the pseudo quantization node is used to simulate the quantization effect of the initial model;

[0096] The discarding module 44 is used to determine the multi-head attention module in the Transformer module of the first model, and discard the attention heads in the multi-head attention module that are lower than a preset threshold to obtain a compressed model.

[0097] Through the acquisition module 40, quantization module 42 and discarding module 44 in the above-mentioned model compression device, the purpose of compressing and optimizing the large language model by combining quantization and pruning technology is achieved, thereby achieving the technical effect of reducing the number of parameters and calculation complexity of the model while maintaining the performance and accuracy of the model, thereby solving the technical problem that the relevant technology cannot reasonably determine the compression method and compression parameters when compressing the large language model.

[0098] The model compression device provided in the embodiment of the present application also includes a processing module 46, which is used to replace the ReLU activation function of the forward feedback layer and the fully connected layer in the Transformer module with the SwiGLU activation function, wherein the SwiGLU activation function is used to enhance the nonlinear expression ability of the initial model.

[0099] In the model compression device provided in the embodiment of the present application, the processing module is also used to determine the output distribution corresponding to the SwiGLU activation function, and discard the output values ​​in the output distribution that exceed the preset distribution threshold; and determine the quantization threshold of the initial model based on the output distribution, wherein the quantization threshold is used to represent the quantization range when performing quantization operations on the initial model.

[0100] In the model compression device provided in the embodiment of the present application, the processing module is also used to determine the input data of the batch normalization layer in the Transformer module, and divide the input data into multiple batches to obtain batch data; determine the first mean and first variance of the batch data, wherein the first mean and first variance are used to maintain the stability of the input data in the batch normalization layer; update the first mean and first variance to obtain the target mean and target variance; determine the parameters of the batch normalization layer based on the target mean and target variance.

[0101] In the model compression device provided in the embodiment of the present application, the quantization module is also used to determine the floating-point number corresponding to the floating-point weight, and to determine the quantization scaling factor and inverse quantization approximation value when performing quantization operations on the initial model; and to perform quantization operations on the floating-point weight based on the pseudo-quantization node, floating-point number, quantization scaling factor and inverse quantization approximation value.

[0102] In the model compression device provided in the embodiment of the present application, the discarding module is also used to obtain the weight matrix of all attention heads in the multi-head attention module; determine the importance score of each attention head based on the weight matrix, wherein the importance score is used to indicate the degree of contribution of each attention head in the first model; sort the importance scores according to the score size to obtain an importance score sequence; and discard the attention heads corresponding to the importance scores in the importance score sequence that are lower than a preset threshold.

[0103] In the model compression device provided in the embodiment of the present application, the discarding module is also used to obtain the query weight matrix, the key weight matrix and the value weight matrix in the weight matrix; determine the first Euclidean norm corresponding to the query weight matrix, the second Euclidean norm corresponding to the key weight matrix and the third Euclidean norm corresponding to the value weight matrix, wherein the first Euclidean norm is used to represent the size of the query weight matrix, the second Euclidean norm is used to represent the size of the key weight matrix, and the third Euclidean norm is used to represent the size of the value weight matrix; determine the importance score based on the first Euclidean norm, the second Euclidean norm and the third Euclidean norm.

[0104] An embodiment of the present application also provides an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned model compression method.

[0105] It should be noted that the above electronic equipment is used to execute Figure 2 The model compression method shown, therefore the relevant explanations in the above model compression method are also applicable to the electronic device and will not be repeated here.

[0106] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned model compression method by running the computer program.

[0107] It should be noted that the above non-volatile storage medium is used to execute Figure 2 The model compression method shown, therefore the relevant explanations in the above model compression method are also applicable to the non-volatile storage medium and will not be repeated here.

[0108] An embodiment of the present application also provides a computer program product, including computer instructions, which implement the above-mentioned model compression method when executed by a processor.

[0109] It should be noted that the above-mentioned computer program product is used to execute Figure 2 The model compression method shown, therefore the relevant explanations in the above model compression method are also applicable to the computer program product and will not be repeated here.

[0110] An embodiment of the present application also provides a computer program, which implements the above-mentioned model compression method when executed by a processor.

[0111] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0112] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0114] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0115] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0117] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A model compression method, It is characterized in that include: Obtaining an initial model, wherein the initial model includes a plurality of Transformer modules, and the initial model is a large language model whose weights are in floating point form; Using a pseudo quantization node to perform a quantization operation on the floating-point weights of the initial model to obtain a first model, wherein the pseudo quantization node is used to simulate the quantization effect of the initial model; Determine a multi-head attention module in the Transformer module of the first model, and discard the attention heads in the multi-head attention module that are below a preset threshold to obtain a compressed model.

2. The method according to claim 1, It is characterized in that The method further comprises: The ReLU activation function of the forward feedback layer and the fully connected layer in the Transformer module are replaced with the SwiGLU activation function, wherein the SwiGLU activation function is used to enhance the nonlinear expression ability of the initial model.

3. The method according to claim 2, It is characterized in that Before using the pseudo quantization node to perform a quantization operation on the floating-point weights of the initial model, the method further includes: Determine the output distribution corresponding to the SwiGLU activation function, and discard the output values ​​in the output distribution that exceed a preset distribution threshold; and A quantization threshold of the initial model is determined according to the output distribution, wherein the quantization threshold is used to represent a quantization range when a quantization operation is performed on the initial model.

4. The method according to claim 1, It is characterized in that Before using the pseudo quantization node to perform a quantization operation on the floating-point weights of the initial model, the method further includes: Determine input data of a batch normalization layer in the Transformer module, and divide the input data into multiple batches to obtain batch data; Determining a first mean and a first variance of the batch data, wherein the first mean and the first variance are used to maintain stability of the input data in the batch normalization layer; Updating the first mean and the first variance to obtain a target mean and a target variance; Parameters of the batch normalization layer are determined according to the target mean and the target variance.

5. The method according to claim 1, It is characterized in that The quantization operation includes quantization and dequantization; The pseudo quantization node is used to perform a quantization operation on the floating point weights of the initial model, including: Determining a floating-point number corresponding to the floating-point weight, and determining a quantization scaling factor and an inverse quantization approximation value when performing a quantization operation on the initial model; A quantization operation is performed on the floating-point weight according to the pseudo quantization node, the floating-point number, the quantization scaling factor and the inverse quantization approximation value.

6. The method according to claim 1, It is characterized in that Discarding the attention heads below a preset threshold in the multi-head attention module, including: Obtain the weight matrix of all attention heads in the multi-head attention module; Determining an importance score of each attention head according to the weight matrix, wherein the importance score is used to indicate a contribution degree of each attention head in the first model; Sorting the importance scores according to their values ​​to obtain an importance score sequence; Discard the attention heads corresponding to the importance scores in the importance score sequence that are lower than a preset threshold.

7. The method according to claim 6, It is characterized in that The importance score of each attention head is determined according to the weight matrix, including: Obtaining a query weight matrix, a key weight matrix, and a value weight matrix in the weight matrix; Determine a first Euclidean norm corresponding to the query weight matrix, a second Euclidean norm corresponding to the key weight matrix, and a third Euclidean norm corresponding to the value weight matrix, wherein the first Euclidean norm is used to represent the size of the query weight matrix, the second Euclidean norm is used to represent the size of the key weight matrix, and the third Euclidean norm is used to represent the size of the value weight matrix; The importance score is determined according to the first Euclidean norm, the second Euclidean norm, and the third Euclidean norm.

8. A model compression device, It is characterized in that include: An acquisition module, used for acquiring an initial model, wherein the initial model includes a plurality of Transformer modules, and the initial model is a large language model whose weights are in floating point form; A quantization module, configured to perform a quantization operation on the floating-point weights of the initial model using a pseudo quantization node to obtain a first model, wherein the pseudo quantization node is used to simulate the quantization effect of the initial model; A discarding module is used to determine the multi-head attention module in the Transformer module of the first model, and discard the attention heads in the multi-head attention module that are lower than a preset threshold to obtain a compressed model.

9. An electronic device, It is characterized in that include: A memory and a processor, wherein the memory is used to store program instructions; The processor is connected to the memory, and is used to execute the model compression method described in any one of claims 1 to 7.

10. A non-volatile storage medium, It is characterized in that The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the model compression method described in any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising computer instructions, It is characterized in that When the computer instructions are executed by a processor, the model compression method described in any one of claims 1 to 7 is implemented.