Compression method, device, storage medium and program product of neural network model

Abstract

One or more embodiments of the present specification provide a compression method, device, storage medium and program product for a neural network model. The method includes: inputting a first data sample into a neural network model to be compressed for inference to obtain a key-value cache data set generated by a multi-head attention structure during the inference process; the key-value cache data set includes a plurality of key-value cache data corresponding to a plurality of attention heads in the multi-head attention structure; dividing the plurality of key-value cache data in the key-value cache data set into a group of at least two attention heads to obtain a key-value cache data subset corresponding to each group; performing singular value decomposition on the key-value cache data subset corresponding to each group to obtain a compression weight corresponding to each group; and compressing the model parameters of at least two attention heads included in each group using the compression weight corresponding to each group to compress the multi-head attention structure into a grouped query attention structure.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of deep learning technology, and in particular, to a compression method for a neural network model, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In recent years, neural network models containing Transformer structures have achieved remarkable success in many NLP (Natural Language Processing) tasks, including machine translation, text summarization, question-answering systems, text generation, etc.

[0003] In the related art, the neural network model containing the Transformer structure adopts an autoregressive decoding mechanism, which supports key-value (KV) caching, that is, caching the key (Key) and value (Value) of the previous context in memory, so that the key (Key) and value (Value) of the previous context cached in memory can be directly used in the process of decoding the current character, avoiding redundant calculations and speeding up the reasoning process. However, during reasoning, the keys and values ​​generated by each multi-head attention structure in the decoder of the Transformer structure need to be cached, so the amount of data in the key-value (KV) cache is usually very high, which will cause the memory usage of the key-value (KV) cache to increase linearly, further causing the decoding process in the neural network model reasoning process to be memory-limited, significantly slowing down the model reasoning speed. Obviously, the linear expansion of the memory usage of the key-value (KV) cache has become a key bottleneck in the deployment of neural network models containing the Transformer structure. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a method for compressing a neural network model, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a compression method for a neural network model is proposed, wherein the neural network model includes a Transformer structure, the Transformer structure includes at least one decoder, and the decoder includes a multi-head attention structure; the method includes:

[0007] Inputting the first data sample into the neural network model to be compressed for inference to obtain a key-value cache data set generated by the multi-head attention structure during the inference process; the key-value cache data set includes a plurality of key-value cache data corresponding to a plurality of attention heads in the multi-head attention structure;

[0008] Dividing the multiple key-value cache data in the key-value cache data set in a manner of forming at least two attention heads into a group, to obtain key-value cache data subsets corresponding to each group;

[0009] Performing singular value decomposition on the key-value cache data subsets corresponding to each group to obtain compression weights corresponding to each group;

[0010] The model parameters of at least two attention heads included in each group are compressed using the compression weights corresponding to each group, so as to compress the multi-head attention structure into a group query attention structure.

[0011] According to a second aspect of the embodiments of this specification, an electronic device is provided, including:

[0012] processor;

[0013] a memory for storing processor-executable instructions;

[0014] When the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0015] According to a third aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any of the above methods are implemented.

[0016] According to a fourth aspect of the embodiments of this specification, a computer program product is provided, including a computer program, which implements the steps of any of the above methods when executed by a processor.

[0017] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0018] In an embodiment of the present specification, a first data sample is input into a neural network model to be compressed for inference, so as to collect a key-value cache data set generated during the inference process, and then the key-value cache data is grouped, and multiple key-value cache data in the key-value cache data set are divided into a group of at least two attention heads to obtain key-value cache data subsets corresponding to each group, and singular value decomposition is performed on the key-value cache data subsets corresponding to each group. The main features (i.e., compression weights) can be extracted by singular value decomposition, and these features are used to compress the model parameters in the multi-head attention structure, and the multi-head attention structure in the decoder is converted into a group query attention structure, which can reduce memory usage and significantly improve the inference speed of the model; further, the reduced memory usage and accelerated inference speed make the model easier to deploy in resource-constrained environments (such as mobile devices, edge computing devices, etc.), thereby improving the scalability and adaptability of the model.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of a Transformer structure provided by an exemplary embodiment.

[0021] Figure 2 It is a flowchart of a compression method of a neural network model provided by an exemplary embodiment.

[0022] Figure 3 It is a flowchart of another compression method of a neural network model provided by an exemplary embodiment.

[0023] Figure 4 It is a schematic diagram of compressing the first model parameter provided by an exemplary embodiment.

[0024] Figure 5 It is a schematic structural diagram of an electronic device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0026] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0027] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0028] Here are some explanations of the relevant terms in the embodiments of the present application:

[0029] 1. Transformer: A deep learning model architecture, a model based on the attention mechanism, specifically designed to process sequence data, especially in the field of natural language processing, it has achieved remarkable success.

[0030] 2. Attention: It is a core component of the Transformer model. It is a mechanism for processing sequence data, especially suitable for natural language processing tasks. The attention mechanism allows the model to consider the information of all other elements in the sequence while processing each element in the sequence, so as to capture long-distance dependencies and better understand the context.

[0031] Exemplarily, the implementation of the attention mechanism includes the following steps:

[0032] First, it involves Query, Key, and Value conversion. For each input position, three vectors are obtained through linear transformation: Query vector ( q )、Key vector( k ) and the Value vector ( v ). These vectors are used to calculate attention scores and generate output. By calculating the similarity between the Query vector and each Key vector (usually using dot product or other similarity functions), the attention score of each position to all other positions is obtained, and finally the attention score is used to weighted sum all Value vectors to generate the final output.

[0033] It can be expressed as the following formula:

[0034] ;

[0035] ;

[0036] ;

[0037]

[0038] .

[0039] in, X represents the input vector, represents the query matrix learned during model training, represents the key matrix learned during model training, represents the matrix of values ​​learned during model training, q represents the Query vector, k represents the Key vector, v represents the Value vector, （） is the scaling and normalization function, is the scaling factor (used to scale the dot product result to a moderate range to alleviate numerical instability issues), d is the dimension of the Query vector, is the transpose of the Key vector, represents the output weights learned during model training, out Represents the final output.

[0040] 3. Multi-Head Attention: It is used to further enhance the model's ability to model input sequences. In the single-head attention mechanism, the model obtains the representation of each position by calculating a single attention distribution. The multi-head attention mechanism extracts information from different subspaces by applying multiple attention heads in parallel, and then merges this information to obtain a richer and more accurate representation.

[0041] It can be expressed as the following formula:

[0042] ;

[0043] ;

[0044] ;

[0045] ;

[0046] ;

[0047] .

[0048] in, X represents the input vector, i The value of 0~h , h is the number of attention heads in the multi-head attention mechanism, and each attention head has a corresponding query matrix ( ), key matrix ( ), value matrix ( ) and output weights ( ), and then the Query vector corresponding to each attention head can be calculated ( )、Key vector( )、Value vector( ) and the output of each attention head ( ), and finally concatenate the outputs of all attention heads together to form the output of the multi-head attention mechanism ( ).

[0049] 4. Grouped Query Attention (GQA): It is an improvement on the traditional attention mechanism, which aims to reduce computational and memory overhead while maintaining good model performance. GQA reduces computational complexity by grouping queries so that queries in each group share the same key and value. In the attention mechanism, each query vector needs to be dot-producted with all key vectors, and then the softmax function is applied to get the weight, and all value vectors are weighted summed, which results in a computational complexity of ,in n is the length of the input sequence. GQA reduces the computational complexity by grouping query vectors and sharing keys and values ​​within each group.

[0050] 5. Rotary Positional Embedding (RoPE) is a position encoding technique used in the Transformer architecture. The core idea of ​​RoPE is to enhance the model's ability to represent the relationship between different positions in a sequence by introducing a learnable rotation matrix;

[0051] After introducing the rotation position embedding, the multi-head attention mechanism can be expressed as the following formula:

[0052] ;

[0053] ;

[0054] ;

[0055]

[0056] ;

[0057] .

[0058] Among them, the location information is introduced in the Query vector and the Key vector respectively. and .

[0059] 6. Singular Value Decomposition (SVD) is an important matrix decomposition technique in linear algebra, which can decompose a matrix into the product of three matrices. By retaining only the part with larger singular values, the original matrix can be approximately reconstructed to achieve data dimensionality reduction and compression while retaining the main features.

[0060] 7. LoRA (Low-Rank Adaptation) is a technique for fine-tuning neural network models by adjusting the parameters of the model by introducing a low-rank structure instead of updating the original model parameters directly. This approach aims to reduce the number of parameters and computing resources required during training while maintaining or improving model performance. The core idea of ​​LoRA is to introduce a low-rank structure into the weight matrix of the model. This means that the weight matrix can be approximated by the product of a smaller basis matrix and a learnable low-rank matrix. During fine-tuning, only the parameters of the low-rank matrix are updated, while the basis matrix remains as it is. This can significantly reduce the number of parameters that need to be adjusted during training.

[0061] 8. Large Language Model (LLM) usually refers to large pre-trained language models. These models are based on deep learning and natural language processing technology and can automatically learn and understand the language structure and semantic information of large-scale text data. Generally, they have more than 100 million model parameters.

[0062] In recent years, neural network models containing Transformer structures (such as LLM models) have achieved remarkable success in many NLP (Natural Language Processing) tasks, including machine translation, text summarization, question-answering systems, text generation, etc.

[0063] Exemplarily, neural network models containing Transformer structures include LLM models. The following is an exemplary explanation of the LLM model. The emergence of pre-trained large language models (LLMs) redefines the performance benchmarks of various natural language processing (NLP) tasks. These models generally exhibit performance comparable to human understanding and generation levels. Generally speaking, LLMs are built on the neural structure of Transformer, which requires a computational cost quadratic to the length of the input sequence, making long sequence reasoning difficult to handle. To alleviate this problem, the LLM model adopts an autoregressive decoding mechanism, which supports key-value (KV) caching, that is, caching the key (Key) and value (Value) of the previous context in memory, so that the key (Key) and value (Value) of the previous context cached in memory can be directly used in the process of decoding the current character, avoiding redundant calculations and speeding up the reasoning process. However, during inference, the keys and values ​​generated by each multi-head attention structure at each step need to be cached. Therefore, the amount of data in the key-value (KV) cache is usually very high, which will cause the memory usage of the key-value (KV) cache to increase linearly, further causing the decoding process during the neural network model inference process to be memory-limited, significantly slowing down the model inference speed. Obviously, the linear expansion of the memory usage of the key-value (KV) cache has become a key bottleneck for the deployment of neural network models containing Transformer structures.

[0064] Based on this, the embodiments of this specification provide a compression method for a neural network model including a Transformer structure, which can convert the multi-head attention structure in the decoder into a group query attention structure through low-rank decomposition, thereby reducing the amount of key-value cache data and reducing the memory usage of the key-value (KV) cache, which is beneficial to improving the model reasoning speed.

[0065] The compression method of the neural network model provided in the embodiments of this specification can be executed by an electronic device, which includes but is not limited to physical servers, server clusters, cloud servers, smart phones / mobile phones, tablet computers, personal digital assistants (PDAs), laptop computers, desktop computers and other devices with computing functions.

[0066] The neural network models to be compressed include: Figure 1 The Transformer structure shown in the figure includes at least one decoder, and the decoder includes a multi-head attention structure. Among them, the main task of the decoder is to convert the input sequence into a hidden representation and is responsible for generating an output sequence.

[0067] For the specific structure of the decoder, please refer to the description in the related art and will not be repeated here.

[0068] See also Figure 2 , a schematic flow chart of a method for compressing a neural network model is shown. The method can be executed by an electronic device, and the method includes:

[0069] In S201, the first data sample is input into the neural network model to be compressed for inference to obtain a key-value cache data set generated by the multi-head attention structure during the inference process; the key-value cache data set includes multiple key-value cache data corresponding one-to-one to multiple attention heads in the multi-head attention structure.

[0070] Exemplarily, the first data sample is used to start the reasoning process of the neural network model. The first data sample can be any representative input data, and this embodiment does not impose any limitation on this.

[0071] In the multi-head attention mechanism, each attention head generates a set of keys and values, which are cached during the inference process for use in subsequent steps. During the inference process of the neural network model, each word segment is generated once, and the corresponding key (k) and value (v) are retained as key-value cache data for subsequent inference processes. For example, in the process of decoding the sth word segment, the previously cached keys and values ​​of the 1st to s-1th characters can be combined for calculation.

[0072] Exemplarily, it can be expressed as the following formula: ; ;in, and Represents a formula based on the following form , The calculated Key vector and Value vector corresponding to the sth word segmentation are: Represents the concatenation result of the Key vectors from the 1st character to the s-1th character. Represents the concatenation result of the Value vectors from the 1st character to the s-1th character. Represents the concatenation result of the Key vectors from the 1st character to the sth character, Represents the concatenation result of the Value vectors from the 1st character to the sth character. In the subsequent calculation of the attention score and output of the sth word segment, and Participate in calculations.

[0073] Step S201 collects the key-value cache data set generated during the inference process to provide basic data for subsequent compression processing.

[0074] In S202, multiple key-value cache data in the key-value cache data set are divided in a manner of forming at least two attention heads into a group, to obtain key-value cache data subsets corresponding to each group.

[0075] By processing in groups of at least two attention heads, the grouping strategy can be flexibly adjusted according to the specific application scenario to balance memory usage and computational complexity. For example, assuming that the multi-head attention structure includes 6 heads (that is, 6 attention heads), it can be divided into 3 groups in a group of two attention heads. The key-value cache data of each group forms a subset, which will be used for subsequent compression processing.

[0076] In S203, singular value decomposition is performed on the key-value cache data subset corresponding to each group to obtain a compression weight corresponding to each group.

[0077] Among them, SVD (Singular Value Decomposition) is a matrix decomposition method that can decompose a matrix into the product of three matrices, extract the main features in the matrix, and reduce data redundancy. In this embodiment, the main features extracted by SVD will be used to compress model parameters.

[0078] In S204, the model parameters of at least two attention heads included in each group are compressed using the compression weights corresponding to each group, so as to compress the multi-head attention structure into a group query attention structure.

[0079] Exemplarily, the compression weights obtained in step S203 can be used to recalculate the model parameters, thereby converting the original multi-head attention structure into a group query attention structure to reduce memory usage and computational complexity.

[0080] In this embodiment, the key-value cache data is grouped and compressed. For example, the main features are extracted through singular value decomposition, and these features are used to compress the model parameters. This can reduce memory usage and significantly improve the reasoning speed of the model. Furthermore, the reduced memory usage and accelerated reasoning speed make the model easier to deploy in resource-constrained environments (such as mobile devices, edge computing devices, etc.), thereby improving the scalability and adaptability of the model.

[0081] In some embodiments, the key-value cache dataset generated by the multi-head attention structure during the inference process includes a key cache dataset and a value cache dataset. Figure 3 , shows a flow chart of another compression method of a neural network model. The method includes:

[0082] In S301, the first data sample is input into the neural network model to be compressed for inference to obtain a key cache data set and a value cache data set generated by the multi-head attention structure during the inference process; the key cache data set includes a plurality of key cache data corresponding one-to-one to the plurality of attention heads in the multi-head attention structure; the value cache data set includes a plurality of value cache data corresponding one-to-one to the plurality of attention heads in the multi-head attention structure.

[0083] In this step, the key cache data set and the value cache data set are obtained to provide basic data for subsequent grouping and compression processing.

[0084] In S302, multiple key cache data in the key cache data set and multiple value cache data in the value cache data set are divided into a group of at least two attention heads to obtain key cache data subsets and value cache data subsets corresponding to each group.

[0085] In this step, the key cache data and the value cache data are divided into groups, which reduces the computational complexity in a single decomposition process and helps improve the decomposition efficiency.

[0086] In S303, singular value decomposition is performed on the key cache data subset corresponding to each group to obtain a first compression weight corresponding to each group.

[0087] In this step, singular value decomposition is performed on the key cache data subset to extract the main features to obtain the first compression weight, thereby compressing the key-related model parameters.

[0088] Exemplarily, the electronic device may perform singular value decomposition on a key cache data subset corresponding to each group to obtain a first right singular vector matrix, and select the first N rows in the first right singular vector matrix as the first compression weights corresponding to each group. In this embodiment, the method of selecting the first N rows retains the most important part of the data, so while compressing the data, the performance of the original model is maintained as much as possible.

[0089] Among them, N can be determined based on the dimension of the input vector indicated by the multi-head attention structure and the number of attention heads included in the multi-head attention structure, for example, N includes the quotient obtained by dividing the dimension of the input vector indicated by the multi-head attention structure by the number of attention heads included in the multi-head attention structure.

[0090] It can be expressed as the following formula: .in, is the left singular vector matrix, is a diagonal singular value matrix, is the right singular vector matrix. The left singular vector matrix and the right singular vector matrix are orthogonal. Select the first N rows in the matrix, which correspond to the first N largest singular values, and use the selected right singular vectors for low-rank approximation to achieve the first N rows in the first right singular vector matrix as the first compression weights corresponding to each group (that is, ).

[0091] In S304, singular value decomposition is performed on the value cache data subset corresponding to each group to obtain a second compression weight corresponding to each group.

[0092] In this step, singular value decomposition is performed on the value cache data subset to extract the main features to obtain the second compression weight, thereby compressing the value-related model parameters.

[0093] Exemplarily, the electronic device can perform singular value decomposition on a subset of the value cache data corresponding to each group to obtain a second right singular vector matrix, and select the first N rows in the second right singular vector matrix as second compression weights corresponding to each group; in this embodiment, the method of selecting the first N rows retains the most important part of the data, thereby maintaining the performance of the original model as much as possible while compressing the data.

[0094] Among them, N is determined based on the dimension of the input vector indicated by the multi-head attention structure and the number of attention heads included in the multi-head attention structure. For example, N includes the quotient obtained by dividing the dimension of the input vector indicated by the multi-head attention structure by the number of attention heads included in the multi-head attention structure.

[0095] It can be expressed as the following formula: .in, is the left singular vector matrix, is a diagonal singular value matrix, is the right singular vector matrix. The left singular vector matrix and the right singular vector matrix are orthogonal. Select the first N rows in , which correspond to the first N largest singular values, and use the selected right singular vectors for low-rank approximation to achieve the first N rows in the second right singular vector matrix as the second compression weights corresponding to each group (expressed as in the formula ).

[0096] In S305, the first model parameters in at least two attention heads included in each group are compressed using the first compression weight corresponding to each group.

[0097] In this step, the key-related model parameters are compressed using the first compression weight to reduce the memory usage and computational complexity of the key cache.

[0098] In S306, the second model parameters in at least two attention heads included in each group are compressed using the second compression weights corresponding to each group; wherein the first model parameters and the second model parameters are different.

[0099] In this step, the model parameters related to the value are compressed using the second compression weight to reduce the memory usage and computational complexity of the value cache.

[0100] In this embodiment, singular value decomposition is performed on the key cache data and the value cache data respectively, which can ensure that the compression processing of the model parameters is more accurate, thereby maintaining the performance of the model during the reasoning process. Figure 3 As shown, the compression process described in step S303+S305 and the compression process described in step S304+S306 can be performed in parallel. In some embodiments, the first model parameter includes a key matrix and a query matrix in the attention head.

[0101] For S305, please refer to Figure 4 ( Figure 4 In the example using two attention heads as a group), when compressing the key matrices in at least two attention heads included in each group, the electronic device can merge the key matrices in at least two attention heads included in each group to obtain a first merged result corresponding to each group; this step reduces the number of key matrices that need to be processed and simplifies the subsequent calculation process. Then the electronic device determines the product between the first compression weight corresponding to each group and the first merged result corresponding to each group as the target key matrix corresponding to each group; this step implements compression processing through matrix multiplication, reducing the computational complexity. Among them, the target key matrix corresponding to each group is shared by at least two attention heads included in each group, so as to realize the compression of the multi-head attention structure into a grouped query attention structure.

[0102] This embodiment merges the key matrices in at least two attention heads included in each group, and uses the first compression weight to uniformly compress the merged result to obtain a target key matrix shared by each group, which can further reduce memory usage because at least two attention heads in each group no longer need to independently store key data, but obtain shared key data through the shared target key matrix.

[0103] In the grouped query attention structure, the query vectors of each attention head are calculated separately, so please refer to Figure 4When compressing the query matrix in at least two attention heads included in each group, the electronic device can split the first compression weight corresponding to each group into at least two third compression weights corresponding to at least two attention heads included in each group; for example, it can be an average splitting method, and the N rows of first right singular vectors in the first compression weight are evenly divided according to the number of attention heads included in each group; it can also be other division methods, and this embodiment does not impose any restrictions on this. Then in each group, the product between the query matrix of each attention head in each group and the third compression weight corresponding to the attention head is determined as the target query matrix corresponding to the attention head; that is, each attention head has its own corresponding target query matrix.

[0104] In this embodiment, by splitting the first compression weight into multiple third compression weights and applying them to the query matrix of each attention head respectively, the compression process can be controlled more finely. This refined processing method can retain the characteristics of each attention head to the greatest extent, thereby improving the accuracy of the model. Although each attention head has an independent target query matrix, the overall memory requirement is still optimized through compression processing, and the memory usage is significantly reduced compared to the uncompressed case.

[0105] In some embodiments, the second model parameters include a value matrix and output weights in the attention head.

[0106] For S306, when compressing the value matrices in at least two attention heads included in each group, the electronic device can merge the value matrices in at least two attention heads included in each group to obtain a second merged result corresponding to each group; this step reduces the number of key matrices that need to be processed and simplifies the subsequent calculation process. The electronic device then determines the product between the second compression weight corresponding to each group and the second merged result corresponding to each group as the target value matrix corresponding to each group; this step implements compression processing through matrix multiplication, reducing the computational complexity. Among them, the target value matrix corresponding to each group is shared by at least two attention heads included in each group to achieve compression of the multi-head attention structure into a grouped query attention structure.

[0107] This embodiment merges the value matrices in at least two attention heads included in each group, and uses a second compression weight to uniformly compress the merged result to obtain a target value matrix shared by each group, which can further reduce memory usage because at least two attention heads in each group no longer need to independently store value data, but instead obtain shared value data through a shared target value matrix.

[0108] In the group query attention structure, the output of each attention head is calculated separately. Therefore, when compressing the output weights of at least two attention heads included in each group, the electronic device can split the second compressed weights corresponding to each group into at least two fourth compressed weights corresponding to the at least two attention heads included in each group; then in each group, the product of the output weights of each attention head in each group and the transposed result of the fourth compressed weight corresponding to the attention head is determined as the target output weight corresponding to the attention head; that is, each attention head has its own corresponding target output weight.

[0109] In this embodiment, by splitting the second compression weight into multiple fourth compression weights and applying them to the output weights of each attention head respectively, the compression process can be controlled more finely. This refined processing method can retain the characteristics of each attention head to the greatest extent, thereby improving the accuracy of the model. Although each attention head has an independent target output weight, the overall memory requirement is still optimized through compression processing, and the memory usage is significantly reduced compared to the uncompressed case.

[0110] Exemplarily, each group includes at least two attention heads, and the first compression weight in each group is t. is split into at least two third compressed weights ( ， ， ), the second compression weight in each group is split into at least two fourth compressed weights ( ， ), where the value range of i is [1,t].

[0111] The output of each attention head can be expressed as:

[0112]

[0113]

[0114]

[0115]

[0116] .

[0117] Among them, the target key matrix corresponding to each group = ; represents the first merging result of the key matrices in at least two attention heads included in each group, Represents the first compression weight.

[0118] The target value matrix corresponding to each group = ; represents the second merging result of the value matrices in at least two attention heads included in each group, represents the second compression weight.

[0119] The target query matrix corresponding to each attention head in each group = ; represents the query matrix of the ith attention head, represents the third compressed weight of the i-th attention head.

[0120] The target output weights corresponding to each attention head in each group ; represents the output weight of the i-th attention head, Represents the transposed result of the fourth compressed weight of the i-th attention head.

[0121] This embodiment achieves compression processing of model parameters in multi-head attention heads by fusing the first compression weight into the key matrix, the second compression weight into the value matrix, the third compression weight into the query matrix, and the transposed result of the fourth compression weight into the output weight. The multi-head attention structure can be compressed into a group query attention structure, thereby effectively reducing the KV data that needs to be cached during the model inference process, reducing memory usage, and further improving the model inference speed.

[0122] In some embodiments, some Transformer architectures introduce rotational position encoding to enhance the model's ability to represent the relationship between different positions in a sequence. When the Transformer structure introduces rotational position encoding, the electronic device inputs the first data sample into the neural network model to be compressed for inference, and the obtained key-value cache data set includes a key cache data set and value cache data, and the key-value cache data included in the key cache data set carries rotational position information.

[0123] For example, when the rotation position encoding is introduced, the above . In the inference process, the neural network model will Cache as key-value cache data, which carries rotation position information .

[0124] Therefore, when the Transformer structure introduces rotational position encoding, the electronic device can directly perform singular value decomposition on the key cache data set corresponding to each attention head to obtain a third right singular vector matrix, and select the first N rows in the third right singular vector matrix as the fifth compression weights corresponding to each attention head; wherein N is determined based on the dimension of the input vector indicated by the multi-head attention structure and the number of attention heads included in the multi-head attention structure; and then use the fifth compression weights corresponding to each attention head as the model parameters of each attention head to cache the key cache data set compressed by the fifth compression weights during the neural network inference process, thereby achieving the purpose of compressing the key cache data.

[0125] The value cache data does not introduce the rotation position information, and can be compressed in a manner similar to the above, which will not be described here.

[0126] In some embodiments, after compressing the neural network model using the above method, the electronic device can fine-tune the compressed neural network model based on the LoRA algorithm. Exemplarily, the electronic device can select at least part of the model parameters in the compressed neural network model as the basis matrix, and determine the low-rank matrix corresponding to each basis matrix; use the second data sample to fine-tune the compressed neural network model; wherein, during the fine-tuning process, the parameters of the low-rank matrix corresponding to each basis matrix are updated, and the combination result of the updated low-rank matrix corresponding to each basis matrix and the basis matrix is ​​determined as the updated model parameter.

[0127] Before fine-tuning begins, some weight matrices in the compressed neural network model are selected as basis matrices, and additional low-rank matrices are introduced for these basis matrices. These low-rank matrices are usually random or use preset small perturbations when initialized. During fine-tuning, the model is iteratively updated through the second data sample. The parameters of the low-rank matrix are updated through gradient descent or similar optimization algorithms so that the model learns features related to the specific task. It should be noted that during fine-tuning, only the parameters of the low-rank matrix will be updated according to the gradient of the loss function. The parameters of the basis matrix remain unchanged at this stage, and they are indirectly adjusted by the update of the low-rank matrix. After training, the parameter update of the low-rank matrix will be reflected in the basis matrix; specifically, the product (or addition, depending on the implementation) of the basis matrix and the low-rank matrix forms a new weight matrix, which contains the information of the original basis matrix and the new features learned by the low-rank matrix. After fine-tuning is completed, the parameter update of the low-rank matrix will stop; at this time, the low-rank matrix has adjusted the basis matrix so that the model can better adapt to the specific task or data set. In this embodiment, the advantage of LoRA fine-tuning is that it can adjust the model by updating the parameters of the low-rank matrix without significantly increasing the number of parameters, which can reduce the requirements for computing and storage while maintaining the model performance.

[0128] For example, the compressed neural network model can be used to process various natural language processing (NLP) tasks, such as: (1) Text classification: used to classify text into different categories, such as sentiment analysis, news classification, etc. (2) Named entity recognition: used to identify named entities in text, such as names of people, places, and organizations. (3) Text generation: can generate text, such as reply generation in dialogue systems, article summary generation, etc. (4) Machine translation: used to translate text from one language to another. (5) Question-answering system: used to understand and answer questions about the content of text, such as reading comprehension tasks. (6) Language modeling: used to predict the next word or character in a text sequence, such as language generation and text auto-completion. (7) Sentiment analysis: analyze the emotional tendency of text, such as judging the positive and negative emotions of comments. (8) Relation extraction: identify the relationship between entities in text, such as character relationships, product features, etc.

[0129] The various technical features in the above embodiments can be arbitrarily combined as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, any combination of the various technical features in the above embodiments also falls within the scope of this specification.

[0130] In some embodiments, the embodiments of this specification also provide an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements any of the above methods by running the executable instructions.

[0131] Figure 5 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 5 At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into the memory 508 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0132] In some embodiments, the present specification also provides a compression device for a neural network model, which can be applied to Figure 5 The device shown in the figure is used to implement the technical solution of this specification. The neural network model includes a Transformer structure, the Transformer structure includes at least one decoder, and the decoder includes a multi-head attention structure; the compression device may include:

[0133] A key-value cache data set acquisition module is used to input the first data sample into the neural network model to be compressed for inference, so as to obtain the key-value cache data set generated by the multi-head attention structure during the inference process; the key-value cache data set includes multiple key-value cache data corresponding one-to-one to multiple attention heads in the multi-head attention structure.

[0134] The grouping module is used to divide multiple key-value cache data in the key-value cache data set into a group of at least two attention heads to obtain key-value cache data subsets corresponding to each group.

[0135] The decomposition module is used to perform singular value decomposition on the key-value cache data subset corresponding to each group to obtain the compression weight corresponding to each group.

[0136] A compression module is used to compress the model parameters of at least two attention heads included in each group using the compression weights corresponding to each group, so as to compress the multi-head attention structure into a group query attention structure.

[0137] In one implementation, the key-value cache data set includes a key cache data set and a value cache data set.

[0138] The key-value cache data subsets corresponding to each group include a key cache data subset and a value cache data subset corresponding to each group.

[0139] The compression weights corresponding to each group include: a first compression weight obtained based on singular value decomposition of a key cache data subset corresponding to each group, and a second compression weight obtained based on singular value decomposition of a value cache data subset corresponding to each group.

[0140] The compression module is specifically used to: compress the first model parameters in at least two attention heads included in each group using the first compression weight corresponding to each group; and compress the second model parameters in at least two attention heads included in each group using the second compression weight corresponding to each group; wherein the first model parameters and the second model parameters are different.

[0141] In one implementation, the decomposition module is specifically used to: perform singular value decomposition on a key cache data subset corresponding to each group to obtain a first right singular vector matrix, and select the first N rows in the first right singular vector matrix as first compression weights corresponding to each group; and / or, perform singular value decomposition on a value cache data subset corresponding to each group to obtain a second right singular vector matrix, and select the first N rows in the second right singular vector matrix as second compression weights corresponding to each group; wherein N is determined based on the dimension of the input vector indicated by the multi-head attention structure and the number of attention heads included in the multi-head attention structure.

[0142] Exemplarily, N includes: the quotient obtained by dividing the dimension of the input vector indicated by the multi-head attention structure by the number of attention heads included in the multi-head attention structure.

[0143] In one implementation, the first model parameters include a key matrix in an attention head.

[0144] The compression module is specifically used to: merge the key matrices in at least two attention heads included in each group to obtain a first merged result corresponding to each group; determine the product between the first compression weight corresponding to each group and the first merged result corresponding to each group as the target key matrix corresponding to each group; wherein the target key matrix corresponding to each group is shared by at least two attention heads included in each group.

[0145] In one implementation, the first model parameters also include a query matrix in the attention head;

[0146] The compression module is specifically used to: split the first compression weight corresponding to each group into at least two third compression weights corresponding to at least two attention heads included in each group; in each group, the product between the query matrix of each attention head in each group and the third compression weight corresponding to the attention head is determined as the target query matrix corresponding to the attention head.

[0147] In one implementation, the second model parameters include a matrix of values ​​in an attention head;

[0148] The compression module is specifically used to: merge the value matrices in at least two attention heads included in each group to obtain a second merged result corresponding to each group; determine the product between the second compression weight corresponding to each group and the second merged result corresponding to each group as the target value matrix corresponding to each group; wherein the target value matrix corresponding to each group is shared by at least two attention heads included in each group.

[0149] In one implementation, the second model parameters also include output weights in the attention head;

[0150] The compression module is specifically used to: split the second compression weight corresponding to each group into at least two fourth compression weights corresponding to at least two attention heads included in each group; in each group, the product between the output weight of each attention head in each group and the transposed result of the fourth compression weight corresponding to the attention head is determined as the target output weight corresponding to the attention head.

[0151] In one implementation, the Transformer structure introduces rotational position encoding;

[0152] The key-value cache data set includes a key cache data set and a value cache data set; the key-value cache data included in the key cache data set carries rotation position information;

[0153] The decomposition module is further used for performing singular value decomposition on the key cache data set corresponding to each attention head when the Transformer structure introduces the rotation position encoding, obtaining a third right singular vector matrix, and selecting the first N rows in the third right singular vector matrix as fifth compression weights corresponding to each attention head; wherein N is determined based on the dimension of the input vector indicated by the multi-head attention structure and the number of attention heads included in the multi-head attention structure;

[0154] The compression module is also used to use the fifth compression weight corresponding to each attention head as the model parameter of each attention head, so as to cache the key cache data set compressed by the fifth compression weight during the neural network inference process.

[0155] In one implementation, it also includes a fine-tuning module for selecting at least part of the model parameters in the compressed neural network model as a basis matrix, and determining a low-rank matrix corresponding to each basis matrix; using a second data sample to fine-tune the compressed neural network model; wherein, during the fine-tuning process, the parameters of the low-rank matrix corresponding to each basis matrix are updated, and the combination result of the updated low-rank matrix corresponding to each basis matrix and the basis matrix is ​​determined as the updated model parameter.

[0156] In one implementation, the neural network model includes an LLM model.

[0157] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of any of the above embodiment methods by running the executable instructions.

[0158] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method in any of the above embodiments are implemented.

[0159] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0160] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of any of the above embodiment methods when executed by a processor.

[0161] The above are only preferred embodiments of one or more embodiments of this specification and are not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification should be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A compression method for a neural network model, wherein the neural network model includes a Transformer structure, wherein the Transformer structure includes at least one decoder, and wherein the decoder includes a multi-head attention structure; the method includes: Inputting the first data sample into the neural network model to be compressed for inference to obtain a key-value cache data set generated by the multi-head attention structure during the inference process; The key-value cache data set includes a plurality of key-value cache data corresponding one-to-one to a plurality of attention heads in the multi-head attention structure; Dividing the multiple key-value cache data in the key-value cache data set in a manner of forming at least two attention heads into a group, to obtain key-value cache data subsets corresponding to each group; Performing singular value decomposition on the key-value cache data subsets corresponding to each group to obtain compression weights corresponding to each group; Using the compression weights corresponding to each group, the model parameters of at least two attention heads included in each group are compressed to compress the multi-head attention structure into a group query attention structure; The key-value cache data set includes a key cache data set and a value cache data set; the key-value cache data subsets corresponding to each group include a key cache data subset and a value cache data subset corresponding to each group; the compression weight corresponding to each group includes a first compression weight obtained based on singular value decomposition of the key cache data subset corresponding to each group; The compressing the model parameters of at least two attention heads included in each group by using the compression weights corresponding to each group includes: Splitting the first compression weight corresponding to each group into at least two third compression weights corresponding one-to-one to at least two attention heads included in each group; In each group, the product of the query matrix of each of the attention heads in each group and the third compression weight corresponding to the attention head is determined as the target query matrix corresponding to the attention head.

2. The method according to claim 1, The compression weights corresponding to each group also include: A second compression weight obtained based on a singular value decomposition of a value cache data subset corresponding to each group; The compressing the model parameters of at least two attention heads included in each group by using the compression weights corresponding to each group includes: Using a first compression weight corresponding to each group, compressing first model parameters in at least two attention heads included in each group; and The second model parameters in at least two attention heads included in each group are compressed using a second compression weight corresponding to each group; wherein the first model parameters and the second model parameters are different.

3. The method according to claim 2, obtaining the first compression weight corresponding to each group comprises: Performing singular value decomposition on the key cache data subsets corresponding to each group to obtain a first right singular vector matrix, and selecting first N rows in the first right singular vector matrix as the first compression weights corresponding to each group; and / or Obtaining the second compression weight corresponding to each group includes: Performing singular value decomposition on the value cache data subsets corresponding to each group to obtain a second right singular vector matrix, and selecting first N rows in the second right singular vector matrix as the second compression weights corresponding to each group; Among them, the N is determined based on the dimension of the input vector indicated by the multi-head attention structure and the number of attention heads included in the multi-head attention structure.

4. The method according to claim 3, wherein N comprises: The quotient obtained by dividing the dimension of the input vector indicated by the multi-head attention structure by the number of attention heads included in the multi-head attention structure.

5. The method of claim 2, wherein the first model parameter comprises a key matrix in the attention head; The compressing the first model parameters in at least two attention heads included in each group by using the first compression weight corresponding to each group includes: Merging the key matrices in at least two attention heads included in each group to obtain a first merged result corresponding to each group; The product of the first compression weight corresponding to each group and the first merging result corresponding to each group is determined as the target key matrix corresponding to each group; wherein the target key matrix corresponding to each group is shared by at least two attention heads included in each group.

6. The method of claim 2, wherein the second model parameters comprise a matrix of values ​​in the attention head; The compressing the second model parameters in at least two attention heads included in each group by using the second compression weight corresponding to each group includes: Merging the value matrices in at least two attention heads included in each group to obtain a second merged result corresponding to each group; The product of the second compression weight corresponding to each group and the second merging result corresponding to each group is determined as the target value matrix corresponding to each group; wherein the target value matrix corresponding to each group is shared by at least two attention heads included in each group.

7. The method according to claim 2 or 6, wherein the second model parameters further include output weights in the attention head; The compressing the second model parameters in at least two attention heads included in each group by using the second compression weight corresponding to each group includes: Splitting the second compression weight corresponding to each group into at least two fourth compression weights corresponding one-to-one to at least two attention heads included in each group; In each group, the product of the output weight of each of the attention heads in each group and the transposed result of the fourth compression weight corresponding to the attention head is determined as the target output weight corresponding to the attention head.

8. According to the method of claim 1, the Transformer structure introduces rotation position encoding; The key-value cache data set includes a key cache data set and a value cache data set; the key-value cache data included in the key cache data set carries rotation position information; The method further comprises: In the case where the Transformer structure introduces rotational position encoding, singular value decomposition is performed on the key cache data set corresponding to each of the attention heads to obtain a third right singular vector matrix, and the first N rows in the third right singular vector matrix are selected as fifth compression weights corresponding to each of the attention heads; wherein N is determined based on the dimension of the input vector indicated by the multi-head attention structure and the number of attention heads included in the multi-head attention structure; The fifth compression weight corresponding to each of the attention heads is used as a model parameter of each of the attention heads to cache the key cache data set compressed by the fifth compression weight during the neural network inference process.

9. The method according to claim 1, further comprising: Selecting at least part of the model parameters in the compressed neural network model as basis matrices, and determining low-rank matrices corresponding to each basis matrix; The compressed neural network model is fine-tuned using the second data sample; wherein, during the fine-tuning process, the parameters of the low-rank matrices corresponding to each basis matrix are updated, and the combination result of the updated low-rank matrix corresponding to each basis matrix and the basis matrix is ​​determined as the updated model parameter.

10. The method according to claim 1, wherein the neural network model comprises an LLM model.

11. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 10 by executing the executable instructions.

12. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

13. A computer program product comprising a computer program / instructions, which when executed by a processor implement the steps of the method according to any one of claims 1 to 10.

Images