Computing method and apparatus for large model inference acceleration

Abstract

The application relates to a computing method and device for large model inference acceleration. The method comprises the following steps: obtaining a query feature vector of a current word element of large model inference and a key feature matrix of a historical word element sequence; determining an attention weight vector of the current word element relative to the historical word element sequence according to the query feature vector of the current word element and the key feature matrix of the historical word element sequence; determining a key weight vector of the current word element according to the attention weight vector and the key feature matrix of the historical word element sequence; and determining an attention output vector of the current word element according to the key weight vector and a preset key value transformation matrix. The method can reduce the cache occupation in the calculation process of the attention output vector, thereby reducing the memory access overhead in the process of determining the attention output vector, and improving the inference speed of the large model.

Description

Technical Field

[0001] This application relates to the field of large model inference technology, and in particular to a computational method and apparatus for accelerating large model inference. Background Technology

[0002] During the inference process, the Large Language Model (LLM) needs to determine the attention output vector of the latest inferred word relative to the entire historical word sequence. The attention output vector is the core technical foundation for the model to accurately predict subsequent output words.

[0003] In current large-scale model inference, a key-value cache (KV Cache) mechanism is typically used to calculate the attention output vector. This approach usually requires storing the key feature matrix and value feature matrix of the historical word sequence, and then determining the attention output vector of the current word relative to the entire historical word sequence based on these stored matrix values.

[0004] However, as the historical word sequence continues to grow, the size of its key feature matrix and value feature matrix also continues to increase. Storing such a large key feature matrix and value feature matrix in the cache will lead to excessive cache usage during the calculation of the attention output vector, which in turn leads to large memory access overhead for the key feature matrix and value feature matrix, reducing the inference speed of large models. Summary of the Invention

[0005] Based on this, this application provides a computational method and apparatus for accelerating large model inference, which can reduce cache usage during the calculation of attention output vector, thereby reducing memory access overhead in the process of determining attention output vector and improving the inference speed of large models.

[0006] Firstly, this application provides a computational method for accelerating inference on large models, the method comprising:

[0007] Obtain the query feature vector of the current word element and the key feature matrix of the historical word element sequence for large-scale model inference;

[0008] Based on the query feature vector of the current word and the key feature matrix of the historical word sequence, determine the attention weight vector of the current word relative to the historical word sequence.

[0009] The key weight vector of the current word is determined based on the attention weight vector and the key feature matrix of the historical word sequence.

[0010] The attention output vector for the current word is determined based on the key weight vector and the preset key value transformation matrix.

[0011] In some embodiments, obtaining the query feature vector of the current lexical unit and the key feature matrix of the historical lexical unit sequence for large model inference includes:

[0012] Embedding is performed on each historical word in the historical word sequence to obtain the embedding vector of each historical word;

[0013] The embedding vectors of each historical word are concatenated to obtain the embedding matrix of the historical word sequence;

[0014] The embedding vector of the current word element is processed using a preset query projection matrix to obtain the query feature vector of the current word element;

[0015] The embedding matrix of the historical word sequence is processed using a preset key projection matrix to obtain the key feature matrix of the historical word sequence.

[0016] In some embodiments, the query projection matrix includes multiple sub-query projection matrices. The embedding vector of the current word is processed using a preset query projection matrix to obtain the query feature vector of the current word, including:

[0017] The embedding vector of the current word is processed by the projection matrix of each subquery to obtain the feature vector of each subquery of the current word;

[0018] The query feature vector of the current word is obtained by concatenating the sub-query feature vectors of the current word.

[0019] In some embodiments, the key projection matrix includes multiple sub-key projection matrices; the embedding matrix of the historical word sequence is processed using a preset key projection matrix to obtain the key feature matrix of the historical word sequence, including:

[0020] The embedding matrix of the historical word sequence is processed by each sub-key projection matrix to obtain the sub-key feature matrix of the historical word sequence;

[0021] By concatenating the sub-key feature matrices of the historical word sequence, the key feature matrix of the historical word sequence is obtained.

[0022] In some embodiments, the method further includes:

[0023] Obtain the preset key projection matrix and the preset value projection matrix;

[0024] Invert the key projection matrix to obtain the inverse key projection matrix;

[0025] The matrix product between the inverse key projection matrix and the value projection matrix is ​​determined as the key-value transformation matrix.

[0026] In some embodiments, the attention weight vector of the current word element relative to the historical word element sequence is determined based on the query feature vector of the current word element and the key feature matrix of the historical word element sequence, including:

[0027] Obtain the sub-query feature vectors in the query feature vector of the current word element and the sub-key feature matrices in the key feature matrix of the historical word element sequence;

[0028] Based on the feature vectors of each sub-query of the current word and the feature matrices of each sub-key of the corresponding historical word sequence, determine the attention weight vector of each sub-word.

[0029] The attention weight vector is determined based on the attention weight vectors of each sub-attention vector.

[0030] In some embodiments, the key weight vector of the current word is determined based on the attention weight vector and the key feature matrix of the historical word sequence, including:

[0031] The sub-key weight vector is determined based on each sub-attention weight vector in the attention weight vector and the sub-key feature matrix of the corresponding historical word sequence.

[0032] Determine the key weight vector based on the weight vectors of each sub-key.

[0033] In some embodiments, the attention output vector of the current lexical is determined based on the key weight vector and a preset key value transformation matrix, including:

[0034] Based on the sub-key weight vectors in the key weight vector and the sub-key value transformation matrices in the corresponding key value transformation matrix, determine the sub-attention output vectors of the current word relative to the historical word sequence; each sub-key value transformation matrix is ​​determined based on the sub-key projection matrices in the preset key projection matrix and the sub-value projection matrices in the corresponding preset value projection matrix.

[0035] The attention output vector is determined based on the output vectors of each sub-attention.

[0036] Secondly, this application provides a computing device for accelerating large-scale model inference, the device comprising:

[0037] The data storage module is used to store the query feature vector of the current word element, the key feature matrix of the historical word element sequence, and the key value transformation matrix for large model inference.

[0038] The processing module is used to determine the attention weight vector of the current word relative to the historical word sequence based on the query feature vector of the current word and the key feature matrix of the historical word sequence; to determine the key weight vector of the current word based on the attention weight vector and the key feature matrix of the historical word sequence; and to determine the attention output vector of the current word based on the key weight vector and the preset key value transformation matrix.

[0039] In some embodiments, the processing module includes:

[0040] The data scheduling unit is used to schedule the key feature matrix of the historical word sequence to the operand transpose unit and the processing unit array, respectively.

[0041] The operand transpose unit is used to transpose the key feature matrix of the historical word sequence to obtain and output the key feature transpose matrix of the historical word sequence to the processing unit array;

[0042] The processing unit array is used to determine and output the attention score vector of the current word relative to the historical word sequence to the normalized index unit based on the query feature vector of the current word and the key feature transpose matrix of the historical word sequence.

[0043] The normalized index unit is used to determine and output the attention weight vector to the processing unit array based on the attention score vector;

[0044] The processing unit array is also used to determine the key weight vector based on the attention weight vector and the key feature matrix of the historical word sequence, and to determine the attention output vector of the current word relative to the historical word sequence based on the key weight vector and the key value transformation matrix.

[0045] In the technical solution provided in this application embodiment, when calculating the attention output vector, only the key feature matrix of the historical word sequence needs to be stored, and the value feature matrix of the historical word sequence does not need to be stored. This reduces the cache storage space required for the historical word sequence feature matrix by half, thereby significantly reducing cache usage during the calculation of the attention output vector. Furthermore, given the attention weight vector, the key weight vector of the current word is determined based on the attention weight vector and the key feature matrix of the historical word sequence. The attention output vector of the current word is also determined based on the key weight vector and a preset key-value transformation matrix. Therefore, the attention output vector of the current word consists entirely of vector-matrix multiplication calculations. While this is less complex than some technologies that use pre-stored value feature matrices of the historical word sequence for multiplication, this approach still offers advantages over some other methods. The approach of determining the attention output vector of the current word element through matrix multiplication involves an additional vector-matrix multiplication. However, this embodiment does not require calculating the value feature matrix of the historical word element sequence. Since the calculation of the value feature matrix of the historical word element sequence in related technologies requires the use of the historical word element quantity length vector, the computational cost of the additional vector-matrix multiplication is less than that of the historical word element sequence value feature matrix calculation in related technologies when the historical word element sequence is long. Therefore, this embodiment also reduces the computational cost of determining the attention output vector. Thus, this embodiment can reduce the computational cost and significantly reduce cache usage while determining the attention output vector, thereby reducing memory access overhead and improving the inference speed of large models. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A flowchart illustrating a computational method for accelerating inference on large models, provided for some embodiments;

[0048] Figure 2 A flowchart illustrating a method for obtaining the query feature vector of the current word in large model inference and the key feature matrix of the historical word sequence, provided in some embodiments;

[0049] Figure 3 A flowchart illustrating a method for determining a key-value transformation matrix provided in some embodiments;

[0050] Figure 4A schematic diagram of the data flow for the computational method for accelerating large model inference provided in the first embodiment;

[0051] Figure 5 A schematic diagram of the data flow for the computational method for accelerating large model inference provided in the second embodiment;

[0052] Figure 6 A schematic diagram of the structure of a computing device for accelerating large model inference provided in the first embodiment;

[0053] Figure 7 A schematic diagram of the computing device for accelerating large model inference provided in the second embodiment;

[0054] Figure 8 A schematic diagram of the structure of a computer device provided for some embodiments. Detailed Implementation

[0055] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0057] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, "multiple groups" means two or more, and "each" means each of the multiple, unless otherwise explicitly defined.

[0058] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0059] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0060] Unless otherwise specified, the order of execution steps in the embodiments of this application is not limited. It should also be noted that any step in the embodiments of this application can be executed independently, that is, the execution of any step in the above embodiments can be performed without depending on the execution of other steps.

[0061] In current large-scale model inference, a key-value cache (KV Cache) mechanism is typically used to calculate the attention output vector. This approach usually requires storing the key feature matrix and value feature matrix of the historical word sequence, and then determining the attention output vector of the current word relative to the entire historical word sequence based on these stored matrix values.

[0062] However, as the historical word sequence continues to grow, the size of its key feature matrix and value feature matrix also continues to increase. Storing such a large key feature matrix and value feature matrix in the cache will lead to excessive cache usage during the calculation of the attention output vector.

[0063] Based on this, the embodiments of this application provide a computational method for accelerating inference in large models, which can reduce the cache usage during the calculation of attention output vectors.

[0064] The method in this application embodiment can be applied to computing devices for accelerating large-model inference. These computing devices can be computer equipment or devices within computing chips. The computer equipment or computing chips can be deployed locally or in the cloud.

[0065] Figure 1 The flowchart illustrates a computational method for accelerating inference on large models, as provided in some embodiments. Figure 1 As shown, the method includes the following steps:

[0066] S101. Obtain the query feature vector of the current word element and the key feature matrix of the historical word element sequence for large model inference.

[0067] For example, the large model can be a deep learning model based on the Transformer architecture. A token, also called a lexical unit, is the basic unit of text processing; it can be a character, a word, or a subword. The current token is the most recently generated token by the large model during inference. The historical token sequence is a sequence consisting of all tokens generated before the large model generated the current token and all tokens received by the large model. The tokens received by the large model can be user input.

[0068] The query feature vector of the current word is the embedding vector of the current word, which is linearly projected onto a preset query projection matrix to obtain a one-dimensional feature vector. The embedding vector of the current word has a dimension of 1×M, and the query projection matrix has a dimension of M×N, so the query feature vector of the current word has a dimension of 1×N.

[0069] The key feature matrix of the historical word sequence is obtained by projecting the embedding vectors of all historical words onto a predefined key projection matrix. The embedding vector of each historical word has a dimension of 1×M, the key projection matrix has a dimension of M×N, and the key feature matrix of the historical word sequence has a dimension of K×N. Here, K represents the number of historical words in the historical word sequence.

[0070] S102. Based on the query feature vector of the current word element and the key feature matrix of the historical word element sequence, determine the attention weight vector of the current word element relative to the historical word element sequence.

[0071] The attention weight vector represents the degree of association between the current word and each historical word in the sequence, i.e., the importance of each historical word to the current word. Each element in the attention weight vector corresponds to the weight value of a historical word.

[0072] In some embodiments, S102 may include: transposing the key feature matrix of the historical word sequence to obtain the key feature transpose matrix of the historical word sequence; determining the attention score vector of the current word relative to the historical word sequence by multiplying the query feature vector of the current word by the key feature transpose matrix of the historical word sequence; and processing the attention score vector using the Softmax function to obtain the attention weight vector. Wherein, the dimension of the key feature transpose matrix of the historical word sequence is N×K, the dimension of the attention score vector is 1×K, and the dimension of the attention weight vector is 1×K.

[0073] S103. Determine the key weight vector of the current word based on the attention weight vector and the key feature matrix of the historical word sequence.

[0074] The key weight vector of the current word represents the weight vector of the current word after weighted aggregation of the key feature matrix of the historical word sequence in the attention mechanism of the large model.

[0075] In some embodiments, S103 may include: determining the key weight vector of the current word by multiplying the attention weight vector with the key feature matrix of the historical word sequence. The key weight vector of the historical word sequence has a dimension of 1×N.

[0076] S104. Determine the attention output vector for the current word based on the key weight vector and the preset key value transformation matrix.

[0077] The key-value transformation matrix is ​​used to convert the key projection matrix into the value projection matrix.

[0078] The attention output vector of the current word represents the feature vector that is finally output after the current word has undergone a complete attention interaction with the historical word sequence in the attention mechanism of the large model, and incorporates historical context information.

[0079] In some embodiments, S104 may include: determining the attention output vector by multiplying the key weight vector with a preset key value transformation matrix. The key value transformation matrix has a dimension of N×N, and the attention output vector has a dimension of 1×N.

[0080] In some embodiments, after S104, the following steps may be included: determining the next word element output by the large model based on the attention output vector of the current word element. Here, the next word element is the word element following the current word element. Exemplarily, the attention output vector of the current word element is input into a feedforward network layer for nonlinear transformation to obtain an intermediate feature vector; the intermediate feature vector is input into an output projection layer for linear mapping to obtain the logical value vectors of each candidate word element corresponding to the current word element; and the next word element output by the large model is determined based on the logical value vectors.

[0081] In the technical solution provided in this application embodiment, when calculating the attention output vector, only the key feature matrix of the historical word sequence needs to be stored, and the value feature matrix of the historical word sequence does not need to be stored. This reduces the cache storage space required for the historical word sequence feature matrix by half, thereby significantly reducing cache usage during the calculation of the attention output vector. Furthermore, given the attention weight vector, the key weight vector of the current word is determined based on the attention weight vector and the key feature matrix of the historical word sequence. The attention output vector of the current word is also determined based on the key weight vector and a preset key-value transformation matrix. Therefore, the attention output vector of the current word consists entirely of vector-matrix multiplication calculations. Although this is less complex than some technologies that store the attention weights... The approach of determining the attention output vector of the current word by multiplying the vector by the value feature matrix of the pre-stored historical word sequence has an extra vector-matrix multiplication calculation. However, the embodiment of this application does not require calculating the value feature matrix of the historical word sequence. When the length of the historical word sequence is large, the computational cost of the extra vector-matrix multiplication calculation is less than that of calculating the value feature matrix of the historical word sequence in related technologies. Therefore, the embodiment of this application also reduces the computational cost of determining the attention output vector. Thus, the embodiment of this application can reduce the computational cost and greatly reduce cache usage in the process of determining the attention output vector, thereby reducing the memory access overhead in the process of determining the attention output vector and improving the inference speed of large models.

[0082] Figure 2 The flowchart illustrates a method for obtaining the query feature vector of the current lexical unit and the key feature matrix of the historical lexical unit sequence for large-scale model inference, as provided in some embodiments. Figure 2 As shown, this method is an exemplary interpretation of S101, and the method includes the following steps:

[0083] S1011. Embedding is performed on each historical word in the historical word sequence to obtain the embedding vector of each historical word.

[0084] Embedding processing refers to the process of mapping discrete and symbolic lexical identifiers (such as lexical IDs) to a continuous and dense vector space. The embedding vectors of each historical lexical can respectively represent the semantic information and syntactic features of each historical lexical.

[0085] The embedding vector of each lexical unit (including the current lexical unit or historical lexical units) has a dimension of 1×M.

[0086] S1012. Concatenate the embedding vectors of each historical word to obtain the embedding matrix of the historical word sequence.

[0087] For example, if the historical word sequence includes K historical words, then the 1×M-sized embedding vectors of each of the K historical words are concatenated to obtain an embedding matrix of the historical word sequence with a size of K×M.

[0088] S1013. The embedding vector of the current word element is processed using a preset query projection matrix to obtain the query feature vector of the current word element.

[0089] The dimension of the query projection matrix is ​​M×N.

[0090] In some embodiments, S1013 includes: determining the query feature vector (dimension 1×N) of the current word as the matrix product between the embedding vector (dimension 1×M) of the current word and the query projection matrix.

[0091] In other embodiments, the query projection matrix includes multiple sub-query projection matrices, and S1013 includes: processing the embedding vector of the current word element using each sub-query projection matrix to obtain each sub-query feature vector of the current word element; concatenating each sub-query feature vector of the current word element to obtain the query feature vector of the current word element.

[0092] In the multi-head attention mechanism, the pre-defined query projection matrix is ​​divided into multiple sub-matrices, each corresponding to an attention head. Each sub-query projection matrix has independent parameters, which are learned separately during model training and are used to map the embedding vector of the current word to the query subspace of the corresponding attention head.

[0093] The subquery feature vector is the feature vector obtained by processing the embedding vector of the current word using the subquery projection matrix of a certain attention head.

[0094] When concatenating the subquery feature vectors of the current term, the subquery feature vectors can be linked end-to-end according to the numbering order of the attention heads to form a complete query feature vector for the current term. The dimension of this query feature vector is equal to the sum of the dimensions of all subquery feature vectors. For example, for an 8-head attention, the dimension of each subquery feature vector is 1×(N / 8). Concatenating the subquery feature vectors in the row direction yields a query feature vector with a dimension of 1×N.

[0095] S1014. The embedding matrix of the historical word sequence is processed using a preset key projection matrix to obtain the key feature matrix of the historical word sequence.

[0096] The dimension of the key projection matrix is ​​M×N.

[0097] In some embodiments, S1014 includes: determining the key feature matrix (with dimensions of K×N) of the historical word sequence by multiplying the embedding matrix (K×M) of the historical word sequence with the key projection matrix.

[0098] In other embodiments, the key projection matrix includes multiple sub-key projection matrices; S1014 includes: processing the embedding matrix of the historical word sequence using each sub-key projection matrix to obtain the sub-key feature matrix of the historical word sequence; and concatenating the sub-key feature matrices of the historical word sequence to obtain the key feature matrix of the historical word sequence.

[0099] In the multi-head attention mechanism, the pre-defined key projection matrix is ​​divided into multiple sub-matrices, each corresponding to an attention head. Each sub-key projection matrix has independent parameters, which are learned separately during model training and used to map the embedding matrix of the historical word sequence to the key subspace of the corresponding attention head.

[0100] The subkey feature matrix is ​​the feature matrix obtained by processing the embedding matrix of the historical word sequence using the subkey projection matrix of a certain attention head.

[0101] When concatenating the sub-key feature matrices of a historical lexical sequence, the sub-key feature matrices can be concatenated column-wise according to the attention head numbering order to obtain the key feature matrix. For example, for an 8-head attention system, the dimension of each sub-key feature matrix is ​​M×(N / 8). Concatenating these sub-key feature matrices column-wise yields a key feature matrix of dimension M×N. The number of columns in this key feature matrix is ​​equal to the sum of the number of columns in all sub-key feature matrices.

[0102] In this embodiment, for different dialogue tasks using the same large model, the key-value transformation matrix, query projection matrix, and key projection matrix are all fixed parameters of the model and are kept consistent. In this embodiment, the key-value transformation matrix, query projection matrix, and key projection matrix are all stored in a computing device designed to accelerate inference for large models.

[0103] Compared to related technologies that require storing query projection matrices, key projection matrices, and value projection matrices, this application embodiment converts the storage of value projection matrices into the storage of key-value transformation matrices, thereby reducing the storage space of the projection matrix compared to related technologies.

[0104] By concatenating the embedding vectors of each historical word, an embedding matrix of the historical word sequence is obtained. This embedding matrix preserves the complete temporal and semantic information of the historical word sequence, providing a structured data foundation for subsequent projection processing. Processing the embedding vector of the current word using a pre-defined query projection matrix maps the current word's embedding vector to the query space of the attention mechanism, ensuring that the query feature vector of the current word and the key feature matrix of the historical word sequence are in the same semantic space, thus facilitating subsequent calculation of attention weights. Processing the embedding matrix of the historical word sequence using a pre-defined key projection matrix maps the historical word sequence's embedding matrix to the key space of the attention mechanism, obtaining the key feature matrix of the historical word sequence. Through this method, both the query feature vector of the current word and the key feature matrix of the historical word sequence are obtained from the original embedding vectors through linear projection. They are comparable in the same semantic space, laying the foundation for accurately calculating the attention weight vector of the current word relative to the historical word sequence, while avoiding the semantic bias caused by directly calculating the similarity of the original embedding vectors.

[0105] Figure 3 A flowchart illustrating a method for determining a key-value transformation matrix provided in some embodiments, such as... Figure 3 As shown, the method includes steps S301 to S303.

[0106] S301. Obtain the preset key projection matrix and the preset value projection matrix.

[0107] The key projection matrix and the preset value projection matrix can be pre-stored in a computing device for accelerating large model inference.

[0108] S302. Perform an inverse operation on the bond projection matrix to obtain the inverse bond projection matrix.

[0109] In some embodiments, the inverse bond projection matrix can be obtained by inverting the bond projection matrix using mathematical methods.

[0110] In some embodiments, when the key projection matrix is ​​a square matrix, the inverse of the key projection matrix is ​​determined as the inverse key projection matrix. In other embodiments, when the key projection matrix is ​​not a square matrix, a pseudo-inverse operation is performed on the key projection matrix to obtain the inverse key projection matrix.

[0111] Taking a key projection matrix of dimension M×N as an example, exemplarily, performing a pseudo-inverse operation on the key projection matrix to obtain the inverse key projection matrix can include: performing singular value decomposition on the key projection matrix to obtain a left singular orthogonal matrix U (dimension M×N), a diagonal singular value matrix W (dimension N×N), and a right singular orthogonal matrix V (dimension N×N); taking the reciprocal of the non-zero elements of the singular value matrix Σ and transposing it to obtain the pseudo-inverse diagonal matrix W.+ VW + U T The calculation result is determined as the inverse bond projection matrix. The dimension of the inverse bond projection matrix is ​​N×M.

[0112] In other embodiments, each subkey projection matrix in the key projection matrix is ​​a square matrix. The inverse of each subkey projection matrix is ​​performed to obtain each sub-inverse key projection matrix. These sub-inverse key projection matrices are then concatenated to obtain the inverse key projection matrix. Each subkey projection matrix corresponds one-to-one with each subvalue projection matrix.

[0113] S303. The matrix product between the inverse key projection matrix and the value projection matrix is ​​determined as the key-value transformation matrix.

[0114] The value projection matrix has a dimension of M×N. Multiplying the inverse key projection matrix and the value projection matrix yields a key-value transformation matrix of dimension N×N.

[0115] By obtaining a preset key projection matrix and a preset value projection matrix, and inverting the key projection matrix to obtain the inverse key projection matrix, the matrix product between the inverse key projection matrix and the value projection matrix is ​​determined as the key-value transformation matrix. This key-value transformation matrix mathematically establishes a mapping relationship from the key space to the value space. During inference, based on the query feature vector of the current word and the key feature matrix of the historical word sequence, the attention weight vector of the current word relative to the historical word sequence is determined. Based on this attention weight vector, the key feature matrix of the historical word sequence is processed to obtain the key weight vector. Multiplying this key weight vector by the key-value transformation matrix yields the attention output vector of the current word. Throughout the determination of the attention output vector, it is not necessary to store the value feature matrix of the historical word sequence in the cache; only the key feature matrix of the historical word sequence needs to be stored to complete the calculation of the attention output vector. Since the key feature matrix and the value feature matrix have the same matrix size, this method can reduce the cache usage by about half, effectively alleviating the problem of excessive cache usage caused by the continuous growth of historical word sequences during large model inference. This enables the inference task of longer historical word sequences to be supported under limited storage resources, thus improving the feasibility of large model inference.

[0116] In other embodiments, the key-value transformation matrix can be obtained through training. For example, during the training of a large model, the key-value transformation matrix is ​​used instead of the value projection matrix for calculation, thereby training the key-value transformation matrix.

[0117] In some embodiments, determining the attention weight vector of the current word relative to the historical word sequence based on the query feature vector of the current word and the key feature matrix of the historical word sequence includes: obtaining each sub-query feature vector in the query feature vector of the current word and each sub-key feature matrix in the key feature matrix of the historical word sequence; determining each sub-attention weight vector based on each sub-query feature vector of the current word and the corresponding sub-key feature matrix of the historical word sequence; and determining the attention weight vector based on each sub-attention weight vector.

[0118] Among them, each subquery feature vector is obtained by processing the embedding vector of the current word element by each subquery projection matrix in the query projection matrix, and each subkey feature matrix is ​​obtained by processing the embedding matrix of the historical word element sequence by each subkey projection matrix in the key projection matrix.

[0119] Each subquery projection matrix corresponds one-to-one with each subkey projection matrix.

[0120] By obtaining the sub-query feature vectors from the query feature vector of the current word and the sub-key feature matrices from the key feature matrix of the historical word sequence, the sub-attention weight vectors are determined based on the sub-query feature vectors of the current word and the corresponding sub-key feature matrices of the historical word sequence. Then, the attention weight vector is determined based on these sub-attention weight vectors. This allows the determination of the attention weight vector to be completed independently in each attention head of the multi-head attention mechanism. Each attention head corresponds to a different semantic subspace. The sub-query feature vectors and sub-key feature matrices in each attention head are used to calculate similarity within that subspace, thus obtaining the sub-attention weight vector for that subspace. The attention weight vector obtained by combining these sub-attention weight vectors can simultaneously capture the degree of association between the historical word sequence and the current word in different semantic subspaces. This avoids the semantic loss caused by prematurely fusing information from different attention heads, improving the accuracy and richness of the association between the current word and the historical word sequence expressed by the attention weight vector. This provides a more precise weight basis for subsequently determining the key weight vector and the attention output vector.

[0121] In some embodiments, determining the key weight vector of the current lexical based on the attention weight vector and the key feature matrix of the historical lexical sequence includes: determining each sub-key weight vector based on each sub-attention weight vector in the attention weight vector and each sub-key feature matrix of the corresponding historical lexical sequence; and determining the key weight vector based on each sub-key weight vector.

[0122] The key weight vector is determined by analyzing the sub-attention weight vectors within the attention weight vector and the corresponding sub-key feature matrices of the historical lexical sequence. Then, the key weight vector is determined based on these sub-key weight vectors. This process allows the determination of the key weight vector to be completed independently within each attention head of the multi-head attention mechanism. Each sub-attention weight vector is a weight distribution calculated in the semantic subspace of its respective attention head, and each sub-key feature matrix is ​​a feature representation in the key subspace of its respective attention head. The sub-attention weight vectors within the same attention head are weighted and summed with their corresponding sub-key feature matrices to obtain the key weight vector for that attention head. This key weight vector accurately reflects the aggregation result of the current lexical sequence's key information within that semantic subspace. The key weight vector obtained by combining the sub-key weight vectors can simultaneously retain the key information extracted from the historical word sequence of the current word in each semantic subspace in a unified vector representation. This avoids the information loss caused by prematurely fusing the key aggregation results of each attention head, improves the integrity and fidelity of the key information carried by the key weight vector, and provides a more accurate input for determining the attention output vector through the key value transformation matrix.

[0123] In some embodiments, determining the attention output vector of the current lexical unit based on the key weight vector and a preset key value transformation matrix includes: determining each sub-attention output vector of the current lexical unit relative to the historical lexical unit sequence based on each sub-key weight vector in the key weight vector and each sub-key value transformation matrix in the corresponding key value transformation matrix; determining each sub-key value transformation matrix based on each sub-key projection matrix in the preset key projection matrix and each sub-value projection matrix in the corresponding preset value projection matrix; and determining the attention output vector based on each sub-attention output vector.

[0124] Each sub-attention output vector is determined based on the sub-key weight vectors in the key weight vector and the corresponding sub-key value transformation matrices in the key value transformation matrix. Then, the attention output vector is calculated using the sub-headings to accelerate large-scale model inference. This allows the determination of the attention output vector to be completed independently in each attention head of the multi-head attention mechanism. Each sub-key value transformation matrix is ​​pre-determined based on the sub-key projection matrix and sub-value projection matrix within the same attention head. This sub-key value transformation matrix establishes the mapping relationship from the key subspace to the value subspace within that attention head. Multiplying the sub-key weight vectors in each attention head with their corresponding sub-key value transformation matrices allows for the independent mapping from the aggregation result in the key space to the value space within that attention head, yielding the sub-attention output vector for that attention head. The attention output vector obtained by combining the sub-attention output vectors can retain the attention output information of each semantic subspace in a unified vector representation. This avoids the information loss caused by prematurely fusing the outputs of each attention head, and enables the final attention output vector to more completely retain the contextual information captured by the multi-head attention mechanism in different semantic subspaces. This improves the representational ability and accuracy of the attention output vector and provides more reliable feature input for subsequent inference of large models.

[0125] In some embodiments, determining the attention output vector based on each sub-attention output vector includes: concatenating the sub-attention output vectors to obtain a concatenated sub-attention vector; and performing a linear projection on the concatenated sub-attention vector to obtain the attention output vector.

[0126] By concatenating the sub-attention output vectors to obtain a concatenated sub-attention vector, and then linearly projecting this concatenated vector to obtain the attention output vector, the sub-attention output vectors independently computed by each attention head in the multi-head attention mechanism can be merged into a unified vector representation. Each sub-attention output vector corresponds to a different semantic subspace. By concatenating these vectors end-to-end according to a preset dimensional order, the contextual information captured by each attention head in its respective semantic subspace can be fully preserved, avoiding information interference between different semantic subspaces caused by vector addition or averaging operations. Linearly projecting the concatenated sub-attention vector maps the vector, which is in a high-dimensional space, to the feature space required by subsequent network layers of the large model. This ensures that the dimension of the output attention vector matches the input dimension of the subsequent network layers. Furthermore, the learnable parameters in the linear projection integrate and transform the information from each attention head, improving the representation quality of the attention output vector. This approach ensures that the final attention output vector retains the diverse information captured by the multi-head attention mechanism in different semantic subspaces, while also enabling it to participate in subsequent model inference computations in a unified vector form, facilitating hardware acceleration and parallel processing.

[0127] Figure 4 A schematic diagram of the data flow for the computational method for accelerating large-model inference provided in the first embodiment, as shown below. Figure 4 As shown, in the first step, the key feature matrix K of the historical word sequence is loaded, the query feature vector q of the current word is loaded, and then p=SoftMax(qK) is calculated. T Let p be the attention weight vector of the current word relative to the historical word sequence; then, calculate j = pK; finally, calculate the attention output vector h of the current word, where h = jW K→V Among them, W K→V This is the key-value transformation matrix. Compared to traditional KV caching methods, the embodiments of this application can reduce KV caching by 50%.

[0128] Figure 5 A schematic diagram of the data flow for the computational method for accelerating large model inference provided in the second embodiment, as shown below. Figure 5 As shown, the first step is to calculate the transpose of vector q and matrix K and multiply them to obtain a = qK. T Then, the SoftMax module processes a to obtain p = SoftMax(a). The second step is to calculate the product of vector p and matrix K to obtain the intermediate result vector j. The third step is to calculate the intermediate result vector j and W. K→VMatrix multiplication yields the attention output vector h for the current word. The computational data stream consists of only three matrix-vector multiplications and one SoftMax computation, without introducing matrix-matrix multiplications, resulting in minimal computational overhead.

[0129] Based on the same concept, this application also provides a computing device for accelerating large-model inference, which implements the above-described computing method for accelerating large-model inference. The solution provided by this device is similar to the solution described in the above-described method. Therefore, the specific limitations of one or more embodiments of the computing device for accelerating large-model inference provided below can be found in the limitations of the computing method for accelerating large-model inference described above, and will not be repeated here.

[0130] In one exemplary embodiment, Figure 6 This is a schematic diagram of the computing device for accelerating large model inference provided in the first embodiment, as shown below. Figure 6 As shown, the computing device 600 for accelerating large model inference includes:

[0131] The data storage module is used to store the query feature vector of the current word element, the key feature matrix of the historical word element sequence, and the key value transformation matrix for large model inference.

[0132] The processing module is used to determine the attention weight vector of the current word relative to the historical word sequence based on the query feature vector of the current word and the key feature matrix of the historical word sequence; to determine the key weight vector of the current word based on the attention weight vector and the key feature matrix of the historical word sequence; and to determine the attention output vector of the current word based on the key weight vector and the preset key value transformation matrix.

[0133] In some embodiments, the processing module includes:

[0134] The data scheduling unit is used to schedule the key feature matrix of the historical word sequence to the operand transpose unit and the processing unit array, respectively.

[0135] The operand transpose unit is used to transpose the key feature matrix of the historical word sequence to obtain and output the key feature transpose matrix of the historical word sequence to the processing unit array;

[0136] The processing unit array is used to determine and output the attention score vector of the current word relative to the historical word sequence to the normalized index unit based on the query feature vector of the current word and the key feature transpose matrix of the historical word sequence.

[0137] The normalized index unit is used to determine and output the attention weight vector to the processing unit array based on the attention score vector;

[0138] The processing unit array is also used to determine the key weight vector based on the attention weight vector and the key feature matrix of the historical word sequence, and to determine the attention output vector of the current word relative to the historical word sequence based on the key weight vector and the key value transformation matrix.

[0139] Figure 7 A schematic diagram of the computing device for accelerating large model inference provided in the second embodiment is shown below. Figure 7 As shown, the data storage module includes a first storage unit and a second storage unit. The first storage unit stores the key feature matrix and key-value transformation matrix of historical word sequences, while the second storage unit stores the query feature vector and attention weight vector of the current word. The computing device for accelerating large model inference also includes an output unit.

[0140] The data scheduling unit is used to obtain the key feature matrix of the historical word sequence from the first storage unit, schedule the key feature matrix of the historical word sequence to the operand transpose unit and the second storage unit respectively, and schedule the key value transformation matrix obtained from the first storage unit to the second storage unit.

[0141] The operand transpose unit is used to transpose the key feature matrix of the historical word sequence to obtain and output the key feature transpose matrix of the historical word sequence to the processing unit array;

[0142] The processing unit array is used to obtain the query feature vector of the current word from the second storage unit, and determine and output the attention score vector of the current word relative to the historical word sequence to the normalized index unit based on the query feature vector of the current word and the key feature transpose matrix of the historical word sequence.

[0143] The normalized index unit is used to determine and output the attention weight vector to the processing unit array based on the attention score vector; the normalized index unit is also used to output the attention weight vector to the output unit so that the output unit sends the attention weight vector to the second storage unit.

[0144] The processing unit array is also used to obtain the attention weight vector and the key feature matrix of the historical word sequence from the second storage unit, determine the key weight vector, and determine the attention output vector of the current word relative to the historical word sequence based on the key weight vector and the key value transformation matrix obtained from the second storage unit.

[0145] The processing unit array can also output an attention output vector to the second storage unit so that the second storage unit stores the attention output vector.

[0146] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0147] The modules in the aforementioned computing device for accelerating large model inference can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0148] The computing device for accelerating large model inference in this embodiment can be integrated into an artificial intelligence chip fabricated using a 22nm process. The main speed bottleneck and power consumption source for large models during inference is the memory access overhead of the KV cache. In this embodiment, the KV cache recalculation technique (K-to-V) can reduce KV cache usage by 50% without additional encoding / decoding or training overhead. Therefore, when running the large Llama2-7B model for inference, the KV cache recalculation technique (K-to-V) improved the throughput of the artificial intelligence chip by 65.3%, improved system energy efficiency by 41.7%, and reduced inference latency by 38.4% in tests. Since the KV cache recalculation technique (K-to-V) is lossless, the perplexity (an indicator of large model capability) remains essentially the same as the original model when running the Llama2-7B model for inference.

[0149] In one exemplary embodiment, Figure 8This is a schematic diagram of the structure of a computer device provided in some embodiments. The computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wireless Fidelity (WIFI), mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a computational method for accelerating large-model inference. The display unit of the computer device is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0150] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0151] For example, a computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method of any of the above embodiments.

[0152] In one embodiment, a computer-readable storage medium is provided, wherein a computer program, when executed by a processor, implements the steps of the method provided in any of the above embodiments.

[0153] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method provided in any of the above embodiments.

[0154] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the methods described above.

[0155] The processor, processing unit, functional module, or functional unit in any embodiment of this application may include an integration of one or more of the following: a general-purpose processor, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a central processing unit (CPU), a graphics processing unit (GPU), an embedded neural network processing unit (NPU), a controller, a microcontroller, a microprocessor, a programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, a quantum computing-based data processing logic device, an artificial intelligence (AI) processor, etc. Exemplarily, the general-purpose processor may be a microprocessor or any conventional processor.

[0156] The memory or computer-readable storage medium in any embodiment of this application may include at least one of non-volatile memory and volatile memory. Non-volatile memory includes integration of one or more of the following: Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Ferromagnetic Random Access Memory (FRAM), Flash Memory, Magnetic Surface Memory, Optical Disc, Compact Disc Read-Only Memory (CD-ROM), Magnetic Tape, Floppy Disk, Flash Memory, Optical Memory, High-Density Embedded Non-Volatile Memory, Resistive Random Access Memory (ReRAM), Magnetoresistive Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (PCM), Graphene Memory, Volatile Memory, etc. Volatile memory includes one or more of the following: Random Access Memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), etc.

[0157] The acquisition, transmission, storage, use, and processing of data in this application comply with relevant national laws and regulations. It should be noted that certain software, components, models, and other existing industry solutions may be mentioned in the embodiments of this application. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0158] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0159] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A computational method for accelerating inference in large models, characterized in that, The method includes: Obtain the query feature vector of the current word element and the key feature matrix of the historical word element sequence for large-scale model inference; Based on the query feature vector of the current word element and the key feature matrix of the historical word element sequence, determine the attention weight vector of the current word element relative to the historical word element sequence; The key weight vector of the current word is determined based on the attention weight vector and the key feature matrix of the historical word sequence. The attention output vector of the current word is determined based on the key weight vector and the preset key value transformation matrix.

2. The method according to claim 1, characterized in that, The process of obtaining the query feature vector of the current word element and the key feature matrix of the historical word element sequence for large-scale model inference includes: Embedding processing is performed on each historical word in the historical word sequence to obtain the embedding vector of each historical word; The embedding vectors of each historical word are concatenated to obtain the embedding matrix of the historical word sequence; The embedding vector of the current word element is processed using a preset query projection matrix to obtain the query feature vector of the current word element; The embedding matrix of the historical lexical sequence is processed using a preset key projection matrix to obtain the key feature matrix of the historical lexical sequence.

3. The method according to claim 2, characterized in that, The query projection matrix includes multiple sub-query projection matrices. The process of using a preset query projection matrix to process the embedding vector of the current word element to obtain the query feature vector of the current word element includes: The embedding vector of the current word element is processed by each of the sub-query projection matrices to obtain the sub-query feature vectors of the current word element; The query feature vectors of the current word are concatenated to obtain the query feature vector of the current word.

4. The method according to claim 2, characterized in that, The key projection matrix includes multiple sub-key projection matrices; the process of using a preset key projection matrix to process the embedding matrix of the historical lexical sequence to obtain the key feature matrix of the historical lexical sequence includes: The embedding matrix of the historical word sequence is processed using the sub-key projection matrices respectively to obtain the sub-key feature matrix of the historical word sequence; The key feature matrix of the historical word sequence is obtained by concatenating the sub-key feature matrices of the historical word sequence.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Obtain the preset key projection matrix and the preset value projection matrix; Invert the key projection matrix to obtain the inverse key projection matrix; The matrix product between the inverse key projection matrix and the value projection matrix is ​​determined as the key-value transformation matrix.

6. The method according to any one of claims 1-4, characterized in that, The step of determining the attention weight vector of the current word element relative to the historical word element sequence based on the query feature vector of the current word element and the key feature matrix of the historical word element sequence includes: Obtain each sub-query feature vector in the query feature vector of the current word element and each sub-key feature matrix in the key feature matrix of the historical word element sequence; Based on the sub-query feature vectors of the current word and the sub-key feature matrices of the corresponding historical word sequence, determine the sub-attention weight vectors; The attention weight vector is determined based on the attention weight vectors described in each sub-vector.

7. The method according to claim 6, characterized in that, Determining the key weight vector of the current word based on the attention weight vector and the key feature matrix of the historical word sequence includes: Each sub-key weight vector is determined based on each sub-attention weight vector in the attention weight vector and each sub-key feature matrix of the corresponding historical word sequence; The key weight vector is determined based on each of the sub-key weight vectors.

8. The method according to claim 7, characterized in that, Determining the attention output vector of the current lexical unit based on the key weight vector and a preset key value transformation matrix includes: Based on each of the sub-key weight vectors in the key weight vector and each of the corresponding sub-key value transformation matrices in the key value transformation matrix, the sub-attention output vectors of the current word relative to the historical word sequence are determined; each of the sub-key value transformation matrices is determined based on each of the sub-key projection matrices in the preset key projection matrix and each of the sub-value projection matrices in the corresponding preset value projection matrix. The attention output vector is determined based on each of the sub-attention output vectors.

9. A computing device for accelerating large-scale model inference, characterized in that, The device includes: The data storage module is used to store the query feature vector of the current word element, the key feature matrix of the historical word element sequence, and the key value transformation matrix for large model inference. The processing module is configured to determine the attention weight vector of the current word relative to the historical word sequence based on the query feature vector of the current word and the key feature matrix of the historical word sequence; determine the key weight vector of the current word based on the attention weight vector and the key feature matrix of the historical word sequence; and determine the attention output vector of the current word based on the key weight vector and a preset key value transformation matrix.

10. The apparatus according to claim 9, characterized in that, The processing module includes: The data scheduling unit is used to schedule the key feature matrix of the historical word sequence to the operand transpose unit and the processing unit array, respectively. The operand transpose unit is used to transpose the key feature matrix of the historical word sequence to obtain and output the key feature transpose matrix of the historical word sequence to the processing unit array. The processing unit array is used to determine and output the attention score vector of the current word relative to the historical word sequence to the normalized index unit based on the query feature vector of the current word and the key feature transpose matrix of the historical word sequence; The normalized index unit is used to determine and output the attention weight vector to the processing unit array based on the attention score vector. The processing unit array is further configured to determine a key weight vector based on the attention weight vector and the key feature matrix of the historical lexical sequence, and to determine the attention output vector of the current lexical relative to the historical lexical sequence based on the key weight vector and the key value transformation matrix.

Images