Partial fixed caching method and system based on large model inference, device and medium

Abstract

A partial fixed caching method and system based on large model inference, a device, and a medium, relating to the technical fields of artificial intelligence and natural language processing. The technical problem to be solved is how to find a balance between computing resources and storage resources that is more suitable for localized inference of large models, thereby improving inference efficiency. The technical solution used is: a pre-filling stage: inputting a sequence or text, preprocessing same to complete format conversion from the original input sequence or text into a large model input matrix, and caching a K matrix and a V matrix of an original input token in each attention layer on the basis of a KVcache method; and a decoding stage: unlike the conventional KVcache method, before each decoding, pre-determining whether to cache a K matrix and a V matrix of a current input token, and on the basis of the result, determining whether to cache the K matrix and the V matrix of the current input token during the current decoding.

Description

Partial fixed cache method, system, device and medium based on large model inference

[0001] The present application claims priority to the Chinese patent application No. 202411224816.2, filed on September 3, 2024, and entitled "Partial fixed cache method, system, device and medium based on large model inference", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence and natural language processing, in particular to a partial fixed cache method, system, device and medium based on large model inference. BACKGROUND

[0003] With the rapid development of artificial intelligence technology, especially in the field of natural language processing, large-scale pre-training models (such as Transformer models) have shown excellent performance in various tasks. However, these models require a large amount of computing resources and storage resources during inference. Traditional Transformer models use a method of storing a large number of key-value pairs to speed up text generation during inference, resulting in a linear correlation between memory consumption and input sequence length and batch size. This method is a typical strategy of using storage resources to replace computing resources.

[0004] There is nothing wrong with using large-scale clusters, but when dealing with customer local deployment and running requirements, customers cannot have large-scale clusters for model deployment and inference. Therefore, the storage requirements of KVcache have become a bottleneck in the inference process.

[0005] Therefore, how to find a balance between computing resources and storage resources that is more suitable for large model local inference and improve inference efficiency is a technical problem to be solved. SUMMARY

[0006] The technical task of the present application is to provide a partial fixed cache method, system, device and medium based on large model inference to solve the problem of how to find a balance between computing resources and storage resources that is more suitable for large model local inference and improve inference efficiency.

[0007] The technical task of the present application is achieved in the following manner, a partial fixed cache method based on large model inference, the method is as follows:

[0008] Pre-filling stage: input sequence or text, pre-processing is completed to convert the original input sequence or text into the format of the large model input matrix, and based on the KVcache method, cache the K matrix and V matrix of the original input token in each attention layer, and convert the input sequence or text into a large model input matrix represented by a fixed length feature vector for each token, and the input matrix is output through model inference The first token;

[0009] Decoding stage: unlike the traditional KVcache method, before each decoding, a pre-judgment is made on whether to cache the K matrix and V matrix of the current input token, and according to the result, it is determined whether to cache the K matrix and V matrix of the current input token in the current decoding process.

[0010] As a preferred, the pre-processing process of the pre-filling stage is as follows:

[0011] tokenizer: convert the original text or sequence into a numerical representation of text composed of tokens, that is, convert the input text into input tokens;

[0012] embedding: convert the input token into a word vector encoding as the input of the model, and fuse the position vector encoding into the word vector encoding to reflect the position correlation of the input token;

[0013] If the input size and the large model input do not match, reshape the input size.

[0014] As a preferred, before the decoding stage, compare the number of current input tokens with the number of tokens n t that can be concurrently executed by the device at a time:

[0015] If the number of output tokens exceeds the number of tokens n t that can be concurrently executed by the device at a time, do not cache the K vector and V vector corresponding to the current input token, and continue to accumulate the tokens output by the large model for the next decoding;

[0016] If the number of input tokens exceeds the number of tokens n t that can be concurrently executed, continue to compare the time T l when the cached K matrix and V matrix are loaded into the computing core and the time T d when the current input token is decoded once:

[0017] If T d < T lIf T

[0018] If T d > T l , the K matrix and the V matrix corresponding to the current input token are cached, and the new output token of the current decoding is used to continue decoding.

[0019] More preferably, the number of tokens n t that the device can concurrently execute at a time is calculated as follows:

[0020] n t = theoretical peak computing power / computational load of the most computationally intensive network layer, that is, the time for the computing device to decode less than n t tokens and the time to decode n t tokens are consistent.

[0021] More preferably, the time T l loaded into the computing core is calculated as follows:

[0022] T l = (2 * d * l / memory bandwidth) * N

[0023] where d is the dimension of a single K or V vector, since in the Transformer model, the length of each K matrix and V matrix is the same, so multiply by 2; l is the number of attention layers in the large model; N is the number of all tokens before the current decoding.

[0024] More preferably, the time T d of a single decoding is calculated as follows:

[0025] T d = current input token number * computational load per token / theoretical peak computing power.

[0026] A partial fixed caching system based on large model inference, the system comprises:

[0027] A pre-filling module for inputting a sequence or text, converting the original input sequence or text to a large model input matrix format, and caching the K matrix and V matrix of the original input token in each attention layer based on the KVcache method, and converting the input sequence or text into a large model input matrix represented by a fixed-length feature vector for each token, and the input matrix is output through model inference to obtain the first token.

[0028] The decoding module, unlike the traditional KVcache method, pre-determines whether to cache the K and V matrices of the current input token before each decoding, and determines whether to cache the K and V matrices of the current input token during the current decoding process based on the result.

[0029] Preferably, before the decoding stage, the number of tokens currently input is compared with the number of tokens n that the device can execute concurrently in a single operation. t :

[0030] If the number of output tokens exceeds the number of tokens n that the device can execute concurrently in a single operation. t If so, the K vector and V vector corresponding to the current input token are not cached, and the tokens output by the large model are accumulated for the next decoding.

[0031] If the number of input tokens exceeds the number of tokens that can be executed concurrently, n t Then continue comparing the time T of loading the cached K matrix and V matrix into the computation core. l and the time T for decoding the current input token once. d :

[0032] If T d <T l If so, the K and V matrices corresponding to the current input token are not cached, and the tokens output by the large model are accumulated for the next decoding.

[0033] If T d >T l If so, the K and V matrices of the current input token are cached, and the new output token from this decoding is used to continue decoding;

[0034] Among them, n is the number of tokens that can be executed concurrently by the device in a single instance. t The formula is as follows:

[0035] n t = Theoretical peak computing power / Computational cost of the most computationally intensive network layer, i.e., word decoding on the computing device is less than n t The time and decoding of each token n t The tokens have the same time;

[0036] Time T for loading to the computing core l The formula is as follows:

[0037] T l = (2 * d * l / memory bandwidth) * N;

[0038] Wherein, d is the dimension of a single K or V vector, multiplied by 2 since the length of each K matrix and V matrix is the same in the Transformer model; l is the number of attention layers in the large model; N is the number of all tokens before the current decoding;

[0039] The time T of one decoding d The formula is as follows:

[0040] T d = current input token number * calculation amount per token / theoretical peak computing power.

[0041] An electronic device comprising a memory and at least one processor;

[0042] Wherein, the memory has a computer program stored thereon;

[0043] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the partial fixed cache method based on large model inference as described above.

[0044] A computer readable storage medium, the computer readable storage medium has a computer program stored therein, the computer program can be executed by a processor to implement the partial fixed cache method based on large model inference as described above.

[0045] The partial fixed cache method based on large model inference, system, device and medium of the application has the following advantages:

[0046] (I) The present application finds a balance between computing resources and storage resources that is more suitable for large model localization inference, improving inference efficiency;

[0047] (II) In the inference process, it is dynamically determined whether to perform KV matrix caching, which can reduce the total inference time, avoid memory overload, and further improve the utilization of computing devices. BRIEF DESCRIPTION OF DRAWINGS

[0048] The present application will be further described below in conjunction with the accompanying drawings.

[0049] Figure 1 is a flow chart of the decoding stage in the partial fixed cache method based on large model inference. DETAILED DESCRIPTION

[0050] The partial fixed cache method based on large model inference, system, device and medium of the application are described in detail below with reference to the accompanying drawings and specific examples.

[0051] Example 1:

[0052] The embodiment provides a partial fixed cache method based on large model inference, and the method is specifically as follows:

[0053] S1, pre-filling stage: input sequence or text, pre-processing is completed to convert the original input sequence or text into a large model input matrix format, and based on the KVcache method, the K matrix and the V matrix of the original input token in each attention layer are cached, and the input sequence or text is converted into a large model input matrix in which each token is represented by a fixed length feature vector, and the input matrix is outputted through model inference to obtain the first token;

[0054] S2, decoding stage: different from the traditional KVcache method, whether the K matrix and the V matrix of the current input token are cached is pre-judged before each decoding, and whether the K matrix and the V matrix of the current input token are cached in the current decoding process is determined according to the result.

[0055] The large model is composed of a plurality of self-attention neural network (Self-Attention Neural Networks) modules stacked together, the modules are mainly in the form of a transformer structure, different large models slightly differ in the network layers included in each module, but basically include an attention layer (Attention Layer), a feedforward neural network (Feedforward Neural Networks), a normalization layer (Normalization Layer) and a residual connection layer (Residual Connection).

[0056] The pre-processing process in the pre-filling stage in step S1 of the embodiment is specifically as follows:

[0057] S101, tokenizer: converting the original text or sequence into an initial numerical representation of the text composed of tokens, that is, converting the input text into input tokens;

[0058] S102, embedding: converting the input token into a word vector encoding (word embedding) as the input of the model, and fusing a position vector encoding (position embedding) into the word vector encoding to reflect the position correlation of the input token;

[0059] If the input size and the large model input do not match, the input size is reshaped.

[0060] As shown in FIG. 1, in the embodiment, before the decoding stage, the number of the current input token is compared with the number n of tokens that can be concurrently executed by a device at a time t :

[0061] If the number of output tokens exceeds the number of tokens n that the device can concurrently execute at a time t , the K vector and V vector corresponding to the current input token are not cached, and the tokens output by the large model are accumulated for the next decoding.

[0062] If the number of input tokens exceeds the number of tokens n that can be concurrently executed t , the time T at which the cached K matrix and V matrix are loaded into the computing core is compared with the time T for decoding the current input token once. l d :

[0063] If T d < T l , the K matrix and V matrix corresponding to the current input token are not cached, and the tokens output by the large model are accumulated for the next decoding.

[0064] If T d > T l , the K matrix and V matrix of the current input token are cached, and the new output token of the current decoding is used to continue decoding.

[0065] The number of tokens n that the device can concurrently execute at a time in the embodiment is as follows: t

[0066] n t = theoretical peak computing power / computing amount of the most computationally intensive network layer, i.e., the time for decoding less than n t tokens and the time for decoding n t tokens are consistent.

[0067] The time T at which the computing core is loaded in the embodiment is as follows: l

[0068] T l = (2 * d * l / memory bandwidth) * N;

[0069] wherein d is the dimension of a single K or V vector, since the length of each K matrix and V matrix is the same in the Transformer model, it is multiplied by 2; l is the number of attention layers in the large model; and N is the number of all tokens before the current decoding.

[0070] The time T for decoding once in the embodiment is as follows: d

[0071] T d ​​​​= current input token number * calculation amount per token / theoretical peak computing power.

[0072] Embodiment 2:

[0073] The embodiment provides a partial fixed cache system based on large model inference, which comprises:

[0074] The pre-filling module is used for inputting a sequence or text, performing pre-processing to complete format conversion of the original input sequence or text into a large model input matrix, caching K matrix and V matrix of the original input token in each attention layer based on a KVcache method, and converting the input sequence or text into a large model input matrix represented by a fixed-length feature vector for each token. The input matrix is outputted with a first token after model inference.

[0075] The decoding module is different from the traditional KVcache method. Before each decoding, it pre-judges whether to cache the K matrix and V matrix of the current input token. According to the result, it determines whether to cache the K matrix and V matrix of the current input token in the current decoding process.

[0076] In the embodiment, before the decoding stage, the number of the current input token is compared with the number of tokens n t that can be concurrently executed by the device at a single time.

[0077] If the number of output tokens exceeds the number of tokens n t that can be concurrently executed by the device at a single time, the K vector and the V vector corresponding to the current input token are not cached, and the tokens outputted by the large model are accumulated for the next decoding.

[0078] If the number of input tokens exceeds the number of tokens n t that can be concurrently executed, the time T l when the cached K matrix and V matrix are loaded to the computing core is compared with the time T d for decoding the current input token once.

[0079] If T d < T l , the K matrix and the V matrix corresponding to the current input token are not cached, and the tokens outputted by the large model are accumulated for the next decoding.

[0080] If T d > T l , the K matrix and the V matrix of the current input token are cached, and the new output token of the current decoding is used for continuing decoding.

[0081] Wherein, the number of tokens n that the device can concurrently execute at a time t The formula is as follows:

[0082] n t = theoretical peak computing power / computing amount of the most computationally intensive network layer, i.e., the time for the computing device to decode less than n t tokens and the time to decode n t tokens is consistent;

[0083] The formula for the time T l loaded to the computing core is as follows:

[0084] T l = (2*d*l / memory bandwidth)*N;

[0085] Wherein, d is the dimension of a single K or V vector, and since the length of each K matrix and V matrix is the same in the Transformer model, it is multiplied by 2; l is the number of attention layers in the large model; and N is the number of all tokens before the current decoding;

[0086] The formula for the time T d of a single decoding is as follows:

[0087] T d = current input token number*computing amount per token / theoretical peak computing power.

[0088] Embodiment 3:

[0089] The embodiment also provides an electronic device, comprising: a memory and a processor;

[0090] Wherein, the memory stores computer execution instructions;

[0091] The processor executes the computer execution instructions stored by the memory, so that the processor executes the partial fixed cache method based on large model inference in any embodiment of the present application.

[0092] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), ready-to-program gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor, or the processor can be any conventional processor.

[0093] The memory can be used to store computer programs and / or modules, and the processor can realize various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function, etc.; and the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state memory devices.

[0094] Embodiment 4:

[0095] The embodiment also provides a computer readable storage medium, in which a plurality of instructions are stored, the instructions are loaded by a processor to make the processor execute the partial fixed cache method based on large model inference in any embodiment of the application. Specifically, a system or device equipped with a storage medium can be provided, and software program code for implementing the functions of any of the above embodiments is stored on the storage medium, and the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0096] In this case, the program code read from the storage medium can itself implement the functions of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the application.

[0097] Embodiments of the storage medium for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0098] In addition, it should be clear that not only the program code read by the computer can be executed, but also the operating system and the like operating on the computer can be completed by the instructions based on the program code to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0099] In addition, it can be understood that the program code read from the storage medium can be written into the memory provided in the expansion board inserted into the computer or the memory provided in the expansion unit connected to the computer, and then the CPU and the like installed on the expansion board or the expansion unit can be executed based on the instructions of the program code to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

A partial fixed cache method based on large model inference, characterized in that, The method specifically comprises the following steps: The pre-filling stage: inputting a sequence or text, performing pre-processing to complete the format conversion of the original input sequence or text into a large model input matrix, and based on the KVcache method, caching the K matrix and V matrix of the original input token in each attention layer, and converting the input sequence or text into a large model input matrix in which each token is represented by a fixed-length feature vector, and the input matrix is outputted to the first token through model inference; The decoding stage: unlike the traditional KVcache method, a pre-judgment is made on whether to cache the K matrix and V matrix of the current input token before each decoding, and it is determined whether to cache the K matrix and V matrix of the current input token in the current decoding process according to the result. The partial fixed cache method based on large model inference according to claim 1, characterized in that, The pre-processing process in the pre-filling stage specifically comprises the following steps: tokenizer: converting the original text or sequence into an initial numerical representation of the text composed of tokens, i.e., converting the input text into input tokens; embedding: converting the input token into a word vector encoding as the input of the model, and fusing the position vector encoding into the word vector encoding to reflect the position correlation of the input token; If the input size and the large model input do not match, reshape the input size. The partial fixed cache method based on large model inference according to claim 1, wherein, Before the decoding stage, compare the number of tokens currently inputted with the number of tokens n that the device can concurrently execute in a single time t : If the number of output tokens exceeds the number of tokens n that the device can concurrently execute at a time t , the K vector and the V vector corresponding to the current input token are not cached, and the tokens output by the large model are accumulated for the next decoding. If the number of input tokens exceeds the number of tokens n that can be executed concurrently t , then continue comparing the time T at which the cached K matrix and V matrix are loaded to the computing core l and the time T for decoding the current input token once d : If T d <T l If the current input token corresponds to the K matrix and the V matrix, the K matrix and the V matrix corresponding to the current input token are not cached, and the token output by the large model is accumulated for the next decoding. If T d > T l , then cache the K matrix and V matrix for the current input token and continue decoding with the new output token from this decoding. The partial fixed cache method based on large model inference according to claim 3, characterized in that, The number of tokens n that the device can concurrently execute at a time t The formula is as follows: n t = theoretical peak computational power / computational load of the most computationally demanding network layer, i.e. the time to decode a word of n t tokens on a computational device and the time to decode n t tokens are consistent. The partial fixed cache method based on large model inference according to claim 3, characterized in that, Time T to load into the computing core l The formula is as follows T l = (2 * d * l / memory bandwidth) * N; Wherein, d is the dimension of a single K or V vector, since in the model, the length of each K matrix and V matrix is the same, therefore multiplied by 2; l is the number of attention layers in the large model; N is the number of all tokens before the current decoding. The partial fixed cache method based on large model inference according to claim 3, characterized in that, Time T for one decoding d The formula is as follows: T d = current input token number * calculation amount per token / theoretical computing power peak. A partial pinned cache system based on large model inference, characterized in that, The system comprises: The pre-filling module is configured to input a sequence or text, perform pre-processing to complete the format conversion of the original input sequence or text into a large model input matrix, and based on the KVcache method, cache the K matrix and V matrix of the original input token in each attention layer, and convert the input sequence or text into a large model input matrix in which each token is represented by a fixed-length feature vector, and the input matrix is outputted to the first token through model inference; The decoding module, unlike the traditional KVcache method, makes a pre-judgment on whether to cache the K matrix and V matrix of the current input token before each decoding, and determines whether to cache the K matrix and V matrix of the current input token in the current decoding process according to the result. The partial fixed cache system based on large model inference according to claim 7, characterized in that, Before the decoding stage, compare the number of tokens currently inputted with the number of tokens n that the device can concurrently execute in a single time t : If the number of output tokens exceeds the number of tokens n that the device can concurrently execute at a time t , the K vector and the V vector corresponding to the current input token are not cached, and the tokens output by the large model are accumulated for the next decoding. If the number of input tokens exceeds the number of tokens n that can be executed concurrently t , then continue comparing the time T at which the cached K matrix and V matrix are loaded to the computing core l and the time T for decoding the current input token once d : If T d <T l If the current input token corresponds to the K matrix and the V matrix, the K matrix and the V matrix corresponding to the current input token are not cached, and the token output by the large model is accumulated for the next decoding. If T d > T l , then cache the K matrix and V matrix for the current input token and continue decoding with the new output token from this decoding. Wherein, the number of tokens n that the device can concurrently execute at a single time t The formula is as follows: n t = theoretical peak computing power / the amount of computation of the most computationally intensive network layer, i.e., the time to decode a word of n tokens on a computing device and the time to decode n tokens are consistent t t tokens.​ Time T to load to the computing core l The formula is as follows: T l = (2 * d * l / memory bandwidth) * N; Wherein, d is the dimension of a single K or V vector, since in the model, the length of each K matrix and V matrix is the same, therefore multiplied by 2: l is the number of attention layers in the large model; N is the number of all tokens before the current decoding. Time T for one decoding d The formula is as follows: T d = current input token number * calculation amount per token / theoretical computing power peak. An electronic device, characterized by comprising: It comprises: a memory and at least one processor; wherein the memory has stored thereon a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the partial fixed caching method based on large model inference as claimed in any one of claims 1 to 6. A computer-readable storage medium, characterized by, The computer readable storage medium has stored therein a computer program, which can be executed by a processor to implement the partial fixed caching method based on large model inference as claimed in any one of claims 1 to 6.

Images