Partial fixed caching method, system and equipment based on large model reasoning and medium

By using partial fixed cache method in large-scale inference, dynamically judging cache K matrix and V matrix, the problem of imbalance in computing resources and storage resources during large-scale pre-training model inference is solved, the inference efficiency is improved and memory over-memory is avoided load, achieving more efficient localized reasoning.

CN119067076APending Publication Date: 2024-12-03SHANDONG INSPUR SCI RES INST CO LTD
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202411224816.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-03
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Large-scale pre-trained models require a large number of computing resources and storage resources during the inference process. The traditional storage key-value pairing method causes memory consumption to be linearly correlated with the input sequence length and batch size, resulting in an imbalance between computing resources and storage resources, especially locally. There are bottlenecks when deploying runtime.

Method used

Using a partial fixed cache method based on big model inference, the input sequence is converted into a big model input matrix through the pre-filling stage, and dynamically determine whether the K matrix and V matrix are cached in the decoding stage, and determine whether it is based on the number of concurrent executions of the device and the loading time. Cache, optimize cache policies to reduce memory load.

Benefits of technology

Finding a more suitable balance between computing resources and storage resources improves inference efficiency, reduces the total inference time, avoids memory overload, and improves the utilization rate of computing devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067076A_ABST
    Figure CN119067076A_ABST
Patent Text Reader

Abstract

The invention discloses a partial fixed caching method, system and equipment based on large model reasoning, and a medium, belongs to the technical field of artificial intelligence and natural language, and aims to solve the technical problem of how to find a balance more suitable for large model localization reasoning between computing resources and storage resources, improve reasoning efficiency and improve the efficiency of the large model localization reasoning. According to the technical scheme, the method comprises the steps that in the pre-filling stage, a sequence or a text is input, format conversion from an original input sequence or text to a large model input matrix is completed through preprocessing, and based on a KVcache method, a K matrix and a V matrix of an original input token in all attention layers are cached; in the decoding stage, different from a traditional KVcache method, whether the K matrix and the V matrix of the current input token are cached or not is pre-judged before decoding each time, and whether the K matrix and the V matrix of the current input token are cached or not in the current decoding process is judged according to a result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and natural language processing technology, and in particular to a partial fixed cache method, system, device and medium based on large model reasoning. Background Art

[0002] With the rapid development of artificial intelligence technology, especially in the field of natural language processing, large-scale pre-trained models (such as Transformer models) have shown excellent performance in various tasks. However, these models require a lot of computing and storage resources during the reasoning process. When the traditional Transformer model is reasoning, most reasoning frameworks choose to use the method of storing a large number of key-value pairs to speed up text generation, resulting in memory consumption that is linearly related to the input sequence length and batch size. This method is a typical strategy of exchanging storage resources for computing resources.

[0003] There is nothing wrong with using it on large-scale clusters, but when responding to customers' needs for local deployment and operation, it must be taken into account that customers are unlikely to have large-scale clusters for model deployment and reasoning. Then the storage requirements of KVcache become a bottleneck in the reasoning process.

[0004] Therefore, how to find a balance between computing resources and storage resources that is more suitable for localized reasoning of large models and improve reasoning efficiency is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The technical task of the present invention is to provide a partial fixed cache method, system, device and medium based on large model reasoning to solve the problem of how to find a balance between computing resources and storage resources that is more suitable for large model localized reasoning and improve reasoning efficiency.

[0006] The technical task of the present invention is achieved in the following way: a partial fixed cache method based on large model reasoning, the method is as follows:

[0007] Pre-filling stage: Input sequence or text, after preprocessing, complete the format conversion of the original input sequence or text to the large model input matrix, and cache the K matrix and V matrix of the original input token in each attention layer based on the KVcache method, and convert the input sequence or text into a large model input matrix in which each token is represented by a fixed-length feature vector. The input matrix is ​​inferred by the model to output the first token;

[0008] Decoding stage: Different from the traditional KVcache method, before each decoding, a pre-judgment is made on whether to cache the K matrix and V matrix of the current input token, and based on the result, it is determined whether to cache the K matrix and V matrix of the current input token during this decoding process.

[0009] Preferably, the pre-processing process in the pre-filling stage is as follows:

[0010] Tokenizer: Converts the original text or sequence into the initial numerical representation of the text composed of tokens, that is, converts the input text into input tokens;

[0011] Embedding: Convert the input token into a word embedding as the input of the model, and fuse the position embedding into the word embedding to reflect the position relevance of the input token.

[0012] If the input size does not match the large model input, the input size is reshaped.

[0013] As a preference, before the decoding phase, compare the number of tokens currently input with the number of tokens n that the device can execute concurrently at a time. t :

[0014] If the number of output tokens exceeds the number of tokens that the device can execute concurrently at one time, t , the K vector and V vector corresponding to the current input token are not cached, and the tokens output by the large model continue to be accumulated for the next decoding;

[0015] If the number of input tokens exceeds the number of tokens that can be executed concurrently, n t , then continue to compare the time T of loading the cached K matrix and V matrix into the computing core l and the time T to decode the current input token d :

[0016] If T d <T l , the K matrix and V matrix corresponding to the current input token are not cached, and the tokens output by the large model continue to be accumulated for the next decoding;

[0017] If T d >T l , the K matrix and V matrix of the current input token are cached, and the new output token of this decoding is used to continue decoding.

[0018] Preferably, the number of tokens that can be executed concurrently at one time on a device is n t The formula is as follows:

[0019] n t = Theoretical peak computing power / the computing power of the most computing-intensive network layer, that is, the computing device word decoding is less than n t The time and decoding of tokens t The time of each token is consistent.

[0020] Preferably, the time T to load into the computing core l The formula is as follows

[0021] T l =(2*d*l / memory bandwidth)*N;

[0022] Where d is the dimension of a single K or V vector. Since in the Transformer model, the length of each K matrix and V matrix is ​​the same, it is multiplied by 2; l is the number of attention layers in the large model; N is the number of all tokens before the current decoding.

[0023] Preferably, the decoding time is T d The formula is as follows:

[0024] T d = Current number of input tokens * amount of computation per token / theoretical peak computing power.

[0025] A partially fixed cache system based on large model reasoning, the system comprising:

[0026] The pre-filling module is used to input sequences or texts. After preprocessing, the format conversion of the original input sequence or text to the large model input matrix is ​​completed. Based on the KVcache method, the K matrix and V matrix of the original input token in each attention layer are cached, and the input sequence or text is converted into a large model input matrix in which each token is represented by a fixed-length feature vector. The input matrix is ​​inferred by the model to output the first token;

[0027] The decoding module is different from the traditional KVcache method. Before each decoding, it will pre-judge whether to cache the K matrix and V matrix of the current input token, and determine whether to cache the K matrix and V matrix of the current input token during this decoding process based on the result.

[0028] As a preference, before the decoding phase, compare the number of tokens currently input with the number of tokens n that the device can execute concurrently at a time. t :

[0029] If the number of output tokens exceeds the number of tokens that the device can execute concurrently at one time,t , the K vector and V vector corresponding to the current input token are not cached, and the tokens output by the large model continue to be accumulated for the next decoding;

[0030] If the number of input tokens exceeds the number of tokens that can be executed concurrently, n t , then continue to compare the time T of loading the cached K matrix and V matrix into the computing core l and the time T to decode the current input token d :

[0031] If T d <T l , the K matrix and V matrix corresponding to the current input token are not cached, and the tokens output by the large model continue to be accumulated for the next decoding;

[0032] If T d >T l , then cache the K matrix and V matrix of the current input token, and continue decoding using the new output token of this decoding;

[0033] Among them, the number of tokens that can be executed concurrently by the device at a time is n t The formula is as follows:

[0034] n t = Theoretical peak computing power / the computing power of the most computing-intensive network layer, that is, the computing device word decoding is less than n t The time and decoding of tokens t The time of each token is consistent;

[0035] Time T for loading into the computing core l The formula is as follows

[0036] T l =(2*d*l / memory bandwidth)*N;

[0037] Where d is the dimension of a single K or V vector. Since in the Transformer model, each K matrix and V matrix have the same length, it is multiplied by 2. l is the number of attention layers in the large model. N is the number of all tokens before the current decoding.

[0038] The time for one decoding is T d The formula is as follows:

[0039] T d = Current number of input tokens * amount of computation per token / theoretical peak computing power.

[0040] An electronic device comprising: a memory and at least one processor;

[0041] Wherein, the memory stores a computer program;

[0042] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the partial fixed cache method based on large model reasoning as described above.

[0043] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the above-mentioned partial fixed cache method based on large model reasoning.

[0044] The partial fixed cache method, system, device and medium based on large model reasoning of the present invention have the following advantages:

[0045] (i) The present invention seeks a balance between computing resources and storage resources that is more suitable for localized reasoning of large models, thereby improving reasoning efficiency;

[0046] (ii) During the reasoning process, the present invention dynamically determines whether to perform KV matrix caching, which can reduce the total reasoning time, avoid memory overload, and further improve the utilization rate of the computing device. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present invention is further described below in conjunction with the accompanying drawings.

[0048] Attached Figure 1 This is a flowchart of the decoding stage in the partially fixed cache method based on large model reasoning. DETAILED DESCRIPTION

[0049] The partial fixed cache method, system, device and medium based on large model reasoning of the present invention are described in detail below with reference to the drawings and specific embodiments of the specification.

[0050] Embodiment 1:

[0051] This embodiment provides a partial fixed cache method based on large model reasoning, and the method is specifically as follows:

[0052] S1. Pre-filling stage: Input sequence or text, complete the format conversion of the original input sequence or text to the large model input matrix after preprocessing, and cache the K matrix and V matrix of the original input token in each attention layer based on the KVcache method, and convert the input sequence or text into a large model input matrix in which each token is represented by a fixed-length feature vector. The input matrix outputs the first token after model inference;

[0053] S2, decoding stage: Different from the traditional KVcache method, before each decoding, a pre-judgment is made on whether to cache the K matrix and V matrix of the current input token, and based on the result, it is determined whether to cache the K matrix and V matrix of the current input token during this decoding process.

[0054] Among them, the large model is composed of multiple stacked self-attention neural network modules, and the modules are mainly transformer structures. Different large models contain slightly different network layers in each module, but basically include attention layer (Attention Layer), feedforward neural network (Feedforward Neural Networks), normalization layer (Normalization Layer), residual connection layer (Residual Connection).

[0055] The pre-processing process in the pre-filling stage in step S1 of this embodiment is specifically as follows:

[0056] S101, tokenizer: converts the original text or sequence into the initial numerical representation of the text composed of tokens, that is, converts the input text into input tokens;

[0057] S102, embedding: convert the input token into a word embedding as the input of the model, and merge the position embedding into the word embedding to reflect the position relevance of the input token;

[0058] If the input size does not match the large model input, the input size is reshaped.

[0059] As attached Figure 1 As shown, in this embodiment, before the decoding stage, the number of tokens currently input is compared with the number of tokens n that can be executed concurrently by the device at a time. t :

[0060] If the number of output tokens exceeds the number of tokens that the device can execute concurrently at one time, t , the K vector and V vector corresponding to the current input token are not cached, and the tokens output by the large model continue to be accumulated for the next decoding;

[0061] If the number of input tokens exceeds the number of tokens that can be executed concurrently, n t , then continue to compare the time T of loading the cached K matrix and V matrix into the computing core land the time T to decode the current input token d :

[0062] If T d <T l , the K matrix and V matrix corresponding to the current input token are not cached, and the tokens output by the large model continue to be accumulated for the next decoding;

[0063] If T d >T l , the K matrix and V matrix of the current input token are cached, and the new output token of this decoding is used to continue decoding.

[0064] The number of tokens that can be concurrently executed by the device in this embodiment is n t The formula is as follows:

[0065] n t = Theoretical peak computing power / the computing power of the most computing-intensive network layer, that is, the computing device word decoding is less than n t The time and decoding of tokens t The time of each token is consistent.

[0066] The time T for loading to the computing core in this embodiment l The formula is as follows

[0067] T l =(2*d*l / memory bandwidth)*N;

[0068] Where d is the dimension of a single K or V vector. Since in the Transformer model, the length of each K matrix and V matrix is ​​the same, it is multiplied by 2; l is the number of attention layers in the large model; N is the number of all tokens before the current decoding.

[0069] The time T for one decoding in this embodiment d The formula is as follows:

[0070] T d = Current number of input tokens * amount of computation per token / theoretical peak computing power.

[0071] Embodiment 2:

[0072] This embodiment provides a partial fixed cache system based on large model reasoning, the system comprising:

[0073] The pre-filling module is used to input sequences or texts. After preprocessing, the format conversion of the original input sequence or text to the large model input matrix is ​​completed. Based on the KVcache method, the K matrix and V matrix of the original input token in each attention layer are cached, and the input sequence or text is converted into a large model input matrix in which each token is represented by a fixed-length feature vector. The input matrix is ​​inferred by the model to output the first token;

[0074] The decoding module is different from the traditional KVcache method. Before each decoding, it will pre-judge whether to cache the K matrix and V matrix of the current input token, and determine whether to cache the K matrix and V matrix of the current input token during this decoding process based on the result.

[0075] In this embodiment, before the decoding stage, the number of tokens currently input is compared with the number of tokens that can be executed concurrently at a time by the device. t :

[0076] If the number of output tokens exceeds the number of tokens that the device can execute concurrently at one time, t , the K vector and V vector corresponding to the current input token are not cached, and the tokens output by the large model continue to be accumulated for the next decoding;

[0077] If the number of input tokens exceeds the number of tokens that can be executed concurrently, n t , then continue to compare the time T of loading the cached K matrix and V matrix into the computing core l and the time T to decode the current input token d :

[0078] If T d <T l , the K matrix and V matrix corresponding to the current input token are not cached, and the tokens output by the large model continue to be accumulated for the next decoding;

[0079] If T d >T l , then cache the K matrix and V matrix of the current input token, and continue decoding using the new output token of this decoding;

[0080] Among them, the number of tokens that can be executed concurrently by the device at a time is n t The formula is as follows:

[0081] n t = Theoretical peak computing power / the computing power of the most computing-intensive network layer, that is, the computing device word decoding is less than n t The time and decoding of tokens t The time of each token is consistent;

[0082] Time T for loading into the computing core l The formula is as follows

[0083] T l =(2*d*l / memory bandwidth)*N;

[0084] Where d is the dimension of a single K or V vector. Since in the Transformer model, each K matrix and V matrix have the same length, it is multiplied by 2. l is the number of attention layers in the large model. N is the number of all tokens before the current decoding.

[0085] The time for one decoding is T d The formula is as follows:

[0086] T d = Current number of input tokens * amount of computation per token / theoretical peak computing power.

[0087] Embodiment 3:

[0088] This embodiment also provides an electronic device, including: a memory and a processor;

[0089] Wherein, the memory stores computer-executable instructions;

[0090] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the partial fixed cache method based on large model reasoning in any embodiment of the present invention.

[0091] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0092] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0093] Embodiment 4:

[0094] This embodiment also provides a computer-readable storage medium, which stores a plurality of instructions, which are loaded by a processor to enable the processor to execute the partial fixed cache method based on large model reasoning in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0095] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0096] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0097] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0098] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A partial fixed cache method based on large model reasoning, characterized in that: The method is as follows: Pre-filling stage: Input sequence or text, after preprocessing, complete the format conversion of the original input sequence or text to the large model input matrix, and cache the K matrix and V matrix of the original input token in each attention layer based on the KVcache method, and convert the input sequence or text into a large model input matrix in which each token is represented by a fixed-length feature vector. The input matrix is ​​inferred by the model to output the first token; Decoding stage: Different from the traditional KVcache method, before each decoding, a pre-judgment is made on whether to cache the K matrix and V matrix of the current input token, and based on the result, it is determined whether to cache the K matrix and V matrix of the current input token during this decoding process.

2. The partially fixed cache method based on large model reasoning according to claim 1 is characterized in that: The preprocessing process of the prefilling stage is as follows: Tokenizer: Converts the original text or sequence into the initial numerical representation of the text composed of tokens, that is, converts the input text into input tokens; Embedding: Convert the input token into a word vector code as the input of the model, and integrate the position vector code into the word vector code to reflect the position relevance of the input token; If the input size does not match the large model input, the input size is reshaped.

3. The partially fixed cache method based on large model reasoning according to claim 1 is characterized in that: Before the decoding phase, compare the number of tokens currently input with the number of tokens that the device can execute concurrently at one time n t : If the number of output tokens exceeds the number of tokens that the device can execute concurrently at one time, t , the K vector and V vector corresponding to the current input token are not cached, and the tokens output by the large model continue to be accumulated for the next decoding; If the number of input tokens exceeds the number of tokens that can be executed concurrently, n t , then continue to compare the time T of loading the cached K matrix and V matrix into the computing core l and the time T to decode the current input token d : If T d <T l , the K matrix and V matrix corresponding to the current input token are not cached, and the tokens output by the large model continue to be accumulated for the next decoding; If T d >T l , the K matrix and V matrix of the current input token are cached, and the new output token of this decoding is used to continue decoding.

4. The partial fixed cache method based on large model reasoning according to claim 3 is characterized in that: The number of tokens that can be executed concurrently by the device at one time, n t The formula is as follows: n t = Theoretical peak computing power / the amount of computing power consumed by the most computing power network layer, that is, the computing device word decoding is less than n t The time and decoding of tokens t The time of each token is consistent.

5. The partial fixed cache method based on large model reasoning according to claim 3 is characterized in that: Time T for loading into the computing core l The formula is as follows T l =(2*d*l / memory bandwidth)*N; Among them, d is the dimension of a single K or V vector. Since in the model, the length of each K matrix and V matrix is ​​the same, it is multiplied by 2; l is the number of attention layers in the large model; N is the number of all tokens before the current decoding.

6. The partial fixed cache method based on large model reasoning according to claim 3 is characterized in that: The time for one decoding is T d The formula is as follows: T d = Current number of input tokens * amount of computation per token / theoretical peak computing power.

7. A partially fixed cache system based on large model reasoning, characterized in that: The system includes: The pre-filling module is used to input sequences or texts. After preprocessing, the format conversion of the original input sequence or text to the large model input matrix is ​​completed. Based on the KVcache method, the K matrix and V matrix of the original input token in each attention layer are cached, and the input sequence or text is converted into a large model input matrix in which each token is represented by a fixed-length feature vector. The input matrix is ​​inferred by the model to output the first token; The decoding module is different from the traditional KVcache method. Before each decoding, it will pre-judge whether to cache the K matrix and V matrix of the current input token, and determine whether to cache the K matrix and V matrix of the current input token during this decoding process based on the result.

8. The partially fixed cache system based on large model reasoning according to claim 7, characterized in that: Before the decoding phase, compare the number of tokens currently input with the number of tokens that the device can execute concurrently at one time n t : If the number of output tokens exceeds the number of tokens that the device can execute concurrently at one time, t , the K vector and V vector corresponding to the current input token are not cached, and the tokens output by the large model continue to be accumulated for the next decoding; If the number of input tokens exceeds the number of tokens that can be executed concurrently, n t , then continue to compare the time T of loading the cached K matrix and V matrix into the computing core l and the time T to decode the current input token d : If T d <T l , the K matrix and V matrix corresponding to the current input token are not cached, and the tokens output by the large model continue to be accumulated for the next decoding; If T d >T l , then cache the K matrix and V matrix of the current input token, and continue decoding using the new output token of this decoding; Among them, the number of tokens that can be executed concurrently by the device at a time is n t The formula is as follows: n t = Theoretical peak computing power / the amount of computing power consumed by the most computing power network layer, that is, the computing device word decoding is less than n t The time and decoding of tokens t The time of each token is consistent; Time T for loading into the computing core l The formula is as follows T l =(2*d*l / memory bandwidth)*N; Where d is the dimension of a single K or V vector. Since in the model, the length of each K matrix and V matrix is ​​the same, it is multiplied by 2. l is the number of attention layers in the large model. N is the number of all tokens before the current decoding. The time for one decoding is T d The formula is as follows: T d = Current number of input tokens * amount of computation per token / theoretical peak computing power.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the partial fixed cache method based on large model reasoning according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the partial fixed cache method based on large model reasoning as described in any one of claims 1 to 6.

Citation Information

Cited By

  • Large-scale language model reasoning acceleration method based on cache fusion, electronic equipment and storage medium

    CN119831032A

  • Cache fusion-based large language model inference acceleration method, electronic device and storage medium

    CN119831032B

  • GPU storage and calculation integrated pre-filling decoding separation large model reasoning acceleration method and system

    CN121960766A

  • Partial fixed caching method and system based on large model inference, device and medium

    WO2026051442A1

  • Method and apparatus for optimizing cache during large language model inference

    WO2026138159A1