Method for recalling key-value data of historical tokens in model inference process

Abstract

The present disclosure provides a method for recalling key-value data of historical tokens in a model inference process. The method comprises: determining a key vector and a value vector of at least one historical token of a model; performing sparse quantization on the key vector of the at least one historical token to obtain a sparse key vector, wherein the sparse key vector is stored in at least one page in a cache in a page-stored manner; for each page in the at least one page, counting a correlation score between each sparse key vector stored in the page and a current query to obtain a page correlation score of the page; determining a page corresponding to the stored key vector and value vector of the page in the cache as a relevant page of the current query when the page correlation score of the page is higher than a page correlation threshold; and recalling the key vector and the value vector stored in each relevant page in batches from each relevant page through a single read operation, for determining an attention output for an output token.

Description

Technical Field

[0001] This disclosure relates to model reasoning, and more particularly to a method, apparatus, computer device, computer-readable storage medium, or computer program product for recalling key-value data of historical lexical units during model reasoning. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have achieved large-scale industrialization and are widely used in various real-world scenarios. Large model inference is the process of deploying a trained large language model to a real-world application environment and generating the required output based on the input task instructions or contextual information. Its core relies on the autoregressive decoding mechanism of the Transformer architecture. Specifically, large model inference uses a successive decoding approach to generate the target output sequence. In each decoding process, attention output is calculated based on historical lexical information through an attention mechanism to predict the generation result of the next lexical term. This process is repeated until the entire output sequence is generated.

[0003] With its massive number of model parameters and excellent feature learning capabilities, large language models have been deeply applied in many technical fields such as natural language processing, multimodal interaction, complex task reasoning, and intelligent decision-making, becoming a core supporting element for the industrial application of artificial intelligence technology. Summary of the Invention

[0004] This disclosure provides a method, apparatus, and computer device, computer-readable storage medium, and computer program product for performing the method for recalling key-value data of historical lexical units during model inference.

[0005] According to one aspect of this disclosure, a method is provided for recalling key-value data of historical lexical units during model inference. The method includes: determining key vectors and value vectors of at least one historical lexical unit of the model, wherein the key vectors and value vectors are stored in at least one first page in a cache in a paginated manner, and each of the at least one first page stores a first predetermined number of key vectors and value vectors of historical lexical units; performing sparse quantization on the key vectors of the at least one historical lexical unit to obtain sparse key vectors, wherein the sparse key vectors corresponding to the historical lexical units stored in each first page are stored in a corresponding second page in the cache in a paginated manner; for each of the at least one second page, calculating the relevance score between the sparse key vectors stored in the second page and the current query to obtain a page relevance score for the second page; determining the first page corresponding to the second page in the cache whose page relevance score is higher than a page relevance threshold as the relevant page of the current query; and batch recalling the key vectors and value vectors stored in the relevant pages from each relevant page through a single read operation to determine attention output for output lexical units.

[0006] In some embodiments, calculating the relevance score between each sparse key vector stored in the second page and the current query to obtain the page relevance score of the second page includes: for each sparse key vector stored in the second page, performing a dot product between the sparse key vector and the current query to obtain the relevance score between the sparse key vector and the current query; and determining the average of the relevance scores between all sparse key vectors stored in the second page and the current query as the page relevance score of the second page.

[0007] In some embodiments, calculating the relevance score between each sparse key vector stored in the second page and the current query to obtain the page relevance score of the second page includes: for each sparse key vector stored in the second page, performing a dot product between the sparse key vector and the current query to obtain the relevance score between the sparse key vector and the current query; and determining the page relevance score of the second page based on the number of relevant key vectors of the current query stored in the second page, wherein the relevant key vectors are sparse key vectors with a relevance score higher than a vector relevance threshold.

[0008] In some embodiments, determining the first page corresponding to the second page in the cache with a page relevance score higher than the page relevance threshold as the relevant page of the current query includes: determining the number of historical terms to be recalled; determining the value N of the page relevance threshold based on the number to be recalled and the first predetermined number, wherein the value of the page relevance threshold is the ratio of the number to be recalled to the first predetermined number; and determining the first page corresponding to the top N second pages with the highest page relevance scores as the relevant page of the current query.

[0009] In some embodiments, the method further includes: retrieving from the cache the key vector and value vector of the last generated historical lexical unit that does not complete a full page, for determining attention output for the output lexical unit.

[0010] In some embodiments, sparse quantization of the key vector to obtain a sparse key vector includes: assigning a predetermined proportion of the values ​​in the key vector to zero; and quantizing the remaining non-zero values ​​of the assigned key vector to obtain the sparse key vector.

[0011] In some embodiments, the sparse key vector is stored in the cache by storing the positions and values ​​of the non-zero values ​​of the sparse key vector.

[0012] According to another embodiment of this disclosure, an apparatus for recalling key-value data of historical lexical units during model inference is also provided. The apparatus includes: a key-value vector determination unit configured to determine the key vector and value vector of at least one historical lexical unit of the model, wherein the key vector and the value vector are stored in at least one first page in a cache in a paginated manner, and each of the at least one first page stores a first predetermined number of key vectors and value vectors of historical lexical units; and a sparse quantization unit configured to perform sparse quantization on the key vector of the at least one historical lexical unit to obtain a sparse key vector, wherein the sparse key vector corresponding to the historical lexical unit stored in each first page is paginated. The sparse key vectors stored in the cache are stored in the corresponding second page. The page relevance determination unit is configured to calculate the relevance score between each sparse key vector stored in the second page and the current query for each second page in at least one second page, so as to obtain the page relevance score of the second page. The relevant page determination unit is configured to determine the first page corresponding to the second page in the cache whose page relevance score is higher than the page relevance threshold as the relevant page of the current query. The recall unit is configured to recall the key vectors and value vectors stored in the relevant pages in batches from each relevant page through a single read operation, so as to determine the attention output for the output token.

[0013] According to another embodiment of this disclosure, a computer device is provided, the computer device comprising: at least one processor; and a memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the aforementioned method.

[0014] According to another embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, causes the processor to perform the aforementioned method.

[0015] According to another embodiment of this disclosure, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, causes the processor to perform the aforementioned method.

[0016] The method for recalling key-value data of historical lexical units during model inference provided by embodiments of this disclosure can simultaneously improve the accuracy and efficiency of recall.

[0017] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0018] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0019] Figure 1 A method for recalling key-value data of historical lexical units during model inference is illustrated according to embodiments of the present disclosure.

[0020] Figure 2A A pre-filling stage in the model inference process according to an embodiment of the present disclosure is illustrated.

[0021] Figure 2B A sparse quantization process in the model inference process according to an embodiment of the present disclosure is illustrated.

[0022] Figure 3A The decoding stage in the model reasoning process according to an embodiment of the present disclosure is illustrated.

[0023] Figure 3B The process of determining a page relevance score in the model inference process according to an embodiment of the present disclosure is illustrated.

[0024] Figure 4An apparatus for recalling key-value data of historical lexical terms during model inference is illustrated according to embodiments of the present disclosure.

[0025] Figure 5 An exemplary block diagram of a computer device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0026] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0027] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0028] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0029] In each decoding process of the large model inference, attention calculation is performed based on historical words using an attention mechanism to predict the next word. This attention mechanism allows the model to dynamically focus on the most relevant content in the context when processing sequence data. During computation, the large model generates two types of key feature information: key vectors and value vectors. Key vectors are used to match the feature association between each historical word and the next output word, while value vectors provide semantic feature support for each historical word. Together, they constitute the core input for the attention mechanism. Specifically, in the pre-filling stage, the model generates corresponding key and value vectors based on the input words and stores them for use in the decoding stage. In the decoding stage, when generating new words, the model needs to recall the key and value vectors of all historical words to participate in the computation, ensuring the accuracy of the inference result. For example, when generating the nth new word, the model needs to recall the key-value data of the 1st to (n-1th)th historical words and perform attention calculation with the current query of the nth word. The recalled historical words can include both input words and already generated output words. Example procedures for the model pre-filling and decoding stages can be combined with embodiments of this disclosure and referred to below. Figures 2A-3B To elaborate in detail.

[0030] However, as the length of the input text increases and the number of polling iterations in human-computer interaction rises, the number of historical word units that need to be recalled grows exponentially. The storage length of the key-value vectors for all historical word units increases linearly, leading to a continuous expansion of the corresponding data volume. In actual model inference deployment, due to limitations in hardware memory resources, the key-value data of historical word units cannot reside in memory for extended periods. Therefore, each decoding process requires a full recall of the key-value data of historical word units from external storage, resulting in significant I / O overhead. When facing extremely long sentence inference tasks, this full recall operation significantly increases decoding latency, severely reducing the model's inference efficiency and interaction response speed, becoming a core performance bottleneck in large-scale, long-text inference scenarios.

[0031] Existing technologies have proposed the following solutions to address the aforementioned technical problems. Specifically, the storage of key and value vectors can be optimized along the sequence dimension. In this solution, the key and value vectors of historical words can be selectively discarded to reduce storage costs. For example, if there are a total of n historical words, the model only stores the key-value data of 1 to n / 2 historical words. However, this solution carries the risk of uncontrollable information loss. In practical applications, if the key and value vectors of the discarded words contain content crucial to text semantic understanding and contextual logical connections, it will cause irreversible information loss during the model's inference process. This directly leads to a significant decrease in the semantic coherence and logical accuracy of the model's output, severely impacting the model's inference performance and limiting its widespread application in real-world scenarios.

[0032] In summary, existing optimization schemes for model inference have insurmountable defects, and a new optimization scheme is urgently needed to overcome the shortcomings of the existing technology.

[0033] To address the aforementioned problems, this disclosure proposes a method for recalling key-value data of historical lexical units during model inference. The inventors recognized that, on the one hand, important keywords and sentences in user-input model content are always concentrated in a few paragraphs or sentences; on the other hand, the key vectors and value vectors of lexical units are always logically stored consecutively in one or more cached pages in a paginated manner (this paginated storage will be described in detail below). That is, important lexical units will be concentrated on a few specific pages. Therefore, this disclosure utilizes the paginated storage characteristic of key-value vectors, first performing sparse quantization on the key vectors, calculating the page relevance score between the page and the current lexical unit for each page storing sparse key vectors, and selectively batch recalling key-value vectors stored in pages with high relevance based on this page relevance score. Thus, this disclosure not only avoids the loss of key information by recalling historical lexical units but also improves the efficiency of recalling historical lexical units, thereby improving memory retrieval efficiency and model inference efficiency.

[0034] The following text will first combine Figure 1 This invention describes a method for recalling key-value data of historical lexical units during model inference according to embodiments of the present disclosure, and then... Figure 2A as well as Figure 3A This invention describes the pre-filling and decoding stages involved in the model inference process according to embodiments of the present disclosure, through... Figure 2B as well as Figure 3B This section describes the sparse quantization and page relevance score determination involved in the model inference process according to embodiments of the present disclosure, and the final... Figures 4-5 Apparatus and computer devices according to embodiments of the present disclosure will also be described. Additionally, computer-readable media and computer program products according to embodiments of the present disclosure will also be described.

[0035] Figure 1 A method 100 for recalling key-value data of historical lexical units during model inference is illustrated according to an embodiment of the present disclosure.

[0036] At step 101, the key vector and value vector of at least one historical word in the model are determined. The key vector and value vector are stored in at least one first page in the cache in a paginated manner. Each of the at least one first page stores a first predetermined number of key vectors and value vectors of historical words.

[0037] In step 102, the key vector of at least one historical word is sparsely quantized to obtain a sparse key vector, wherein the sparse key vector corresponding to the historical word stored in the first page is stored in the corresponding second page in the cache in a paginated manner.

[0038] In step 103, for each of the at least one second page, the relevance score between each sparse key vector stored in the second page and the current query is calculated to obtain the page relevance score of the second page.

[0039] In step 104, the first page corresponding to the second page in the cache with a page relevance score higher than the page relevance threshold is determined as the relevant page for the current query.

[0040] At step 105, the key vectors and value vectors stored in each relevant page are retrieved in batches through a single read operation to determine the attention output for the output lexicon.

[0041] After executing the method provided in this embodiment, firstly, by statistically analyzing relevance scores, the pages containing key terms highly relevant to the current query are identified. This allows for selective recall of key-value data stored on these pages, thereby improving the accuracy of model inference while maintaining a limited total number of recalled terms. Secondly, the method provided in this embodiment can be combined with the hardware characteristics of paginated key-value data storage to batch recall key-value data from certain important and relevant pages, rather than selecting multiple key-value data points from each page individually. This improves the recall efficiency of historical terms, thereby increasing the computer's memory read efficiency and computational efficiency, ultimately balancing model inference efficiency and accuracy.

[0042] The principles of this disclosure will now be described in detail.

[0043] At step 101, the key vector and value vector of at least one historical word of the model can be determined. The key vector and value vector are stored in at least one first page in the cache in a paginated manner. Each of the at least one first page stores a first predetermined number of key vectors and value vectors of historical words.

[0044] As mentioned earlier, the model generates key and value vectors for each word during inference. Transformer-based model inference can include a pre-filling stage and a decoding stage. In the pre-filling stage, for the input prompt word sequence, the model first decomposes the prompt words into one or more words and obtains the word embedding vector for each dimension. For each word embedding vector, the model can perform a linear transformation on the vector and determine its corresponding query vector, key vector, and value vector, storing the key and value vectors for use in the decoding stage. In the decoding stage, to generate the (n+1)th word, the model calculates the current query vector based on historical words and performs attention calculation with the key and value vectors of the historical words to predict the (n+1)th word. The key and value vectors of the (n+1)th word can then be stored for use in the subsequent decoding stage. Therefore, determining the key and value vectors of at least one historical word in the model can at least include determining the key and value vectors of the words input in the pre-filling stage, as well as the key and value vectors of the words already generated in the decoding stage.

[0045] refer to Figure 2A and Figure 3A Let me explain this step. Figure 2A An example of a pre-filling stage according to an embodiment of the present disclosure is illustrated, in which, for input lexical units 1 to n, the model calculates query vectors Q1 to Qn, key vectors K1 to Kn, and value vectors V1 to Vn corresponding to each lexical unit, and stores the key vectors K1 to Kn and the value vectors V1 to Vn for use in the subsequent decoding stage. Figure 3A An example of a decoding stage according to an embodiment of the present disclosure is illustrated. In this stage, the model uses historical lexical units 1 to n to obtain the prediction result of the (n+1)th lexical unit to be generated, and obtains the key vector Kn+1 and value vector Vn+1 of the (n+1)th lexical unit. Furthermore, the model can store the key vector Kn+1 and value vector Vn+1 of the generated (n+1)th lexical unit for use when generating subsequent lexical units. Therefore, when the model generates the (n+1)th lexical unit, it can utilize all or a portion of the key vectors K1 to Kn and value vectors V1 to Vn of the input lexical units 1 to n. Similarly, when the model generates the (n+2)th lexical unit, it can utilize all or a portion of the key vectors K1 to Kn+1 and value vectors V1 to Vn+1 of the input lexical units 1 to n+1.

[0046] In large-scale model inference, to enable the attention mechanism to efficiently read the key and value vectors of historical words, the key and value vectors need to be stored contiguously. Paging storage can store the key and value vectors of historical words in one or more cached pages. These pages assign contiguous logical addresses to the key and value vectors, and establish a unique mapping between these logical addresses and physical memory blocks. In some examples, the memory space of each page is fixed. In other embodiments, each page can be configured to store a predetermined amount of information, such as 64 key and / or value vectors. The page size can be dynamically determined based on the amount of data stored. When needed, the model can read the key and value vectors of historical words at contiguous logical addresses, without being aware of the scattered arrangement of physical memory. Using pages allows for precise location of the physical memory storage address corresponding to each page of key and value vectors, thus ensuring computational efficiency. Paging storage effectively avoids the memory fragmentation problem caused by traditional contiguous storage, supports flexible adaptation to sequences of different lengths, reduces storage overhead and I / O time, and provides storage support for high-concurrency, high-efficiency inference of large models.

[0047] The initial predetermined number can be set to 64, 256, or 512 key / value vectors of historical lexical units. The specific number can be flexibly set according to actual needs to achieve a balance between memory utilization and management overhead.

[0048] At step 102, the key vector of at least one historical word can be sparsely quantized to obtain a sparse key vector, wherein the sparse key vector corresponding to the historical word stored in the first page is stored in the corresponding second page in the cache in a paginated manner.

[0049] In some embodiments, sparse quantization of a key vector to obtain a sparse key vector may include: assigning a predetermined proportion of the values ​​in the key vector to zero; and then quantizing the remaining non-zero values ​​in the assigned key vector to obtain the sparse key vector.

[0050] In the example, the key vectors of historical terms can be sparsified by a predetermined proportion, that is, a predetermined proportion of the values ​​in the vectors can be assigned the value of zero. In other examples, the key vectors can also be sparsified by a random proportion.

[0051] After assigning 0 to some values ​​in the key vector, the remaining non-zero values ​​in the key vector can be further quantized. Quantization can be performed not only by bit width, but also by normalization quantization, bucketing quantization, product quantization, symmetric / asymmetric quantization, etc., and this disclosure does not impose any limitations on this. By performing sparse quantization on the key vector, the storage space of the key vector can be reduced while ensuring the integrity of the word key vector sequence, thus lowering the recall cost of subsequent decoding. In some cases, those skilled in the art can also use similar methods to perform sparse quantization on the value vectors of historical words, which will not be elaborated upon here.

[0052] For example, you can refer to Figure 2B This illustrates the process of performing sparse quantization on key vectors to obtain sparse key vectors. Figure 2B In the illustrated sparse quantization process 210, taking key vectors as an example, each of the key vectors K1 to Kn of the input word 1 to n is an 8-dimensional vector. For Figure 2B The key vector Ki shown can undergo the following sparse process: the values ​​of each dimension in the 8-dimensional vector are sorted in descending order of absolute value, and a predetermined proportion of the values ​​with smaller values ​​are assigned the value 0. This predetermined proportion can be flexibly set according to actual needs such as model parameters and inference performance, and this disclosure does not impose any restrictions on it. Figure 2B This example illustrates a predetermined ratio of 1:2, or twice the sparsity. The model retains only the values ​​of the 4-dimensional vector (-0.52, 0.82, 0.64, -1.0), assigning a value of 0 to the values ​​of the other dimensions. Then, for the assigned key vector... Figure 2B The following quantization process is performed: For the sparse non-zero values ​​in the key vector Ki, namely -0.52, 0.82, 0.64, and -1.0, a 2-bit quantization strategy is adopted. In this quantization method, the maximum absolute value of the eigenvalue, 1.0, is divided into four step intervals, each with a step size of 0.25. Following the principle of nearest neighbor approximation, the original eigenvalues ​​are mapped to the nearest discrete value in {0, ±0.25, ±0.5, ±0.75, ±1.0}. Therefore, -0.52 is quantized to -0.5, 0.82 to 0.75, 0.64 to 0.5, and -1.0 to -1.0. After the above sparse quantization, the sparse key vector Ki' is obtained. The predetermined ratio, in addition to the 1:2 example shown above, can also include sparse ratios such as 1:8. The specific ratio can be flexibly set according to the output effect of the model, and this disclosure does not impose any restrictions on it. Figure 3A In the decoding stage, the above sparse quantization steps can be performed on the key vector of each generated word, such as the key vector Kn+1 of the (n+1)th word, which will not be elaborated here.

[0053] After obtaining the sparse key vector through the aforementioned sparse quantization process, the sparse key vector can be stored in a cache by storing the positions and values ​​of the non-zero values. In other words, the cache only needs to store the positions and values ​​of the non-zero values, without storing the positions and values ​​of the zero values ​​themselves. Using this method, the model can infer the other positions that should be zero values ​​from the positions and values ​​of the non-zero values ​​read from the sparse key vector. Since the cache only stores information related to the non-zero values ​​of the key vector, it effectively reduces the storage overhead of the cache without affecting the integrity of the key vector sequence.

[0054] It is understood that the storage method of sparse key vectors is not limited to this. Those skilled in the art can use any suitable method to store sparse key vectors, such as through masks, bitmap indexes and other methods and combinations thereof.

[0055] In the embodiments of this disclosure, two sets of key / value data are stored for each historical lexical unit. The original key vector and value vector data can be stored on a first page, while the sparsely quantized sparse key vector can be stored on a second page corresponding to the first page. The first and second pages can be associated using various methods, such as mapping table association or step-size association. The original lexical units corresponding to the data stored in each first page are consistent with the original lexical units corresponding to the data stored in their respective second pages.

[0056] Furthermore, various methods can be used to store key vectors and value vectors, as well as sparse key vectors, in the first and second pages. In some embodiments, a single page may store only (sparse) key vectors, only (sparse) value vectors, or both (sparse) key vectors and (sparse) value vectors. In some examples, a single page may include a first storage region for contiguous storage of (sparse) key vectors and a second storage region for contiguous storage of (sparse) value vectors, with the first and second storage regions being physically adjacent and non-overlapping. As will be explained in detail in step 103, only the relevance score between the sparse key vector and the current query needs to be calculated during the decoding process. By storing key vectors and value vectors separately on different pages or centrally storing them in physically adjacent but non-overlapping regions on the same page, the location speed and addressing efficiency of key vectors in the cache can be further improved.

[0057] In step 103, for each of the at least one second page, the relevance score between the sparse key vector stored in the second page and the current query is statistically analyzed to obtain the page relevance score of the second page.

[0058] In some embodiments, step 103 may include: for each sparse key vector stored in the second page, performing a dot product between the sparse key vector and the current query to obtain a relevance score between the sparse key vector and the current query; and determining the average of the relevance scores between all sparse key vectors stored in the second page and the current query as the page relevance score of the second page. In other examples, the page relevance score may also be determined based on the maximum value of the relevance scores between all sparse key vectors stored in the page and the current query, or the Top-K sampling result.

[0059] Specifically, for each page, each sparse key vector stored on that page is multiplied by the current query to obtain a relevance score between the sparse key vector and the current query. A higher relevance score indicates that the sparse key vector is more important and relevant to the current query. This relevance score can be, for example, the original attention score obtained by multiplying the key vector and the query vector in the Transformer attention mechanism, or the final attention score obtained after scaling, softmax normalization, etc. This disclosure does not impose any limitations.

[0060] Furthermore, the average relevance score between all sparse key vectors stored on the page and the current query is determined as the page relevance score. Specifically, for each of the sparse key vectors stored on the page, its relevance score with the current query can be calculated, and the average of these relevance scores is then used to determine the page relevance score. This page relevance score indicates the relevance of the page to the current query.

[0061] For example, you can refer to Figure 3B Let's explain the above steps. In the decoding stage, as shown in step 102, the sparse key vector Ki' is obtained and stored in page Pi. A dot product is performed with the current query Qn+1, resulting in a relevance score of 0.99 between the sparse key vector Ki' and the current query Qn+1. Similarly, for each sparse key vector K1' to Kn' in page Pi, its respective relevance score is obtained. Finally, the average of all relevance scores is calculated, determining that the page relevance score of page Pi is 0.72. It should be noted that although... Figure 3B The diagram shows the relevance scores of five sparse key vectors and calculates the page relevance score of page Pi based on these scores. However, in practice, as mentioned earlier, pages can be flexibly configured to store fixed specifications such as 64, 256, or 512 key / value vectors per page. Figure 3B This is just an example and is not intended to be limiting.

[0062] In other embodiments, step 103 may include determining the center vector of all sparse key vectors stored in the second page. This can be achieved by using the arithmetic mean of all sparse key vectors stored in the second page, or by using a weighted average of all sparse key vectors stored in the second page. In the weighted average calculation, more recently generated sparse key vectors may have a higher weight. Then, the page relevance score of the second page can be obtained by performing a dot product between the center vector and the current query. The center vector may be stored in the second page. Since the embodiments of this disclosure focus on the overall relevance score of the page, pre-calculating and storing the center vectors of each sparse key vector stored in the page further reduces the computational load of subsequent dot product calculations.

[0063] In some other embodiments, step 103 may include: for each sparse key vector stored in the page, performing a dot product between the sparse key vector and the current query to obtain a relevance score between the sparse key vector and the current query; determining the page relevance score of the page based on the number of relevant key vectors of the current query stored in the page, wherein the relevant key vectors are sparse key vectors with a relevance score higher than a vector relevance threshold.

[0064] Specifically, this embodiment does not determine the page relevance score by averaging all relevance scores. Instead, it determines the page relevance score by calculating the number of relevant key vectors. Relevant key vectors are sparse key vectors that are important and relevant to the current query. If a sparse key vector has a relevance score higher than a vector relevance threshold, then that vector is a relevant key vector. It can be seen that the vector relevance threshold is the minimum boundary for determining whether a key vector is a relevant key vector. An appropriate vector relevance threshold can be set based on the model's inference performance or training effectiveness; for example, the vector relevance threshold could be 1.5. Furthermore, the page relevance score can be determined based on the number of relevant key vectors stored on the page for the current query. The more highly relevant sparse key vectors (with relevance scores higher than the vector relevance threshold) a page includes, the higher its page relevance score. In some examples, the page relevance score can be determined based on the ratio of the number of relevant key vectors to the total number of key vectors on the page. For example, if the page stores key vectors with 64 terms and 32 relevant key vectors, the page relevance score is 0.5. In other examples, the page relevance score can be determined based on a predetermined mapping relationship. For example, if the ratio of the number of relevant key vectors to the total number of key vectors on the page is greater than 0.6, the page relevance score is 1; if the ratio is less than 0.1, the page relevance score is 0; otherwise, the page relevance score can be 0.5. It is understood that the specific results and settings of the page relevance scores described above are merely examples, and any suitable method can be used to determine the page relevance score without departing from the principles of this disclosure. In step 104, the first page corresponding to the second page in the cache with a page relevance score higher than the page relevance threshold is determined as the relevant page for the current query.

[0065] In some embodiments, step 104 may include: determining the number of historical terms to be recalled; determining the value N of a page relevance threshold based on the number to be recalled and a first predetermined number, wherein the value of the page relevance threshold is the ratio of the number to be recalled to the first predetermined number; and determining the first page corresponding to the top N second pages with the highest page relevance scores as the relevant pages for the current query.

[0066] Specifically, the number of historical words to be recalled can be determined through quantitative evaluation of model performance. For example, during model training, different recall numbers of historical words can be set for multiple test groups, the actual performance index of the model under each test group can be calculated, and all test groups with actual performance loss less than a threshold can be selected. The number of historical words to be recalled with the smallest recall number among them can then be determined. Alternatively, the number of historical words to be recalled can be determined by the real-time load of the hardware used for large model inference. This disclosure does not impose any limitations on this.

[0067] Furthermore, since the number of historical terms to be recalled has been determined, and the first predetermined number, as described in step 102, can be set based on actual needs, the number of pages that should be relevant to the batch recall of key vectors and value vectors can be determined based on the number to be recalled and the first predetermined number. Therefore, the page relevance threshold N can be determined based on the ratio of the number to be recalled to the first predetermined number, and the top N pages with the highest page relevance scores are determined as relevant pages for the current query. That is, these top N pages have the highest relevance to the current query and can be used as relevant pages for the batch recall of key vectors and value vectors in subsequent steps. Here, N is the smallest integer not less than the ratio of the number to be recalled to the first predetermined number.

[0068] As can be seen from steps 103 and 104, unlike the general case where only the relevance of the key vector of each historical word to the current query is determined, the inventors realized that the important content of the user input model will be concentrated in a few pages. Therefore, as long as the relevance of each page to the current query is determined, it is possible to identify the relevant pages with high relevance to the current query, making it possible to subsequently recall key vectors and value vectors in batches based on relevant pages for subsequent model operations.

[0069] In step 105, key vectors and value vectors stored in each relevant page are retrieved in batches using a single read operation to determine the attention output for the output word. Specifically, since the N most relevant pages with the highest relevance to the current query have been identified in step 104, each relevant page can be read in a single step in step 105, and the key vectors and value vectors stored in that relevant page can be retrieved in batches until all N relevant pages have been read. By selectively retrieving the key vectors and value vectors of historical words stored in these relevant pages, they can be used together with the key vectors and value vectors of the word to be generated to determine the attention output for the output word. (See reference...) Figure 3A As illustrated, the key vector and value vector in the recalled relevant pages can be used together with the query vector Qn+1 of the (n+1)th term to determine the attention output Attention(Q,K,V)=softmax(QK) based on the Transformer's attention mechanism. )V.

[0070] Additionally, in some examples, the key and value vectors of the last generated historical lexical units that do not fill a full page are retrieved from the cache to determine the attention output for the output lexical unit. Specifically, although the (sparse) key and value vectors of historical lexical units are stored across multiple pages in the cache using a paginated storage approach, each page has a fixed size, for example, storing a fixed number of (sparse) key and value vectors of 64, 128, etc. Therefore, some scattered historical lexical units may not fill a single page. For example, a page size may be able to store the key and value vectors of 64 lexical units, but the key and value vectors of the last generated 23 lexical units cannot fill a single page. These vectors are also retrieved from the cache to determine the attention output for the output lexical unit. This is because the inventors realized that these sparse key vectors were generated from the most recent historical words relative to the output words, and the sparse key vectors of the most recent historical words may have a relatively high relevance to the output words. Therefore, these vectors can be directly retained for subsequent reasoning without having to calculate the relevance between these historical words and the current query.

[0071] Figure 2A A pre-filling stage 200 in the model inference process according to an embodiment of the present disclosure is illustrated.

[0072] like Figure 2A As shown, in the pre-filling stage 200, the model has 1 to n input words. For each input word, its corresponding query vector, key vector, and value vector can be determined, and attention is calculated based on these. Among them, a sparse quantization process 210 can be performed on the key vector and / or value vector.

[0073] Figure 2B A sparse quantization process 210 in the model inference process according to an embodiment of the present disclosure is illustrated.

[0074] like Figure 2B As shown, sparse quantization of the key vector Ki yields a sparse key vector Ki'. The method described in step 102 above can be executed during the sparse quantization process 210, and will not be repeated here. It should be noted that although... Figure 2B The diagram illustrates sparse quantization of the key vector; however, this sparse quantization process 210 can also be used for sparse quantization of the value vector.

[0075] Figure 3A A decoding stage 300 in a model inference process according to an embodiment of the present disclosure is illustrated. This decoding stage 300 may perform the method described in steps 101 to 105 as described above.

[0076] like Figure 3A In the decoding stage of the (n+1)th output term, the process 310 for determining the page relevance score can be performed using the previously stored sparse key vectors K1' to Kn' and the query vector Qn+1. As previously mentioned, the relevant pages for the current query vector Qn+1 can be determined based on the page relevance score.

[0077] Then, the key vectors and value vectors stored in the relevant pages can be retrieved in batches from the previously stored key vectors K1...Kn and value vectors V1...Vn for use in attention calculations with the current query vector Qn+1. Based on the result of the attention calculation, the prediction result for the (n+1)th output term can be determined. Further, a key vector Kn+1 and a value vector Vn+1 can be determined for the (n+1)th output term, and the key vector Kn+1 can be sparsely quantized to obtain a sparse key vector K'n+1, which is then stored for subsequent decoding processes.

[0078] Figure 3B An example of a process 310 for determining a page relevance score during model inference according to an embodiment of the present disclosure is illustrated. This process 310 for determining a page relevance score can perform the method described in step 103 above, and will not be repeated here.

[0079] Below, refer to Figure 4 This describes an apparatus 400 for recalling key-value data of historical lexical units during model inference, according to embodiments of the present disclosure. Figure 4 An apparatus 400 for recalling key-value data of historical lexical terms during model inference is illustrated according to an embodiment of the present disclosure.

[0080] The apparatus 400 includes a KV vector determination unit 401, a sparse quantization unit 402, a page relevance determination unit 403, a relevant page determination unit 404, and a recall unit 405. In addition to these units, the apparatus 400 may also include other components; however, since these components are not relevant to the content of this embodiment, their illustrations and descriptions are omitted here. Furthermore, the specific details of the operations performed by the apparatus 400 according to this embodiment are consistent with those described above. Figure 1 The details described are the same, so repeated descriptions of the same details are omitted here to avoid repetition.

[0081] The KV vector determination unit 401 is configured to determine the key vector and value vector of at least one historical word in the model. The key vector and value vector are stored in at least one first page in a cache in a paginated manner, and each first page stores a first predetermined number of key vectors and value vectors of historical words.

[0082] The sparse quantization unit 402 is configured to perform sparse quantization on the key vector to obtain a sparse key vector, wherein the sparse key vector corresponding to the historical word stored in each first page is stored in the corresponding second page in the cache in a paginated manner, and at least one page stores the sparse key vector of a first predetermined number of historical words.

[0083] The page relevance determination unit 403 is configured to perform a statistical analysis on each of the sparse key vectors stored in the second page and the current query for each of the at least one second page, so as to obtain the page relevance score of the second page.

[0084] The relevant page determination unit 404 is configured to determine the first page corresponding to the second page in the cache whose page relevance score is higher than the page relevance threshold as the relevant page for the current query.

[0085] Recall unit 405 is configured to batch recall key vectors and value vectors stored in relevant pages from each relevant page through a single read operation, for use in determining attention output for output lexical units.

[0086] Figure 5 An example computer device 500 is shown in which any of the embodiments described herein may be implemented. The computer device 500 may be used to implement one or more components of the systems and methods described above. The computer device 500 may include a bus 502 or other communication mechanism for communicating information, and one or more processors 504 coupled to the bus 502 for processing information. The processor 504 may be, for example, one or more general-purpose microprocessors.

[0087] Computer device 500 may also include main memory 506, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to bus 502, for storing information and instructions to be executed by processor 504. Main memory 506 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 504. Such instructions, when stored in a storage medium accessible to processor 504, can make computer device 500 a special-purpose machine customized to perform the operations specified in the instructions. Main memory 506 may include non-volatile media and / or volatile media. Non-volatile media may include, for example, optical discs or magnetic disks. Volatile media may include dynamic memory. Common media formats may include, for example, floppy disks, collapsible disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs (read-only optical disc drives), any other optical data storage media, any physical media with a perforated arrangement, RAM (random access memory), DRAM (dynamic random access memory), PROM (programmable read-only memory) and EPROM (erasable programmable read-only memory), FLASH-EPROM (fast erase programmable read-only memory), NVRAM (non-volatile random access memory), any other memory chips or tape cartridges, or network versions of the above.

[0088] Computer device 500 may implement the techniques described herein using custom hardwired logic, one or more ASICs (Application-Specific Integrated Circuits) or FPGAs (Field-Programmable Gate Arrays), firmware, and / or program logic, which, when combined with computer device 500, enable computer device 500 to become a special-purpose machine or to be programmed therein. According to one embodiment, the techniques described herein are executed by computer device 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 508. Executing the sequence of instructions contained in main memory 506 causes processor 504 to perform the processing steps described herein. For example, the processes / methods disclosed herein may be implemented by computer program instructions stored in main memory 506. When these instructions are executed by processor 504, they may perform the steps shown in the corresponding figures and as described above. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0089] Computer device 500 also includes a network interface 510 coupled to bus 502. Network interface 510 can provide bidirectional data communication coupled to one or more network links connected to one or more networks. As another example, network interface 510 can be a local area network (LAN) card to provide data communication connectivity with a compatible LAN (or a WAN component communicating with a WAN (wide area network)). Wireless links can also be implemented.

[0090] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0091] Each process, method, and algorithm described in the preceding sections can be embodied in a code module executed by one or more computer systems or computer processors including computer hardware, and can be fully or partially automated by them. These processes and algorithms can be implemented, in part or in whole, in a specific application circuit.

[0092] When the functions disclosed herein are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, can be embodied in the form of a software product. This software product can be stored in a storage medium and includes instructions to cause a computer device (which may be a personal computer, server, network device, etc.) to perform all or part of the steps of the methods described in the embodiments of this application. The storage medium may include a flash drive, a portable hard drive, ROM, RAM, a magnetic disk, an optical disk, another medium suitable for storing program code, or any combination thereof.

[0093] The embodiments disclosed herein can be implemented via a cloud platform, server, or group of servers that interact with a client. The client can be a terminal device or a client registered by a user on the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can install platform applications.

[0094] The various features and processes described above can be used independently or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in a non-specifically disclosed order, or multiple blocks or states may be combined in a single block or state. Exemplary blocks or states may be executed serially, in parallel, or otherwise. Blocks or states may be added to or removed from the disclosed exemplary embodiments. The exemplary systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the disclosed exemplary embodiments.

[0095] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. An algorithm may consist of program code or instructions stored in memory (such as the non-transitory computer-readable storage medium described above). Such an algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not be explicitly programmed into the computer to perform the function, but may learn from training data to obtain a predictive model for performing that function.

[0096] The various operations of the exemplary methods described herein can be performed at least in part by one or more processors, which are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute the engine of a processor implementation whose operation is to perform one or more of the operations or functions described herein.

[0097] Similarly, the methods described herein can be implemented at least partially by a processor, where a specific processor or one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors can also run in a “cloud computing” environment or as “Software as a Service” (SaaS) to support the execution of the relevant operations. For example, at least some operations can be performed by a group of computers (as an example of a machine including processors), which can be accessed via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0098] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0099] The following describes a computer storage medium according to the present invention, on which a computer program is stored, which, when executed by a processor, causes the processor to perform the aforementioned method for recalling key-value data of historical lexical units during model reasoning.

[0100] The following describes a computer program product according to the present invention, comprising computer program instructions, wherein, when executed by a processor, the computer program instructions cause the processor to perform the aforementioned method for recalling key-value data of historical lexical units during model reasoning.

[0101] In this specification, multiple instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are described and illustrated as independent operations, one or more individual operations may be performed concurrently, and these operations are not required to be performed in the order shown. Structures and functionalities presented as independent components in the example configuration may be implemented as combined structures or components. Similarly, structures and functionalities presented as individual components may be implemented as independent components. These and other variations, modifications, additions, and improvements are all within the scope of this document.

[0102] As used herein, “or” is inclusive rather than exclusive unless explicitly stated or indicated by context. Furthermore, “and” is both common and individual unless explicitly stated or indicated by context. Moreover, multiple instances may be provided for the resources, operations, or structures described herein as a single example. Furthermore, the boundaries between various resources, operations, engines, and data stores are somewhat arbitrary, and specific operations are illustrated within the context of a particular illustrative configuration. The allocation of other functionalities is conceivable and may fall within the scope of various embodiments of this disclosure. Generally, structures and functionalities presented as independent resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functionalities presented as individual resources may be implemented as independent resources. These and other variations, modifications, additions, and improvements are all within the scope of embodiments of this disclosure. Therefore, this specification and accompanying drawings should be viewed in an illustrative rather than restrictive sense.

[0103] The terms “comprising” or “including” are used to indicate the presence of a subsequently stated feature, but do not preclude the addition of other features. Conditional language, in particular, such as “may,” “can,” or “may,” unless specifically stated or otherwise understood in the context of use, is generally intended to express that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Therefore, such conditional language generally does not imply that a feature, element, and / or step is necessary in any way for one or more embodiments, or that one or more embodiments must include logic that, with or without user input or prompting, determines whether such features, elements, and / or steps are included in any particular embodiment, or whether they are to be performed in any particular embodiment.

Claims

1. A method for recalling key-value data of historical lexical units during model text reasoning, characterized in that, The method includes: The key vector and value vector of at least one historical word of the model are determined, and the key vector and the value vector are stored in at least one first page in the cache in a paginated storage manner. Each of the at least one first page stores a first predetermined number of key vectors and value vectors of historical words, wherein the value vector of the historical word indicates the text semantic features corresponding to the historical word. The key vector of the at least one historical word is sparsely quantized to obtain a sparse key vector, wherein the sparse key vector corresponding to the historical word stored in the first page is stored in the corresponding second page in the cache in a paginated manner; For each of the at least one second page, the relevance score between the sparse key vector stored in that second page and the current query is calculated to obtain the page relevance score of that second page. The first page corresponding to the second page in the cache whose page relevance score is higher than the page relevance threshold is determined as the relevant page for the current query; and The key vectors and value vectors stored in each relevant page are retrieved in batches through a single read operation to determine the attention output for the output lexicon.

2. The method as described in claim 1, characterized in that, The page relevance score of the second page is obtained by statistically analyzing the relevance scores between each sparse key vector stored in the second page and the current query, including: For each sparse key vector stored in the second page, the sparse key vector is multiplied by the current query to obtain a relevance score between the sparse key vector and the current query; and The average relevance score of the second page is determined by comparing the relevance scores of all sparse key vectors stored on the second page with the current query.

3. The method as described in claim 1, characterized in that, The page relevance score of the second page is obtained by statistically analyzing the relevance scores between each sparse key vector stored in the second page and the current query, including: For each sparse key vector stored in the second page, the sparse key vector is multiplied by the current query to obtain a relevance score between the sparse key vector and the current query; and The page relevance score of the second page is determined based on the number of relevant key vectors of the current query stored in the second page, wherein the relevant key vectors are sparse key vectors with relevance scores higher than the vector relevance threshold.

4. The method according to any one of claims 1-3, characterized in that, Determining the first page corresponding to the second page in the cache whose page relevance score is higher than the page relevance threshold as the relevant page for the current query includes: Determine the number of historical lexical units to be recalled; The page-related threshold value N is determined based on the quantity to be recalled and the first predetermined quantity, wherein the page-related threshold value is the ratio of the quantity to be recalled to the first predetermined quantity; and The first page corresponding to the top N second pages with the highest page relevance scores is determined as the relevant page for the current query.

5. The method as described in claim 1, characterized in that, The method further includes: Retrieve the key and value vectors of the last generated historical lexicon that does not fill a full page from the cache, and use them to determine the attention output for the output lexicon.

6. The method as described in claim 1, characterized in that, Sparse quantization of the key vector to obtain a sparse key vector includes: Assign a predetermined proportion of the values ​​in the key vector to zero; and For the assigned key vector, the remaining non-zero values ​​are quantized to obtain the sparse key vector.

7. The method as described in claim 6, characterized in that, The sparse key vector is stored in the cache by storing the positions and values ​​of the non-zero values ​​of the sparse key vector.

8. An apparatus for recalling key-value data of historical lexical units during model text reasoning, characterized in that, The device includes: A key-value vector determination unit is configured to determine the key vector and value vector of at least one historical word element of the model. The key vector and the value vector are stored in at least one first page in a cache in a paginated manner. Each of the at least one first page stores a first predetermined number of key vectors and value vectors of historical words elements, wherein the value vector of the historical word element indicates the text semantic features corresponding to the historical word element. A sparse quantization unit is configured to sparsely quantize the key vector of the at least one historical word to obtain a sparse key vector, wherein the sparse key vector corresponding to the historical word stored in each first page is stored in the corresponding second page in the cache in a paginated manner. The page relevance determination unit is configured to, for each of at least one second page, calculate the relevance score between each sparse key vector stored in the second page and the current query to obtain the page relevance score of the second page. The relevant page determination unit is configured to determine the first page corresponding to the second page in the cache whose page relevance score is higher than the page relevance threshold as the relevant page of the current query; and The recall unit is configured to recall key vectors and value vectors stored in each relevant page in batches through a single read operation, for determining attention output for output lexical units.

9. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-7.

Images