Inference method, system, equipment and medium for generative model

Abstract

The invention provides a reasoning method, system and device for a generative model and a medium. The method comprises the following steps: acquiring to-be-processed data, and inputting the to-be-processed data into a large language model; for each attention layer, performing the following reasoning: generating a query vector, a key vector and a value vector based on data input to the attention layer, and combining the generated key vector and value vector into a cache block; monitoring cache blocks stored in the cache queue, and when the cache blocks exceed the upper limit of the capacity of the cache space, performing screening based on the correlation score of the query vector and the cache blocks so as to update the cache queue; obtaining an intermediate reasoning result according to the query vector and each cache block in the updated cache queue, and inputting the intermediate reasoning result into the next attention layer for reasoning until a reasoning result generated by the last attention layer of the large language model is obtained and stored; and finally obtaining a generation result of the to-be-processed data. According to the method, the aim of efficiently supporting the infinitely long context can be fulfilled under the condition of limited computing resources.

Description

Technical Field

[0001] This invention relates to the field of model deployment technology, and more particularly to a reasoning method, system, device, and medium for generative models. Background Technology

[0002] In the field of artificial intelligence, with the continuous expansion of large language models, their role in tasks such as content generation, cross-modal understanding, and human-computer interaction is becoming increasingly prominent. As application scenarios continue to evolve, large language models need to be able to handle tasks with extremely long contexts in practical applications such as long text analysis, multi-turn dialogue interaction, and large-scale knowledge retrieval. However, when processing long sequence inputs, the storage overhead and computational complexity of the key-value cache (KV Cache) of the attention mechanism based on the Transformer architecture increase linearly with the sequence length, making it difficult for edge devices with limited storage capacity and computing resources to load related application tasks.

[0003] To address these issues, existing technologies typically employ strategies such as sparse attention computation or key-value cache discarding. However, both have drawbacks: sparse attention methods offload excess key-value cache to slower storage areas and index and load it only when needed, but this relies on a multi-level storage architecture, making it unsuitable for processors with only a single storage level; while key-value cache discarding directly discards historical key-value pairs exceeding the window range. Although simple to implement and theoretically capable of supporting infinitely long contexts, this leads to a significant decrease in model accuracy due to the loss of long-range dependency information. Therefore, there is a need to provide an inference method, system, device, and medium for generative models. Summary of the Invention

[0004] This invention provides a reasoning method, system, device, and medium for generative models to improve the technical problem in the prior art where it is difficult to balance the accuracy and computational efficiency of large language models when deployed on edge or end-side devices with limited storage resources.

[0005] This invention provides an inference method for generative models, applied to a processor. The processor has a cache queue set according to the cache space. The method includes: acquiring data to be processed and inputting the data to be processed into a large language model; wherein the large language model is a generative model based on an attention mechanism and includes multiple cascaded attention layers; for each attention layer, the following inference is performed: based on the data input to the attention layer, a corresponding query vector, key vector, and value vector are generated, and the generated key vector and value vector are combined into at least one cache block; the cache blocks already stored in the cache queue are monitored, and when they exceed the capacity limit of the cache space, the cache blocks are filtered based on the relevance score between the query vector and each cache block to update the cache queue; based on the attention mechanism, an intermediate inference result is obtained according to the query vector and each cache block in the updated cache queue, and it is input into the next attention layer to continue inference until the inference result generated by the last attention layer of the large language model is obtained and saved; it is determined whether the inference result meets a preset termination condition. If it does not meet the condition, the inference result is input into the large language model again for inference until the inference result meets the termination condition, and the generation result of the data to be processed is obtained.

[0006] In one embodiment of the present invention, the cache queue includes an initial queue, a recent queue, and a historical queue, and the initial queue, recent queue, and historical queue are empty before the first storage. The step of monitoring the cache blocks already stored in the cache queue and, when the number exceeds the upper limit of the cache space capacity, filtering cache blocks based on the correlation score between the query vector and each cache block to update the cache queue includes: determining whether the number of cache blocks already stored in the initial queue is less than a preset initial queue upper limit; wherein, for the same data to be processed, the initial queue remains unchanged during the generation process, and the initial queue upper limit is the upper limit of the number of cache blocks that the initial queue can store in the cache space; if so, then based on the number of generated cache blocks and the initial queue... The number of remaining storable cache blocks in the cache queue determines the number of cache blocks generated and stored in the initial queue. If not, it determines whether the difference between the preset recent queue upper limit and the number of cache blocks already stored in the recent queue is greater than or equal to the number of generated cache blocks. Here, the recent queue upper limit is the upper limit of the number of cache blocks that the recent queue can store in the cache space. If yes, all generated cache blocks are stored in the recent queue. If not, based on the number of generated cache blocks and the number of remaining storable cache blocks in the recent queue, the number of cache blocks to be transferred from the recent queue is determined, and after being transferred to the historical queue, all generated cache blocks are stored in the recent queue. Here, the historical queue is used for capacity control based on a relevance scoring mechanism.

[0007] In one embodiment of the present invention, the step of determining the number of cache blocks to be transferred from the recent queue based on the number of generated cache blocks and the number of remaining storable cache blocks in the recent queue, and storing all generated cache blocks in the recent queue after transferring them to the historical queue, includes: determining whether the number U of generated cache blocks is less than or equal to the number A of remaining storable cache blocks in the recent queue; if yes, then storing all generated cache blocks in the recent queue; if no, then based on the first-in-first-out principle, transferring the first UA cache blocks generated in the recent queue as cache blocks to be transferred to the historical queue, and after the transfer is completed, storing all generated cache blocks in the recent queue.

[0008] In one embodiment of the present invention, the step of capacity control of the historical queue based on the relevance scoring mechanism includes: determining whether the sum M of the number of cache blocks to be transferred and the number of cache blocks already stored in the historical queue is greater than the preset historical queue upper limit P: wherein the historical queue upper limit is the upper limit of the number of cache blocks that the historical queue can store in the cache space; if yes, then calculate the relevance score between the query vector and each cache block already stored in the historical queue, and the relevance score between the query vector and each cache block to be transferred, and sort each cache block accordingly, delete the MP cache blocks with the lowest relevance scores, and save the remaining cache blocks to the historical queue; if no, then directly transfer the cache block to be transferred from the recent queue to the historical queue.

[0009] In one embodiment of the present invention, the steps of calculating the relevance score between the query vector and each cache block already stored in the historical queue, and the relevance score between the query vector and each cache block to be transferred, and sorting each cache block accordingly, deleting the MP cache blocks with the lowest relevance scores, and saving the remaining cache blocks to the historical queue include: for each cache block: aggregating each key vector in the cache block to obtain the block representative vector corresponding to the cache block, and calculating the relevance score between the query vector and the block representative vector; sorting all cache blocks according to the relevance scores, deleting the MP cache blocks with the lowest relevance scores, and saving the remaining cache blocks to the historical queue.

[0010] In one embodiment of the present invention, the steps of aggregating the key vectors in the cache block to obtain the block representative vector corresponding to the cache block, and calculating the relevance score between the query vector and the block representative vector include: calculating the mean of each key vector in the cache block in the channel direction as the block representative vector corresponding to the cache block; and calculating the relevance score between the query vector and the block representative vector.

[0011] In one embodiment of the present invention, during the pre-filling stage, multiple query vectors are generated for the input sequence input to the attention layer, and the final query vector is obtained by calculating the average of all query vectors generated by the attention layer in the channel direction.

[0012] This invention also provides an inference system for generative models, applied to a processor. The processor has a cache queue configured according to a cache space. The system includes: a data acquisition module for acquiring data to be processed and inputting the data to be processed into a large language model; wherein the large language model is a generative model based on an attention mechanism and includes multiple cascaded attention layers; and an inference module for performing the following inference for each attention layer: generating corresponding query vectors, key vectors, and value vectors based on the data input to the attention layer, and combining the generated key vectors and value vectors into at least one cache block; monitoring the cache blocks already stored in the cache queue, and... When the cache space capacity limit is exceeded, the cache blocks are filtered based on the relevance scores between the query vector and each cache block to update the cache queue. Based on the attention mechanism, intermediate inference results are obtained according to the query vector and the filtered cache blocks in the cache queue, and then input into the next attention layer to continue inference until the inference result generated by the last attention layer of the large language model is obtained and saved. The result generation module is used to determine whether the inference result meets the preset termination condition. If it does not meet the condition, the inference result is input into the large language model again for inference until the inference result meets the termination condition, and the generated result of the data to be processed is obtained.

[0013] The present invention also provides an electronic device, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the reasoning method for generative models described above.

[0014] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer processor, causes the computer to perform any of the above-mentioned reasoning methods for generative models.

[0015] The beneficial effects of this invention are as follows: This invention proposes an inference method, system, device, and medium for generative models. By setting up a cache queue within the processor and dynamically managing the cache block as the smallest management unit, this invention supports infinitely long contexts under the constraint of limited cache space. Specifically, when the cache queue capacity reaches its limit, cache blocks are filtered based on the relevance score between the query vector and the cache block, thereby retaining context information associated with the current inference task and reducing storage and computational overhead. This achieves a balance between inference accuracy and computational efficiency for long contexts under limited hardware resources. This dynamic management mechanism enables large language models to be deployed on resource-constrained devices such as edge or endpoint devices, thus supporting near-infinite long contexts with limited computing resources. Attached Figure Description The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0016] In the attached diagram: Figure 1 This is a flowchart illustrating a reasoning method for generative models provided in an embodiment of the present invention. Figure 2 This is a structural block diagram of an inference system for generative models provided in one embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0017] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0018] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0019] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0020] The inventors discovered that in the field of artificial intelligence, with the continuous expansion of neural network scale, large language models and multimodal large models based on the Transformer architecture have demonstrated outstanding application value in various application scenarios. As these application scenarios evolve, supporting extremely long contexts has become a fundamental requirement for large language models. For example, tasks such as ultra-long text analysis, ultra-large-scale video understanding, or image understanding all require models to handle longer contexts during the inference phase. However, existing large language models are typically trained on datasets of finite length, thus lacking sufficient support for extremely long contexts during actual inference. Taking the Llama3-8B model as an example, when the context length expands to 32K, its required KV Cache size is approximately 16GB (calculated in BF16 format). As the model size and context length further increase, the memory usage of the KV Cache during inference will continue to grow, and the computational load for attention will also increase linearly, posing a significant challenge for resource-constrained devices such as edge devices.

[0021] To improve upon the above situation, this invention provides an inference method for generative models. Generated cache blocks are sequentially stored in an initial queue. When the initial queue reaches its capacity limit, newly generated cache blocks are moved to a recent queue. When the recent queue reaches its capacity limit, older cache blocks are moved to a historical queue according to a first-in, first-out (FIFO) principle. When the number of cache blocks in the historical queue exceeds its capacity limit, they are sorted based on the relevance score between the query vector and each cache block, retaining only those with higher scores and discarding those with lower scores. This cache management mechanism effectively preserves contextual information highly relevant to the current inference task within a limited cache space, thereby reducing storage and computational overhead while maintaining inference accuracy. This allows generative models to be deployed on edge devices and supports infinitely long contexts.

[0022] like Figure 1 As shown, the inference method for generative models includes the following steps: S100. Obtain the data to be processed and input the data to be processed into the large language model; wherein, the large language model is a generative model based on the attention mechanism and includes multiple cascaded attention layers.

[0023] Understandably, in large language models based on attention mechanisms, each piece of data input to the attention layer generates a query vector (Q), a key vector (K), and a value vector (V). The key and value vectors are frequently used in subsequent inference. To avoid redundant computation, the key vectors and their corresponding value vectors are typically cached to form a KV Cache. This invention builds upon this by combining several key vectors and their corresponding value vectors into a single cache block. Subsequent storage and scheduling within the KV Cache are then performed using this cache block as the smallest management unit, thereby improving cache management efficiency.

[0024] First, it should be noted that this invention applies to processors. A cache space is pre-allocated within the processor's storage resources, and cache blocks are managed in the form of queues. This invention is particularly suitable for neural network processors, which have only one level of storage. The data to be processed in this invention can be single-modal data such as text sequences, single images, or video frame sequences, or multimodal data containing text, images, etc., without specific limitations. The large language model of this invention is a generative model based on an attention mechanism, such as a generative model based on the Transformer architecture. This large language model includes an embedding layer, multiple cascaded attention layers, a feedforward neural network, and an output layer. The data to be processed is input into the embedding layer of the large language model. For text sequences, word segmentation and encoding can be used to convert them into an input sequence composed of multiple tokens. For non-text data such as images or videos, feature extraction can be used to represent them as an input sequence composed of multiple feature vectors. The above input sequences are then input into the attention layers of the large language model, and inference is performed through multiple attention layers to obtain the final generated result.

[0025] S200. For each attention layer, perform the following inference process from S210 to S230: S210. Based on the data input to the attention layer, generate corresponding query vectors, key vectors, and value vectors, and combine the generated key vectors and value vectors into at least one cache block; wherein, the input data of the first attention layer is the data to be processed or the result of feature extraction from the data to be processed.

[0026] Understandably, for the first attention layer, the input sequence can be either the data to be processed itself or the result of feature extraction. For example, for an image or video frame, the input sequence to the first attention layer can be either the original pixel sequence or a feature vector sequence processed by the feature extraction network; there is no specific limitation. After the input sequence is input to this attention layer, for each input element (token) in the input sequence, a query vector, key vector, and value vector corresponding to that input element are generated. After the input sequence is processed according to the above process, the generated key vectors and value vectors are combined according to a preset block size to form at least one cache block, which is stored in a cache queue for subsequent attention calculation and context management.

[0027] It should be noted that, in order to be compatible with the FlashAttention mechanism of the neural network processor, in an optional embodiment of the present invention, the key vector and value vector of every 128 tokens are combined to form a cache block and stored in KVCache. The cache block can be stored using either unquantized data in BF16 format or quantized data formats with low bit widths such as INT8 or INT4, to reduce memory usage and improve inference efficiency; the specific storage method is not limited.

[0028] S220. Monitor the cache blocks already stored in the cache queue, and when they exceed the upper limit of the cache space capacity, filter the cache blocks based on the correlation score between the query vector and each cache block in order to update the cache queue.

[0029] The cache queue is continuously monitored to determine if the number of existing cache blocks exceeds the preset cache space capacity limit. When the number of existing cache blocks is less than or equal to the cache space capacity limit, newly generated cache blocks can be directly written to the cache queue for subsequent computation. Conversely, when the number of existing cache blocks exceeds the cache space capacity limit, cache management is required. Specifically, while saving newly generated cache blocks to the cache queue, the cache blocks are filtered by calculating the correlation score between the query vector generated by the attention layer and each previously stored cache block in the cache queue. Cache blocks with low correlation scores are discarded to update the cache queue. This allows the cache queue to maximize the retention of cached information that contributes most to the current inference task under limited capacity, thus supporting extremely long contexts.

[0030] In an optional embodiment of the present invention, the cache queue includes an initial queue, a recent queue, and a historical queue, and the initial queue, recent queue, and historical queue are empty before the first storage. Step S220 includes S221 to S225: S221. Determine whether the number of cache blocks already stored in the initial queue is less than the preset initial queue upper limit: Wherein, for the same data to be processed, the initial queue remains unchanged during the generation process, and the initial queue upper limit is the upper limit of the number of cache blocks that the initial queue can store in the cache space.

[0031] Specifically, the initial queue is monitored to determine if the number of cached blocks stored within it is less than a preset initial queue limit. This initial queue limit is determined based on the neural network processor's cache space and the language model's network information, limiting the maximum number of cached blocks the initial queue can store. Since the initial queue stores cached blocks generated from the initial input of the prompt sequence, its limit can be 128, meaning it can hold a maximum of 128 cached blocks. It's important to note that the initial queue primarily stores cached blocks generated from the beginning of the input sequence. These cached blocks often carry global context information, significantly impacting the model's subsequent generation results. Therefore, for the same data to be processed, the initial queue remains unchanged during generation; once full, its cached blocks are not replaced but continue to participate in attention calculations as context, ensuring the effectiveness of global information during inference.

[0032] If step S221 is determined to be yes, then step S222 is executed to determine the number of cache blocks generated and stored in the initial queue based on the number of generated cache blocks and the number of remaining storable cache blocks in the initial queue.

[0033] If the number of cache blocks already stored in the initial queue is less than the initial queue's capacity limit, the number of cache blocks that can be written to the initial queue will be determined based on the number of currently generated cache blocks and the number of remaining storable cache blocks in the initial queue. The number of remaining storable cache blocks refers to the number of cache blocks that can still be stored in the initial queue without exceeding its capacity limit. Specifically, if the number of generated cache blocks is greater than the number of remaining storable cache blocks in the initial queue, the first portion of the generated cache blocks will be stored in the initial queue, and the remaining cache blocks will be stored in the recent queue. Conversely, if the number of generated cache blocks is less than or equal to the number of remaining storable cache blocks in the initial queue, all generated cache blocks can be stored in the initial queue. This dynamic allocation mechanism allows the initial queue to maximize the use of cache space to store relevant cache blocks without exceeding its capacity limit. The recent queue is used to store newly generated cache blocks, and its maximum capacity is 4096, which can accommodate 32 cache blocks. The recent queue mainly stores the most recently generated KV cache. Since these cache blocks are highly relevant to the current inference process, they will all participate in attention calculation to ensure the consistency of the generated result with the recent context and the coherence of the generated result with the previous generated result.

[0034] If step S221 is determined to be no, then step S223 is executed, which determines whether the difference between the preset recent queue upper limit and the number of cache blocks already stored in the recent queue is greater than or equal to the number of generated cache blocks: where the recent queue upper limit is the upper limit of the number of cache blocks that the recent queue can store in the cache space.

[0035] If the number of cache blocks already stored in the initial queue reaches the initial queue limit, then all newly generated cache blocks need to be stored in the recent queue. At this point, the difference F (i.e., F=LM) between the recent queue limit L and the number of cache blocks currently stored M needs to be calculated, and it needs to be determined whether the difference F is greater than or equal to the number of cache blocks generated by the current attention layer: if so, it means the recent queue has sufficient capacity, and all newly generated cache blocks can be written into the recent queue. Otherwise, it needs to be processed in conjunction with subsequent queue transfer and historical queue capacity control mechanisms. Furthermore, in another optional embodiment of the invention, to improve cache management efficiency, a sliding window can be set to transfer cache blocks located within the sliding window in the recent queue to the historical queue, thereby achieving ordered updates of cache blocks.

[0036] If step S223 is correct, then step S224 is executed to store all generated cache blocks into the recent queue.

[0037] If the remaining capacity of the recent queue is sufficient to accommodate all currently generated cache blocks, then the currently generated cache blocks can be directly stored in the recent queue.

[0038] If step S223 is not determined, then step S225 is executed. Based on the number of generated cache blocks and the number of remaining storable cache blocks in the recent queue, the number of cache blocks to be transferred from the recent queue is determined, and after being transferred to the historical queue, all generated cache blocks are stored in the recent queue. The historical queue is used for capacity control based on a relevance scoring mechanism.

[0039] If the remaining capacity of the recent queue is insufficient to accommodate all currently generated cache blocks, the number of cache blocks to be moved from the recent queue to the historical queue must be determined based on the number of currently generated cache blocks and the number of remaining storable cache blocks in the recent queue. A first-in, first-out (FIFO) principle can be used, selecting the earliest generated cache blocks from the recent queue as the cache blocks to be moved. After the move is complete, all currently generated cache blocks are stored in the recent queue to ensure that the recent queue maintains the latest context information.

[0040] In an optional embodiment of the present invention, step S225 includes the following steps: Determine if the number of generated cache blocks U is less than or equal to the number of remaining storable cache blocks A in the recent queue: if yes, store all generated cache blocks in the recent queue; if no, based on the first-in-first-out principle, move the first UA cache blocks generated in the recent queue to the historical queue as cache blocks to be moved, and after the move is completed, store all generated cache blocks in the recent queue.

[0041] If the remaining capacity of the recent queue is insufficient to accommodate all newly generated cache blocks, in order to store all these cache blocks into the recent queue, it is necessary to determine whether the number of generated cache blocks (denoted as U) is less than or equal to the number of remaining storable cache blocks in the recent queue (denoted as A): If yes, it means that the recent queue still has enough space, and the generated cache blocks can be directly stored into the recent queue. Otherwise, the table needs to select the earliest generated U from the recent queue based on the first-in, first-out principle. A cache blocks are selected as cache blocks to be moved. After they are moved to the historical queue, the newly generated cache blocks are stored in the recent queue.

[0042] In an optional embodiment of the present invention, the step of performing capacity control of the historical queue based on a correlation scoring mechanism includes: Determine if the sum M of the number of cache blocks to be moved and the number of cache blocks already stored in the historical queue is greater than the preset historical queue limit P: where the historical queue limit is the maximum number of cache blocks that the historical queue can store in the cache space. If yes, calculate the relevance score between the query vector and each cache block already stored in the historical queue, and the relevance score between the query vector and each cache block to be moved. Based on this, sort the cache blocks, delete the MP cache blocks with the lowest relevance scores, and save the remaining cache blocks to the historical queue. If not, directly move the cache block to be moved from the recent queue to the historical queue.

[0043] Specifically, after determining the number of cache blocks to be transferred, the number of cache blocks to be transferred (i.e., U) is calculated. A) Check if the sum of M and the number of cached blocks already stored in the historical queue (denoted as N) is greater than the preset historical queue upper limit P. Here, the historical queue upper limit P is the maximum number of cached blocks that the historical queue can store in the cache space, and the historical queue is empty before the first write. If M... P, at this time U can be A cache blocks to be transferred are directly stored in the history queue. Conversely, if M... At this point, based on the relevance scores between the query vector and existing cache blocks in the historical queue, and the relevance scores between the query vector and the cache blocks to be moved, all cache blocks can be sorted, and the M with the lowest score can be deleted. P cache blocks are used, and the remaining cache blocks are kept in the history queue to ensure that the capacity of the history queue is controlled and to retain the context information that is most relevant to the current inference.

[0044] It should be noted that the historical queue, as a cyclic scheduling region, is primarily used for computation based on the sparse attention mechanism. With TOPK=16, the upper limit of the historical queue can be no less than 4096, meaning it can hold at least 32 cache blocks. When the number of cache blocks in the historical queue is less than 16, all of its cache blocks can be concatenated with cache blocks in the initial and recent queues to participate in attention computation. When the number of cache blocks in the historical queue is greater than 16, a dot product operation (e.g., using GEMV parallel computation) can be performed based on the current query vector and the block representative vector of each cache block to obtain a relevance score, which is then sorted. The 16 cache blocks with the highest scores are selected and used in subsequent attention computations along with cache blocks in the initial and recent queues, thus reducing computational complexity while maintaining model inference accuracy.

[0045] In an optional embodiment of the present invention, the steps of calculating the relevance score between the query vector and each cache block already stored in the historical queue, and the relevance score between the query vector and each cache block to be transferred, sorting the cache blocks accordingly, deleting the MP cache blocks with the lowest relevance scores, and saving the remaining cache blocks to the historical queue include: For each cache block: aggregate the key vectors within the cache block to obtain the block representative vector corresponding to the cache block, and calculate the relevance score between the query vector and the block representative vector.

[0046] The block representative vector is used to characterize the overall characteristics of the corresponding cache block. To control the capacity of the historical queue, the following processing is performed on each cache block: All key vectors in the cache block are aggregated to obtain the block representative vector. The aggregation method includes, but is not limited to, mean aggregation along the channel direction or weighted averaging, etc., and is not limited here. The relevance between the current query vector and the block representative vector is calculated to obtain a relevance score. The relevance score characterizes the similarity between the query vector and the cache block. The calculation method for the relevance score includes, but is not limited to, dot product similarity, cosine similarity, etc., and is not limited here. Further, in an optional embodiment of the present invention, the steps of aggregating the key vectors within the cache block to obtain the block representative vector corresponding to the cache block, and calculating the relevance score between the query vector and the block representative vector include: calculating the mean of each key vector within the cache block in the channel direction as the block representative vector corresponding to the cache block; and calculating the relevance score between the query vector and the block representative vector.

[0047] Specifically, for all key vectors within a cache block, the mean of these key vectors is calculated along the channel direction to obtain the block representative vector. The current query vector is then compared with this block representative vector using methods such as dot product or cosine similarity to calculate the relevance score. This invention leverages the accelerated attention calculation capabilities of neural network processors, using the channel mean vector of the cache block and the query vector for efficient relevance calculation. This reduces computational complexity while improving cache management efficiency, thus better adapting to the inference needs of large models in resource-constrained environments.

[0048] It should be noted that in the pre-filling stage, multiple query vectors are generated for the input sequence to the attention layer. The final query vector is obtained by calculating the average of all query vectors generated by the attention layer along the channel direction. Specifically, after the input sequence is processed by the attention layer, multiple corresponding query vectors are generated. To facilitate subsequent cache management, this invention performs an average operation on all query vectors generated by the attention layer along the channel direction to obtain a final query vector. This final query vector can effectively represent the overall characteristics of the current input sequence at the attention layer, and thus be used to calculate the correlation with the block representative vectors of each cache block. In the encoding stage, only one query vector is generated for the input sequence to the attention layer, which can be used as the final query vector to participate in sparse attention calculation.

[0049] After obtaining the relevance score, all cache blocks are sorted according to the relevance score, the MP cache blocks with the lowest relevance scores are deleted, and the remaining cache blocks are kept in the history queue.

[0050] Specifically, based on the aforementioned relevance calculation method, the relevance score between the query vector and each cache block is obtained. All cache blocks are then sorted accordingly, and the MP cache blocks with the lowest relevance scores are deleted. Here, M is the total number of cache blocks in the current historical queue and the number of cache blocks to be moved, and P is the upper limit of the historical queue. In this way, a few cache blocks most relevant to the current query semantics can be retained within a limited cache space. After deletion, the remaining cache blocks are retained in the historical queue, completing the cache block update and achieving dynamic control of the cache space.

[0051] S230. Based on the attention mechanism, the intermediate inference results are obtained according to the query vector and the filtered cache blocks in the cache queue. The intermediate inference results are then input into the next attention layer to continue inference until the inference results generated by the last attention layer of the large language model are obtained and saved.

[0052] Specifically, the query vector generated by the current attention layer interacts with each cached block retained in the cache queue after relevance filtering, performing computations to obtain contextual information through attention weighting. This contextual information is then combined with the key and value vectors of this layer to obtain an intermediate inference result. This intermediate inference result not only reflects the relevance features between the current input and the historical context but also incorporates the filtered contextual information. This intermediate inference result is then passed as input to the next attention layer for further processing, and the model iteratively optimizes layer by layer in the multi-layered cascaded attention computation until the final attention layer generates and saves the final inference result.

[0053] S300. Determine whether the reasoning result meets the preset termination condition. If not, input the reasoning result back into the large language model for reasoning until the reasoning result meets the termination condition, and obtain the generated result of the data to be processed.

[0054] After obtaining the inference result output from the last attention layer of the large language model, the inference result can be further evaluated to determine whether it meets the preset termination conditions. These termination conditions include, but are not limited to, whether a preset termination symbol appears in the generated result or whether the length of the output sequence reaches a preset length threshold. If the current inference result does not meet the termination conditions, it is used as new input and fed back into the large language model for further inference, continuing to be processed through each level of attention layer to generate a new inference result. If the current inference result meets the termination conditions, the iterative inference stops, and the obtained inference results are arranged sequentially to form the final generated result. This approach ensures the integrity of the generated content while avoiding redundant computation, thereby improving not only inference efficiency but also enhancing the final generation quality by incorporating the most relevant contexts.

[0055] As shown in Table 1, existing solutions each have their shortcomings when supporting ultra-long contexts: the token-by-token filtering method has excessive computational cost, making it difficult to deploy efficiently on resource-constrained devices. The Quest paging scheme relies on external storage, and its computational complexity remains high when using sparse attention indexing and computation. The sparse attention method is only suitable for the pre-filling stage and is not suitable for execution under neural network processor architectures. Although the Stream LLM sliding window method has a lower computational cost, its accuracy suffers significant loss in long context scenarios. In contrast, this invention achieves a dual reduction in indexing and attention computational costs by using a mechanism that combines block management with sparse attention computation within a fixed-size cache space, and ensures the accuracy of long context inference under limited hardware resource constraints, with minimal accuracy loss. Therefore, this invention can achieve efficient support for ultra-long contexts in resource-constrained scenarios such as edge devices and end-device devices. Furthermore, this method of the invention was tested on the LongBench benchmark. Experimental results show that, with concise and efficient indexing computation, high memory utilization in cache block management, and support for both cache discarding and sparse attention computation, the inference accuracy of this invention is close to that of full attention without cache discarding, with an accuracy loss of less than 1%.

[0056] Table 1. Comparison of the present invention with existing context processing schemes.

[0057] The overall process of this invention is as follows: The cache blocks generated from the input sequence are first stored in the initial queue. After the initial queue is full, new cache blocks enter the recent queue. When the capacity of the recent queue reaches its limit, the earliest generated cache blocks (denoted as U) are then processed. A (where U is the number of generated cache blocks, and A is the number of remaining storable cache blocks in the recent queue) will be moved to the historical queue. In the historical queue, the relevance score between the block representative vector and the query vector is calculated for all cache blocks, and they are sorted based on their scores. The K most important cache blocks are selected to participate in the attention calculation, while the U with the lowest score is removed. A P cache blocks (where P is the upper limit of the history queue) are discarded to keep the history queue full and prevent overflow. The retained history queue cache blocks, along with the cache blocks in the initial queue and the recent queue, participate in attention calculation, thereby achieving efficient management and sparse computation of long contexts under the constraint of limited cache space.

[0058] In summary, the two innovations of this invention—support for infinitely long contexts under finite KV cache constraints and sparse computation based on attention mechanisms—enable the efficient deployment of large language models on resource-constrained devices such as edge and endpoints. Under these conditions, the model can not only support tasks with extremely long prompt sequences and multi-turn dialogues, but also ensure that inference speed does not increase linearly with the length of the context, thus guaranteeing the universality and efficiency of model inference. This invention organizes and manages the KV cache in a block-based manner, combining multiple key vectors and value vectors to form cache blocks, and storing and scheduling these cache blocks as the smallest management unit. As the context input to the large language model increases, the KV cache space pre-allocated by the compiler is gradually filled. When the capacity limit is reached, the cache management strategy of this scheme is triggered. By prioritizing the existing cache blocks in the historical queue and the cache blocks to be moved, the highest-scoring cache blocks are selected and used together with the cache blocks in the recent queue and the initial queue to participate in attention calculation. When the sum of the number of cache blocks to be moved and the existing cache blocks in the historical queue exceeds the upper limit of the historical queue, several cache blocks with lower scores in the ranking are discarded, thereby achieving efficient context management with limited cache resources.

[0059] like Figure 2 As shown, the inference system for the generative model includes a data acquisition module 210, an inference module 220, and a result generation module 230. The data acquisition module 210 acquires the data to be processed and inputs it into the large language model; the large language model is a generative model based on an attention mechanism and includes multiple cascaded attention layers. The inference module 220 performs the following inference for each attention layer: based on the data input to the attention layer, it generates corresponding query vectors, key vectors, and value vectors, and combines the generated key vectors and value vectors into at least one cache block; it monitors the cache blocks already stored in the cache queue, and when they exceed the upper limit of the cache space capacity, it filters the cache blocks based on the relevance score between the query vector and each cache block to update the cache queue; based on the attention mechanism, it obtains intermediate inference results according to the query vector and the filtered cache blocks in the cache queue, and inputs them into the next attention layer to continue inference until the inference result generated by the last attention layer of the large language model is obtained and saved. The above-mentioned result generation module 230 is used to determine whether the reasoning result meets the preset termination condition. If it does not meet the condition, the reasoning result is input into the large language model again for reasoning until the reasoning result meets the termination condition, and the generation result of the data to be processed is obtained.

[0060] Specific limitations regarding the inference system used for generative models can be found in the limitations on inference methods used for generative models described above, and will not be repeated here. Each module in the aforementioned inference system for generative models can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware format, or stored in the memory of a computer device in software format, so that the processor can call the corresponding operations of each module.

[0061] It should be noted that, in order to highlight the innovative aspects of this invention, this embodiment does not include modules that are not closely related to solving the technical problems proposed by this invention, but this does not mean that there are no other modules in this embodiment.

[0062] like Figure 3 As shown, the electronic device 3 may include a memory 31, a processor 32 and a bus, and may also include a computer program stored in the memory 31 and executable on the processor 32, such as an inference program for generative models.

[0063] The memory 31 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 31 can be an internal storage unit of the electronic device 3, such as a portable hard drive. In other embodiments, the memory 31 can be an external storage device of the electronic device 3, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 3. Furthermore, the memory 31 can include both internal and external storage units of the electronic device 3. The memory 31 can be used not only to store application software and various types of data installed on the electronic device 3, such as code used for generative model inference, but also to temporarily store data that has been output or will be output.

[0064] In some embodiments, processor 32 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. Processor 32 is the control unit of electronic device 3, connecting various components of the entire electronic device 3 via various interfaces and lines. It executes programs or modules stored in memory 31 (e.g., inference programs for generative models) and calls data stored in memory 31 to perform various functions of electronic device 3 and process data.

[0065] Processor 32 executes the operating system of electronic device 3 and various installed applications. Processor 32 executes the applications to implement the steps in the inference method for generative models described above.

[0066] For example, a computer program may be divided into one or more modules, one or more of which are stored in memory 31 and executed by processor 32 to complete this application. One or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in electronic device 3. For example, the computer program may be divided into a data acquisition module 210, an inference module 220, and a result generation module 230.

[0067] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium, which can be non-volatile or volatile. The software functional module stored in the storage medium includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute some of the functions of the reasoning methods for generative models in the various embodiments of this application.

[0068] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A reasoning method for generative models, characterized in that, Applied to a processor, wherein a cache queue is configured according to the cache space, the method includes: Acquire the data to be processed and input the data to be processed into the large language model; wherein the large language model is a generative model based on the attention mechanism and includes multiple cascaded attention layers; For each attention layer, the following reasoning is performed: Based on the data input to the attention layer, corresponding query vectors, key vectors, and value vectors are generated, and the generated key vectors and value vectors are combined into at least one cache block; Monitor the cache blocks already stored in the cache queue, and when they exceed the capacity limit of the cache space, filter the cache blocks based on the correlation score between the query vector and each cache block in order to update the cache queue; Based on the attention mechanism, intermediate inference results are obtained according to the query vector and each cache block in the updated cache queue, and then input into the next attention layer to continue inference until the inference result generated by the last attention layer of the large language model is obtained and saved. Determine whether the reasoning result meets the preset termination condition. If not, input the reasoning result back into the large language model for reasoning until the reasoning result meets the termination condition, and obtain the generation result of the data to be processed.

2. The reasoning method for generative models according to claim 1, characterized in that, The cache queue includes an initial queue, a recent queue, and a historical queue, and the initial queue, the recent queue, and the historical queue are empty before the first storage. The step of monitoring the cache blocks already stored in the cache queue and, when they exceed the capacity limit of the cache space, filtering the cache blocks based on the correlation score between the query vector and each cache block to update the cache queue includes: Determine whether the number of cache blocks already stored in the initial queue is less than the preset initial queue upper limit: wherein, for the same data to be processed, the initial queue remains unchanged during the generation process, and the initial queue upper limit is the upper limit of the number of cache blocks that the initial queue can store in the cache space; If so, the number of cache blocks generated and the number of remaining storable cache blocks in the initial queue are used to determine the number of cache blocks generated and stored in the initial queue. If not, determine whether the difference between the preset recent queue upper limit and the number of cache blocks already stored in the recent queue is greater than or equal to the number of generated cache blocks: wherein, the recent queue upper limit is the upper limit of the number of cache blocks that the recent queue can store in the cache space; If so, all generated cache blocks will be stored in the recent queue; If not, based on the number of generated cache blocks and the number of remaining storable cache blocks in the recent queue, determine the number of cache blocks to be transferred from the recent queue, and after transferring them to the historical queue, store all generated cache blocks in the recent queue; wherein, the historical queue is used for capacity control based on a relevance scoring mechanism.

3. The reasoning method for generative models according to claim 2, characterized in that, The step of determining the number of cache blocks to be transferred from the recent queue based on the number of generated cache blocks and the number of remaining storable cache blocks in the recent queue, and storing all generated cache blocks in the recent queue after transferring them to the historical queue, includes: Determine whether the number of generated cache blocks U is less than or equal to the number of remaining storable cache blocks A in the recent queue: If so, all generated cache blocks will be stored in the recent queue; If not, based on the first-in-first-out principle, the first UA cache blocks generated in the recent queue are transferred to the historical queue as cache blocks to be transferred, and after the transfer is completed, all generated cache blocks are stored in the recent queue.

4. The reasoning method for generative models according to claim 2, characterized in that, The steps for capacity control of the historical queue based on the correlation scoring mechanism include: Determine whether the sum M of the number of cache blocks to be transferred and the number of cache blocks already stored in the historical queue is greater than the preset historical queue upper limit P: where the historical queue upper limit is the upper limit of the number of cache blocks that the historical queue can store in the cache space; If so, calculate the correlation score between the query vector and each cache block already stored in the historical queue, and the correlation score between the query vector and each cache block to be transferred, and sort each cache block accordingly, delete the MP cache blocks with the lowest correlation scores, and save the remaining cache blocks to the historical queue. If not, the cache block to be transferred is directly transferred from the recent queue to the historical queue.

5. The reasoning method for generative models according to claim 4, characterized in that, The steps of calculating the relevance scores between the query vector and each cache block already stored in the historical queue, and the relevance scores between the query vector and each cache block to be transferred, and sorting the cache blocks accordingly, deleting the MP cache blocks with the lowest relevance scores, and saving the remaining cache blocks to the historical queue include: For each cache block: aggregate the key vectors within the cache block to obtain the block representative vector corresponding to the cache block, and calculate the relevance score between the query vector and the block representative vector; Sort all cache blocks according to their relevance scores, delete the MP cache blocks with the lowest relevance scores, and keep the remaining cache blocks in the history queue.

6. The reasoning method for generative models according to claim 5, characterized in that, The steps of aggregating the key vectors within the cache block to obtain the block representative vector corresponding to the cache block, and calculating the relevance score between the query vector and the block representative vector include: Calculate the mean of each key vector in the channel direction within the cache block, and use it as the block representative vector corresponding to the cache block; Calculate the relevance score between the query vector and the block representative vector.

7. The reasoning method for generative models according to claim 1, characterized in that, During the pre-filling stage, multiple query vectors are generated for the input sequence to the attention layer. The final query vector is obtained by calculating the mean of all query vectors generated by the attention layer in the channel direction.

8. A reasoning system for generative models, characterized in that, The system is applied to a processor, wherein a cache queue is configured according to the cache space, and the processor includes: The data acquisition module is used to acquire data to be processed and input the data to be processed into the large language model; wherein the large language model is a generative model based on an attention mechanism and includes multiple cascaded attention layers; The inference module is used to perform the following inference for each attention layer: Based on the data input to the attention layer, corresponding query vectors, key vectors, and value vectors are generated, and the generated key vectors and value vectors are combined into at least one cache block; Monitor the cache blocks already stored in the cache queue, and when they exceed the capacity limit of the cache space, filter the cache blocks based on the correlation score between the query vector and each cache block in order to update the cache queue; Based on the attention mechanism, intermediate inference results are obtained according to the query vector and each cache block filtered in the cache queue, and then input into the next attention layer to continue inference until the inference result generated by the last attention layer of the large language model is obtained and saved. The result generation module is used to determine whether the reasoning result meets the preset termination condition. If it does not meet the condition, the reasoning result is input into the large language model again for reasoning until the reasoning result meets the termination condition, and the generation result of the data to be processed is obtained.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the reasoning method for generative models as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the reasoning method for generative models as described in any one of claims 1 to 7.

Images