Context vector generation method and device

Abstract

The invention discloses a context vector generation method and device, and relates to the technical field of machine learning, and the method comprises the steps: receiving a target vector; a first sub-vector of the target vector is generated according to the first vector information, a second sub-vector generated by a processor of the reasoning equipment is obtained, the first vector information is stored in a first storage space on the acceleration card, and the first sub-vector is used for indicating the relation between the target vector and a first historical vector; the second sub-vector is used for indicating a relationship between the target vector and a second historical vector, and the second vector information is stored in a second storage space on the processor; and generating a context vector of the target vector according to the first sub-vector and the second sub-vector. According to the method and the device, the technical problem that the storage space of the accelerator card is occupied by the historical information is solved, and the technical effect of reducing the storage space occupied by the historical information on the accelerator card under the condition of keeping the model operation precision is achieved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of machine learning, and specifically, to a method and device for generating a context vector. Background Art

[0002] With the continuous advancement of modern technology, using models to solve problems is becoming increasingly common. To avoid repeated processing of input tokens during inference, current models typically store information about previously input tokens on an accelerator card for subsequent inference. However, as models scale, the overhead of caching historical token information increases, becoming a key bottleneck in model deployment. Summary of the Invention

[0003] The embodiments of the present application provide a method and apparatus for generating a context vector, so as to at least solve the technical problem in the related art that historical information occupies a large amount of storage space on an accelerator card.

[0004] According to one embodiment of the present application, a method for generating a context vector is provided, which is applied to an accelerator card of an inference device, and the method includes: receiving a target vector; generating a first sub-vector of the target vector based on first vector information, and obtaining a second sub-vector generated by a processor of the inference device, wherein the first vector information is stored in a first storage space on the accelerator card, the first vector information is used to indicate a first historical vector received historically, the first sub-vector is used to indicate a relationship between the target vector and the first historical vector, the second sub-vector is used to indicate a relationship between the target vector and the second historical vector, the second vector information is stored in a second storage space on the processor, the second vector information is used to indicate a second historical vector received historically, and the influence of the first historical vector on the target vector is higher than the influence of the second historical vector on the target vector; and generating a context vector of the target vector based on the first sub-vector and the second sub-vector.

[0005] According to another embodiment of the embodiments of the present application, a context vector generation device is also provided, which is applied to an accelerator card of an inference device, and the device includes: a receiving module for receiving a target vector; a first generating module for generating a first sub-vector of the target vector based on the first vector information, and obtaining a second sub-vector generated by the processor of the inference device, wherein the first vector information is stored in a first storage space on the accelerator card, the first vector information is used to indicate the first historical vector received historically, the first sub-vector is used to indicate the relationship between the target vector and the first historical vector, the second sub-vector is used to indicate the relationship between the target vector and the second historical vector, the second vector information is stored in a second storage space on the processor, the second vector information is used to indicate the second historical vector received historically, and the influence of the first historical vector on the target vector is higher than the influence of the second historical vector on the target vector; and a second generating module is used to generate a context vector of the target vector based on the first sub-vector and the second sub-vector.

[0006] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned methods for generating a context vector when executing the computer program.

[0007] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for generating a context vector are implemented.

[0008] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned methods for generating a context vector when executed by a processor.

[0009] Through the present application, the accelerator card of the inference device receives a target vector, generates a first subvector of the target vector based on the first vector information stored in the first storage space of the accelerator card, and obtains a second subvector generated by the processor of the inference device, wherein the first subvector is used to indicate the relationship between the target vector and the first historical vector, and the second subvector is used to indicate the relationship between the target vector and the second historical vector stored in the second storage space of the processor. The influence of the first historical vector on the target vector is higher than the influence of the second historical vector on the target vector. The historical vector information of the historical vector no longer needs to be stored only on the accelerator card, which reduces the storage space occupied by the accelerator card. The accelerator card will eventually generate a context vector of the target vector based on the first subvector and the second subvector, thereby achieving the goal of maintaining the model's operating accuracy as much as possible while reducing the storage space occupied by the accelerator card and avoiding the introduction of additional errors. Therefore, it can solve the technical problem in the related art that historical information occupies a large amount of storage space of the accelerator card, and achieve the technical effect of reducing the storage space occupied by historical information on the accelerator card while maintaining the model's operating accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1 This is a hardware structure block diagram of a computer device for a method for generating a context vector according to an embodiment of the present application;

[0012] Figure 2 is a flowchart of a method for generating a context vector according to an embodiment of the present application;

[0013] Figure 3 is a schematic diagram of a method for generating a context vector according to an embodiment of the present application;

[0014] Figure 4 This is a schematic diagram of a process of a key-value cache uninstallation method according to an embodiment of the present application;

[0015] Figure 5 is a structural block diagram of a device for generating a context vector according to an embodiment of the present application;

[0016] Figure 6 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0018] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0019] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0020] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a computer device for a method of generating a context vector according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. The server device may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0021] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for generating the context vector in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to a server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0022] Transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a communication provider of the server device. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0023] This application proposes a method for generating a context vector. Before describing the optional embodiments of this application, in order to better understand the inventive concept and creativity of this application, the relevant technologies are first described:

[0024] The inference process of a model, such as the Large Language Model (LLM), generates tokens incrementally, a process called autoregressive generation. First, the text input by the user is segmented into a series of tokens, embedded into word vectors, and fed into the model. The model then uses forward computation to predict the next most likely token in the current context. The model then samples a token based on the predicted probability distribution and adds it to the existing sequence, performing inference again as new input. This process is repeated, generating only one token at a time, until a terminator is generated or the maximum length is reached. The entire generation process relies on previous tokens, gradually building a coherent text.

[0025] To improve efficiency during model inference, you can use the KV (Key-Value) cache mechanism. This helps the model avoid recalculating previous token information during runtime, thereby accelerating the inference process. The following briefly describes how this mechanism is implemented:

[0026] In the model's self-attention mechanism, each input token generates three vectors: the query, key, and value. During standard inference, all inputs require recalculating these three vectors each time they are generated. When using KV caching, the model caches the previously calculated key and value vectors. This allows it to directly use these cached keys and values ​​when generating the next token, without having to recalculate all inputs. This caching mechanism is particularly efficient when generating long text, reducing computational effort.

[0027] The pre-population phase of the model inference process processes the input sequence in the user request through the model all at once, converting it into a key-value cache for use in the decoding phase. Decoding is based on the context of the pre-population phase, with the model gradually generating new tokens until the target length or the end symbol is reached. Using the key-value cache during decoding avoids repeated computations and improves efficiency.

[0028] Currently popular models generally use a multi-layered architecture, with each layer containing a multi-head attention mechanism and a feed-forward neural network (FFN), with embedding layers and MLPs (multi-layer perceptrons) at the beginning and end of the model. The multi-head attention mechanism consists of multiple self-attention mechanisms, so the size of the key-value cache increases linearly with the number of heads in the multi-head attention mechanism and the number of model layers.

[0029] LLM deployment is extremely expensive, typically requiring the use of multiple high-performance GPUs (Graphics Processing Units) or other hardware accelerators to meet the dual requirements of computational speed and storage capacity during inference. As context length increases, the storage of the KV cache stored on the GPU quickly becomes a new bottleneck in the system. Therefore, effectively reducing the storage space and bandwidth pressure of the KV cache without compromising model accuracy and inference quality has become a core issue that needs to be addressed in current LLM deployments.

[0030] This embodiment provides a method for generating a context vector, which is applied to an accelerator card of an inference device. Figure 2 is a flowchart of a method for generating a context vector according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0031] Step S202, receiving a target vector;

[0032] Step S204: Generate a first subvector of the target vector based on the first vector information, and obtain a second subvector generated by the processor of the inference device, wherein the first vector information is stored in a first storage space on the accelerator card, the first vector information is used to indicate a historically received first historical vector, the first subvector is used to indicate a relationship between the target vector and the first historical vector, the second subvector is used to indicate a relationship between the target vector and the second historical vector, the second vector information is stored in a second storage space on the processor, the second vector information is used to indicate a historically received second historical vector, and the influence of the first historical vector on the target vector is greater than the influence of the second historical vector on the target vector;

[0033] Step S206 : Generate a context vector of the target vector according to the first sub-vector and the second sub-vector.

[0034] Through the above steps, the accelerator card of the inference device receives the target vector, generates a first subvector of the target vector based on the first vector information stored in the first storage space of the accelerator card, and obtains the second subvector generated by the processor of the inference device, the first subvector is used to indicate the relationship between the target vector and the first historical vector, and the second subvector is used to indicate the relationship between the target vector and the second historical vector stored in the second storage space of the processor. The influence of the first historical vector on the target vector is higher than the influence of the second historical vector on the target vector. The historical vector information of the historical vector no longer needs to be stored only on the accelerator card, which reduces the storage space occupied by the accelerator card. The accelerator card will eventually generate a context vector of the target vector based on the first subvector and the second subvector, thereby achieving the goal of maintaining the model's operating accuracy as much as possible while reducing the storage space occupied by the accelerator card and avoiding the introduction of additional errors. Therefore, the technical problem in the related art that the historical information occupies a large amount of storage space of the accelerator card can be solved, and the technical effect of reducing the storage space occupied by the historical information on the accelerator card while maintaining the model's operating accuracy can be achieved.

[0035] Optionally, in this embodiment, the method may be applied to, but is not limited to, an accelerator card for an inference device. An inference device may refer to, but is not limited to, a hardware device used to perform machine learning model inference (or inference) tasks. This type of device is not limited to traditional CPUs (Central Processing Units) but also includes various hardware accelerators specifically designed to accelerate deep learning or machine learning operations. For example, inference devices may include, but are not limited to: servers, desktop computers or workstations, embedded devices (such as IoT devices, smart cameras, smart speakers, etc.), mobile devices (such as smartphones and tablets, which have built-in APUs (AI Processing Units) or dedicated AI accelerators for real-time AI tasks such as image recognition or voice assistants), ASICs (Application-Specific Integrated Circuits), etc. Optionally, an accelerator card may be, but is not limited to, a plug-in hardware device used to expand the computing power of a host computer (such as a server or workstation). Specifically, accelerator cards may include, but are not limited to, GPUs, FPGAs (Field-Programmable Gate Arrays), ASICs, DSPs (Digital Signal Processors), etc. Optionally, the inference device may include, but is not limited to, a processor in addition to an accelerator card. The processor may refer to, but is not limited to, a CPU, which is the brain of the computer system and is responsible for interpreting and executing software instructions. The processor may be connected to, but is not limited to, one or more accelerator cards. The context vector generation method described in this application may be run independently on each accelerator card, but is not limited to running independently on each accelerator card.

[0036] Optionally, in this embodiment, a large language model including a multi-layer structure and a multi-head attention mechanism can be deployed in a system including a processor and one or more accelerator cards, but is not limited to it. In the multi-head attention mechanism of LLM, each time a token is input into the model, the key vector (hereinafter referred to as key) and value vector (hereinafter referred to as value) calculated in the process will be cached. In this way, when generating each token, the model can directly use these cached keys and values ​​without recalculating all the inputs. Specifically, in the multi-head attention mechanism, for the input vector of the tth token , first through three linear transformation matrices Get the query, key, and value vectors:

[0037] ;

[0038] ;

[0039] ;

[0040] Among them, the embedding dimension Represents the dimension of the input vector (such as the embedding of a token), represents the number of heads in multi-head attention, Represents the dimension of each attention head, the input vector Represents the input of the tth token in the attention layer. , which is the overall representation of multiple heads. In order to calculate multiple attention heads in parallel, the above vector can be split into dimensional subvectors: ; ; . Each subvector here Corresponding to the i-th attention head. The output of each attention head is The calculation is as follows:

[0041] ;

[0042] Because the softmax function involves exponential operations, in actual engineering use, in order to avoid overflow and underflow problems of softmax, you can use, but are not limited to, the following formula:

[0043] ;

[0044] Subtract one The maximum value in , thus avoiding the overflow problem of the softmax function. The output of all attention heads is concatenated, and then an output mapping matrix is ​​used Perform a linear transformation to obtain the final multi-head attention output (i.e. the generated context vector):

[0045] ;

[0046] This vector Dimensions and input The same is d, and can continue to be input to subsequent layers. The method for generating a context vector described in this application can be, but is not limited to, applied to each of the multiple attention heads mentioned above. Each attention head will not affect other attention heads or other structural layers when applying this method, that is, each attention head can independently use the method given in this application, and each structural layer can independently apply the method given in this application according to the layer relationship between each structural layer. Specifically, the multiple attention heads in each structural layer can, but are not limited to, independently but simultaneously execute the method for generating a context vector described in this application, and each structural layer can, but is not limited to, independently but sequentially execute the method for generating a context vector described in this application according to the order of the structural layers.

[0047] Optionally, in this embodiment, context vectors play a core role in natural language processing and deep learning, especially in recurrent neural networks, long short-term memory networks, gated recurrent units, and modern Transformer models (an encoder-decoder architecture). The context vector can be, but is not limited to, representing a summary of information in a certain sequence or context, reflecting the model's understanding of the current input and the relationship between the current input and historical input. As mentioned above, in LLM reasoning, the context vector is particularly important. The context vector can be, but is not limited to, being able to capture the historical context information of the text sequence, helping the model to generate more coherent and meaningful output. Specifically, the context vector can be, but is not limited to, when processing each input token, the model generates a vector by considering all previous tokens to form a prediction for the current token. This vector integrates the features of previous tokens, helping the model make decisions based on more comprehensive historical information when generating the next token.

[0048] It should be noted that the context vector generation method provided in this application can be applied to, but is not limited to, large language models, or can also be applied to other models that require the generation of context vectors, such as video action recognition models and audio recognition and synthesis models. When applied to a video action recognition model, the video action recognition model processes a sequence of video frames and can, but is not limited to, understand the continuity of actions and changes in scenes by generating context vectors. When applied to an audio recognition and synthesis model, it can, but is not limited to, generate context vectors by processing the time series of audio signals to identify or generate syllables, melodies, etc.

[0049] Optionally, in the embodiment provided in step S202, the target vector may be, but is not limited to, a formatted representation of the basic reasoning information received by the model. For example, the target vector may be, but is not limited to, the embedding vector obtained after the input token is converted. As previously discussed, whenever a new token t is input into the model, it will be converted into the corresponding vector form (i.e., the input vector ), which lays the foundation for the subsequent reasoning process.

[0050] Optionally, in the embodiment provided in step S204, the first vector information may be, but is not limited to, used to indicate a previously received first historical vector, and the second vector information may be, but is not limited to, used to indicate a previously received second historical vector. It should be noted that the first historical vector and the second historical vector are different historical vectors, and the influence of the first historical vector on the target vector may be, but is not limited to, greater than the influence of the second historical vector on the target vector.

[0051] Optionally, in this embodiment, the influence of each historical vector on the currently input target vector may refer to, but is not limited to, the strength of the correlation or information contribution between them. This influence assessment may be based on, but is not limited to, the weights or scores of the historical vectors and the target vector in a specific calculation process. For example, in time series analysis, the correlation between historical data points and current data points can be measured by recent similarity, trend matching, or statistical correlation; in video action recognition, the influence of historical frames on the current frame can be assessed by the similarity of motion patterns or the continuity of features.

[0052] Optionally, in this embodiment, the influence of each historical vector on the currently input target vector may be determined by, but is not limited to, comparing the cumulative importance of each historical vector to the last input vector before the target vector.

[0053] Optionally, in this embodiment, the first vector information may be, but is not limited to, stored in a first storage space on the accelerator card. For example, the first vector information may be, but is not limited to, stored in a video memory on a GPU.

[0054] Optionally, in this embodiment, the second vector information may be, but is not limited to, stored in a second storage space on the processor. For example, the second vector information may be, but is not limited to, stored in a memory on the CPU.

[0055] Optionally, in this embodiment, the first sub-vector may be used, but is not limited to, to indicate the relationship between the newly input target vector and the first historical vector, and the second sub-vector may be used, but is not limited to, to indicate the relationship between the newly input target vector and the second historical vector. The nature and calculation method of this relationship may, but is not limited to, vary according to the data type and task requirements processed by the model. For example, in a time series prediction model, this relationship may, but is not limited to, be reflected as a trend continuity or periodic association between historical data points and current data points; in a video analysis model, this relationship may, but is not limited to, represent the continuity between the previous frame and the current frame in object recognition or motion trajectory. In all different application contexts, the first sub-vector and the second sub-vector may, but is not limited to, help the model more accurately understand the position and role of the current input in the sequence by capturing and quantifying this relationship, thereby optimizing the subsequent reasoning process.

[0056] Optionally, in this embodiment, obtaining the second sub-vector generated by the processor of the inference device may include, but is not limited to, waiting to receive the second sub-vector sent by the processor of the inference device within a target time period before and after generating the first sub-vector, or may include, but is not limited to, reading the second sub-vector from a fixed memory area of ​​the processor of the inference device after generating the first sub-vector, etc.

[0057] Optionally, in this embodiment, obtaining the second sub-vector generated by the processor of the inference device may include, but is not limited to, obtaining the second sub-vector of the target vector generated by the processor of the inference device based on the second vector information stored in the second storage space, or obtaining the second sub-vector of the target vector generated by the processor of the inference device based on partial vector information in the second vector information stored in the second storage space, and so on.

[0058] Optionally, in the embodiment provided in step S206, after the accelerator card generates the first sub-vector based on the first vector information and obtains the second sub-vector generated by the processor of the inference device, a context vector of the target vector is generated based on the first sub-vector and the second sub-vector. The context vector here may refer to, but is not limited to, the output of each attention head as described above. .

[0059] Optionally, in this embodiment, generating the context vector of the target vector based on the first sub-vector and the second sub-vector may include, but is not limited to, fusing the first sub-vector and the second sub-vector according to a default first fusion ratio (e.g., 70%) and a default second fusion ratio (e.g., 30%) to obtain the context vector. The default first fusion ratio and the default second fusion ratio may be, but are not limited to, flexibly set based on experience.

[0060] Optionally, in this embodiment, generating the context vector of the target vector based on the first sub-vector and the second sub-vector may also, but is not limited to, include generating a first fusion factor based on the first vector information while generating the first sub-vector, and obtaining a second fusion factor generated by the processor while obtaining the second sub-vector generated by the processor, and fusing the first sub-vector and the second sub-vector according to the first fusion factor and the second fusion factor to obtain the context vector.

[0061] Optional, Figure 3 is a schematic diagram of a method for generating a context vector according to an embodiment of the present application. Figure 3 As shown, a large language model comprising N layers can be deployed, but is not limited to being deployed. Each layer includes a multi-head attention mechanism and a feedforward neural network, with embedding layers and MLP layers at the beginning and end of the large language model. Research has found that in attention calculations, the most recent token and a few distant key tokens are more likely to be attended to by the current token. When GPU memory resources are limited, it is possible to evict distant and less influential tokens to reduce memory usage. Because the importance of a small number of tokens changes dynamically, tokens that are initially evicted may become important again during subsequent decoding. Therefore, it is possible to evict the KV cache of distant and less influential tokens from GPU memory and offload them to CPU memory. Subsequently, during each decoding iteration, in addition to using the KV cache in GPU memory for dense attention calculations, it is possible to retrieve important tokens from the KV cache in the CPU for sparse attention calculations. After computing sparse attention on the CPU, an independent computation strategy is used to calculate the CPU's attention head output, transfer it to the GPU, and merge it with the output of the same attention head on the GPU. By using this attention head input calculation method (i.e., the method for generating context vectors), the KV cache can reduce GPU memory usage while maintaining the accuracy and inference quality of large language models.

[0062] As an optional solution, generating a first sub-vector of the target vector based on the first vector information can be achieved but is not limited to the following method: generating a first sub-vector based on the first vector information, and generating a first fusion factor of the first sub-vector based on the first vector information, wherein the first fusion factor is used to indicate the fusion ratio of the first sub-vector, and the first fusion factor is used to fuse the first sub-vector and the second sub-vector to generate a context vector.

[0063] Optionally, in this embodiment, the first fusion factor of the first sub-vector can be generated according to the first vector information at the same time as the first sub-vector, but is not limited to it. Compared with the solution of directly using the default fusion factor, the use of the first fusion factor calculated according to the first vector information can better fuse the first sub-vector in combination with the distribution of the first historical vector, thereby reducing the model operation accuracy deviation caused by generating the context vector by vector fusion.

[0064] Optionally, in this embodiment, the first sub-vector can be generated according to the first vector information in the following manner, but is not limited to: calculating the correlation between each first vector and the target vector based on the query vector and the key vector of each first vector, wherein the first vector information includes the key vector of the first historical vector and the value vector of the first historical vector, and the first vector includes the first historical vector and the target vector; calculating the correlation between each first vector and the target vector, and adding the product of the product and the value vector of the corresponding first vector to obtain the first sub-vector.

[0065] Optionally, in this embodiment, the first subvector can be calculated by, but is not limited to, the following formula: (i.e. O1):

[0066] ;

[0067] in, That is, the correlation between each first vector and the target vector, This is the value vector of each first vector. The first subvector can be obtained by, but is not limited to, traversing each first vector, calculating the correlation between each first vector and the target vector, then multiplying each correlation by the value vector of the corresponding first vector, and finally summing the products to obtain the first subvector.

[0068] Optionally, in this embodiment, the correlation between each first vector and the target vector can be calculated based on the query vector and the key vector of each first vector using the following formula, but is not limited to: : , where t1 is the number of the first vector, and max(c) can be but is not limited to refer to hour, The maximum value of That is, the query vector, That is, the key vector of each first vector, That is, the vector dimension of the query vector.

[0069] As an optional solution, the first fusion factor of the first sub-vector can be generated based on the first vector information in the following manner, but is not limited to: calculating the dot product of the query vector of the target vector and the key vector of each first vector to obtain multiple first dot product values ​​corresponding one-to-one to each first vector, wherein the first vector includes the first history vector and the target vector, and the first vector information includes the key vector of the first history vector; calculating the quotient between each first dot product value and the square root of the vector dimension of the query vector to obtain multiple first intermediate values; and generating the first fusion factor based on each first intermediate value.

[0070] Optionally, in this embodiment, the target vector may be, but is not limited to, in the form of a column vector, and the query vector for calculating the target vector may be, but is not limited to, Key vectors with each first vector The dot product of: , where x belongs to [1, t1], and t1 is the number of the first vector.

[0071] Optionally, in this embodiment, the target vector may also exist in the form of, but not limited to, a row vector, and the query vector for calculating the target vector may be, but not limited to, Key vectors with each first vector The dot product of: , where x belongs to [1, t1], and t1 is the number of the first vector. Various representations of the target vector have been shown here. The following procedures use column vectors as examples. The procedures for row vector representation are similar and will not be repeated here.

[0072] Optionally, in this embodiment, the vector dimension of each first dot product value and the query vector can be calculated but is not limited to: The quotient between the square roots of , to obtain multiple first intermediate values ​​C (x), C (x) = , where x belongs to [1, t1].

[0073] As an optional solution, generating the first fusion factor according to each first intermediate value may be achieved in the following manner, but is not limited to:

[0074] The first fusion factor is generated by the following formula:

[0075] ,

[0076] Wherein, LSE1 is the first fusion factor, t1 is the number of first vectors, C(x) is the first intermediate value corresponding to the x-th first vector, and M1 is the largest first intermediate value.

[0077] Optionally, in this embodiment, It can be but not limited to traversing each first vector and calculating each first vector , and finally calculate the first vectors The harmony.

[0078] As an optional solution, obtaining the second sub-vector generated by the processor of the inference device can be achieved, but is not limited to, in the following manner: obtaining the second sub-vector generated by the processor of the inference device, and obtaining a second fusion factor of the second sub-vector generated by the processor of the inference device, wherein the second fusion factor is used to indicate the fusion ratio of the second sub-vector, and the second fusion factor is used to fuse the first sub-vector and the second sub-vector to generate a context vector.

[0079] Optionally, in this embodiment, it is possible but not limited to obtaining the second fusion factor of the second sub-vector generated by the processor of the inference device at the same time as obtaining the second sub-vector generated by the processor of the inference device. Compared with the solution of directly using the default fusion factor, using the generated second fusion factor is more flexible, and the second sub-vector can be better fused in combination with the distribution of the second historical vector, thereby reducing the model operation accuracy deviation caused by generating the context vector by using the vector fusion method.

[0080] Optionally, in this embodiment, the processor of the inference device may, but is not limited to, generate a second sub-vector according to the second vector information and generate a second fusion factor of the second sub-vector according to the second vector information.

[0081] As an optional solution, obtaining the second sub-vector generated by the processor of the inference device and obtaining the second fusion factor of the second sub-vector generated by the processor of the inference device can be achieved but is not limited to the following methods: sending a query vector of the target vector to the processor; receiving the second sub-vector and the second fusion factor sent by the processor in response to the query vector, wherein the processor is used to filter out a reference historical vector from multiple second historical vectors based on the query vector and the second vector information, generate a second sub-vector based on the third vector information corresponding to the reference historical vector and the query vector, and generate a second fusion factor based on the third vector information corresponding to the reference historical vector and the query vector, wherein the cumulative importance of the reference historical vector is greater than the importance threshold, and the cumulative importance is the cumulative value of the correlation between each second historical vector and each second historical vector itself and the correlation between each second historical vector and the input vector received after each second historical vector, and the second vector information includes the third vector information.

[0082] Optionally, in this embodiment, while the first fusion factor and the first sub-vector are calculated on the accelerator card side, the processor side may, but is not limited to, have an asynchronous sparsification thread that starts to evaluate the vector information (such as the KV cache) on the processor in parallel with the accelerator card inference, and adds part of the vector information to the context set S (i.e., determines the reference historical vector), and then generates the second sub-vector and the second fusion factor based on the third vector information of the reference historical vector.

[0083] Optionally, in this embodiment, the query vector of the target vector can be sent to the processor but is not limited to it, and the computing power of the processor is used to generate the second sub-vector and the second fusion factor according to the second vector information, thereby reducing the computing power burden of the accelerator card.

[0084] As an optional solution, the second fusion factor can be generated based on the third vector information corresponding to the reference historical vector and the query vector in the following manner, but is not limited to: calculating the dot product of the query vector and the key vector of each reference historical vector to obtain multiple second dot product values ​​corresponding one-to-one to each reference historical vector, wherein the third vector information includes the key vector of the reference historical vector; calculating the quotient between each second dot product value and the square root of the vector dimension of the query vector to obtain multiple second intermediate values; and generating the second fusion factor based on each second intermediate value.

[0085] Optionally, in this embodiment, the target vector may be, but is not limited to, in the form of a column vector, and the query vector for calculating the target vector may be, but is not limited to, Key vectors with each reference history vector The dot product of: , where y belongs to [1, t2], and t2 is the number of reference history vectors.

[0086] Optionally, in this embodiment, the vector dimension of each second dot product value and the query vector can be calculated but is not limited to: The quotient between the square roots of , to obtain multiple second intermediate values ​​C (y), C (y) = , where y belongs to [1, t2].

[0087] As an optional solution, the second fusion factor may be generated according to each second intermediate value in the following manner, but is not limited to: the second fusion factor is generated by the following formula:

[0088] ,

[0089] Among them, LSE2 is the second fusion factor, t2 is the number of reference history vectors, C(y) is the second intermediate value corresponding to the y-th reference history vector, and M2 is the largest second intermediate value.

[0090] Optionally, in this embodiment, It can be but not limited to traversing each reference history vector and calculating each reference history vector , and finally calculate the reference history vectors The harmony.

[0091] As an optional solution, it is possible but not limited to the following method to filter out a reference historical vector from multiple second historical vectors based on the query vector and the second vector information: calculate the correlation between each second historical vector and the target vector based on the query vector and the key vector of each second historical vector, wherein the second vector information includes the key vector of the second historical vector; update the cumulative importance of each second historical vector based on the correlation between each second historical vector and the target vector; filter out the second historical vector whose cumulative importance is greater than the importance threshold from multiple second historical vectors to obtain a reference historical vector.

[0092] Optionally, in this embodiment, generally speaking, the influence of the second historical vector on the target vector is lower than that of the first historical vector on the target vector, and it is not very effective to use all the second historical vectors stored in the second storage space in the process of generating the context vector of the target vector. It is possible, but not limited to, to select a portion of more important reference historical vectors from the second historical vectors, and use the third vector information of the selected reference historical vectors as the basis for generating the second sub-vector. By selecting reference historical vectors, the computing power of the processor can be more efficiently utilized, avoiding unnecessary calculations that waste the computing power of the processor and the inference time of the model.

[0093] Optionally, in this embodiment, a cumulative weighted average attention weight (i.e., cumulative importance) may be maintained for each KV cache (or for each second history vector), but is not limited to: Value, which can be defined as, but not limited to:

[0094] ;

[0095] in, , t3 is the number of the second history vector, max(c) can be but is not limited to refer to hour, The maximum value of That is, the query vector, That is, the key vector of each second history vector, That is, the vector dimension of the query vector.

[0096] That is, for the jth history token (i.e., the second history vector) of the i-th attention head of the n-th layer structure, this is a weighted average of the historical average attention weight (i.e., the cumulative importance of the history) and the current newly calculated attention weight (i.e., the degree of relevance), where , It can be used, but not limited to, to accumulate the weighted average attention weight (i.e., the cumulative importance) to the historical and current attention weights. The larger the value, the more importance is attached to the current token, and the faster the importance of historical tokens decays. Compared with simple average, weighted average introduces time sequence, which can be adjusted by This makes it focus more on the recently generated KV cache (or second historical vector), which is more in line with the LLM model's autoregressive generation characteristics with a sequential order.

[0097] Optionally, in this embodiment, the accelerator card may asynchronously After the query vector is transferred from the GPU to the CPU (i.e., the processor), the processor Perform attention calculation on the KV cache (i.e., the key vector of the second history vector) of historical token j (i.e., the second history vector) and update its cumulative weighted average attention weight The processor can, but is not limited to, normalize the accumulated weighted average attention weight of the KV cache corresponding to the attention head i and then construct the context set S (i.e., select the reference history vector). The normalization here is to facilitate the subsequent importance evaluation. ( The second historical vector (i.e., the importance threshold) can be, but is not limited to, added to S (i.e., determined as a reference historical vector). It can, but is not limited to, represent the number of KV caches in the processor under the attention head (i.e., the number of second history vectors). The meaning of is equivalent to an average importance. This formula means that the importance of the KV cache is higher than the average importance. times. The threshold parameter Can be used, but not limited to, to control the degree of sparsification. The larger the value, the stricter the filtering, and the fewer KV caches are retained (that is, the fewer reference history vectors are ultimately determined), which can reduce CPU computing overhead but may sacrifice accuracy. The smaller the size, the more number is retained, the more emphasis is placed on accuracy but the greater the cost. This mechanism can be, but is not limited to, dynamically run on each attention head and adapt its attention distribution: the head with concentrated distribution retains less KV cache (i.e., determines the larger , to eventually get fewer reference history vectors), the head with flat distribution retains more KV cache (i.e., determines smaller , to eventually get more reference history vectors). It can be set to 1, but is not limited to it. It is a relatively balanced setting. When the value is less than 1, more KV cache is reserved with the priority of accuracy. When set to greater than 1, efficiency is prioritized to reduce CPU computation. For all history tokens (i.e., all second history vectors) of the i-th attention head, the cumulative weighted average attention weight can be calculated and then a decision is made whether to add S. For all attention heads in all layers, the above steps can be performed to complete the processing of all attention heads.

[0098] As an optional solution, the second sub-vector can be generated based on the third vector information corresponding to the reference historical vector and the query vector in the following manner, but is not limited to: calculating the degree of correlation between each reference historical vector and the target vector based on the query vector and the key vector of each reference historical vector, wherein the third vector information includes the key vector of the reference historical vector and the value vector of the reference historical vector; calculating the degree of correlation between each reference historical vector and the target vector, and adding the product of the product with the value vector of the corresponding reference historical vector to obtain the second sub-vector.

[0099] Optionally, in this embodiment, the second sub-vector can be calculated by, but is not limited to, the following formula: (i.e. O2):

[0100] ;

[0101] in, That is, the degree of correlation between each reference history vector and the target vector, This is the value vector of each reference history vector. The second subvector can be obtained by, but is not limited to, traversing each reference history vector, calculating the correlation between each reference history vector and the target vector, then multiplying each correlation by the value vector of the corresponding reference history vector, and finally summing the products to obtain the second subvector.

[0102] Optionally, in this embodiment, the correlation between each reference history vector and the target vector can be calculated based on the query vector and the key vector of each reference history vector using the following formula, but is not limited to: : , where t2 is the number of reference history vectors, and max(c) can be but is not limited to refer to hour, The maximum value of That is, the query vector, That is, the key vector of each reference history vector, That is, the vector dimension of the query vector.

[0103] As an optional solution, the context vector of the target vector can be generated based on the first subvector and the second subvector in the following manner, but is not limited to: obtaining a first fusion factor of the first subvector and a second fusion factor of the second subvector, wherein the first fusion factor is used to indicate the fusion ratio of the first subvector, and the second fusion factor is used to indicate the fusion ratio of the second subvector; generating the context vector by the following formula:

[0104] ,

[0105] Among them, O is the context vector, O1 is the first sub-vector, O2 is the second sub-vector, LSE1 is the first fusion factor, and LSE2 is the second fusion factor.

[0106] Optionally, in this embodiment, the attention calculation (i.e., the generation of the context vector) may include, but is not limited to, two parts: a complete attention calculation is performed on the GPU (i.e., the accelerator card) using the most recent token and a few distant key tokens; and a sparse attention calculation is performed on the CPU (i.e., the processor), i.e., no longer using the entire KV cache, but instead using the re-evaluated token S set (i.e., the reference history vector) retrieved from the unloaded KV cache (i.e., the second history vector). This reduces the computational load imbalance between the GPU and the CPU, allowing the GPU to perform small-scale, high-density computations while the CPU processes a lightweight, sparse subset, preventing either from becoming a bottleneck. Furthermore, the GPU and CPU can be run concurrently through efficient collaboration and parallelism, utilizing multi-core CPUs to parallelize sparse attention computations, employing efficient data structures, and reducing latency through communication optimizations such as zero copy.

[0107] Optionally, for the i-th attention head in layer n, use First, calculate on the GPU (accelerator card): , t1 represents the number of KV caches retained by the attention head on the GPU is t1. While calculating the attention on the GPU side, It is passed to the CPU (i.e., processor) side and the CPU-side sparse attention calculation is performed in parallel using the context set S (i.e., reference history vector): , t2 represents the number of KV caches retained by the attention head in the set S is t2. Fusion and It should be noted that the above As can be seen from the calculation formula, when calculating the attention weight (i.e., calculating the degree of relevance), both the CPU side and the GPU side will use the KV cache they have saved. The maximum value of the KV cache and the attention weight of all historical tokens saved by itself is 1, that is, in the calculation and When , only the CPU or GPU part is numerically stabilized and normalized, so the following strategy is needed to perform numerical stability and normalization on the fusion operation: It is known that, , can be but not limited to record , where K is t1 or t2. It can be obtained by recording on the CPU but is not limited to Maximum for , obtained on the GPU Maximum for , then the overall maximum value on CPU and GPU .So , , in order to obtain the required attention head output , can be but not limited to need to use certain strategies 、 Fusion transformation , the following strategies can be used for integration, but are not limited to:

[0108] ;

[0109] From the above analysis, it can be seen that 、 、 、 、 and Can restore , we can further get .

[0110] Further, it is possible but not limited to further optimize on this basis, and it is possible but not limited to record 、 ,

[0111] because 、 ,So,

[0112] ,

[0113] So,

[0114] ,

[0115] and ,

[0116] and ,

[0117] So,

[0118] ,

[0119] So the fusion formula is:

[0120] (Right now );

[0121] That is, it can be but not limited to only recording 、 , you can restore .

[0122] Since the final fusion operation is performed on the GPU, it is possible but not limited to only (i.e., LSE2, the second fusion factor) and (i.e. O2, the second sub-vector) are passed to the GPU, which calculates (i.e. O, context vector).

[0123] Optional, When the data is transferred to the GPU, it can be but not limited to using zero-copy memory access, so that the GPU can directly read the CPU data, bypassing the CPU to GPU memory copy step, thereby improving the access speed and speeding up the generation of the context vector. It can be but not limited to following the formula (Right now ) performs a fusion operation. This fusion can occur at the end of the attention layer, ensuring that the next layer of the feedforward network can directly use the output. The fusion operation of the first and second sub-vectors can be performed on all attention heads in all layers, completing all attention fusion calculations.

[0124] Through the above fusion strategy, the separately calculated attention outputs can be fused to obtain results equivalent to unified calculations while maintaining numerical stability.

[0125] As an optional solution, after receiving the target vector, the vector information stored in the first storage space and the vector information stored in the second storage space can be adjusted in the following manner, but is not limited to: storing the target vector information of the target vector in the first storage space; adjusting the vector information stored in the first storage space to obtain a first storage space storing the first vector information, and the vector information stored in the second storage space to obtain a second storage space storing the second vector information according to the usage parameter, vector parameter and time parameter of the first storage space, wherein the usage parameter is used to indicate the occupancy degree of the first storage space, the vector parameter is used to indicate the degree of correlation between the vector corresponding to the vector information currently stored in the first storage space and the target vector, and the time parameter is used to indicate the storage time of the vector information currently stored in the first storage space.

[0126] Optionally, in this embodiment, a cache pool can be initialized when the model is initialized, taking the LLM model as an example, but not limited to. A continuous cache block is allocated in the GPU for the LLM model with an N-layer network structure, and the cache block is evenly divided into N blocks to store the KV cache of each layer (i.e., the first storage space corresponding to the multiple attention heads in each layer), and an idle ratio T is set for the buffer block of each layer. When the stored KV cache is greater than the ratio T, the unloading operation is performed. Such a hierarchical division not only fits the model structure, but also allows the cache of each layer to be managed independently - when unloading is performed, each layer is unloaded separately, so that the unloading of the KV cache of a certain layer will not affect other layers.

[0127] Optionally, in this embodiment, when performing inference, for the input token t (corresponding to the target vector), the KV cache is calculated on the nth layer to generate a cache (i.e., target vector information):

[0128] ;

[0129] ;

[0130] ;

[0131] Each subvector here corresponds to the i-th attention head.

[0132] Optionally, in this embodiment, after receiving the target vector, the target vector information of the target vector can be stored in the first storage space, but is not limited to. After storing the new vector information in the first storage space, the vector information stored in the first storage space is adjusted according to the usage parameters, vector parameters and time parameters of the first storage space, and the vector information stored in the second storage space is adjusted.

[0133] Optionally, in this embodiment, the vector information stored in the first storage space and the vector information stored in the second storage space may be adjusted to correspond to each other, but is not limited to, for example, including deleting part of the vector information stored in the first storage space from the first storage space and storing it in the second storage space.

[0134] Optionally, in this embodiment, the vector information stored in the first storage space and the vector information stored in the second storage space may be adjusted to be independent of each other, but not limited to, including deleting part of the vector information stored in the first storage space from the first storage space, and extracting zero, one or more vector information from the part of the vector information deleted from the first storage space and storing them in the second storage space. The extraction criteria may be, but not limited to, the degree of correlation between the corresponding historical vector and the target vector or the storage time of the vector information, etc.

[0135] Optionally, in this embodiment, the target vector information may include, but is not limited to, a query vector, a key vector, and a value vector of the target vector.

[0136] Optionally, in this embodiment, the usage parameter can be, but is not limited to, used to indicate the occupancy level of the first storage space, and can be, but is not limited to, setting a usage parameter threshold for the first storage space. When the usage parameter is greater than or equal to the usage parameter threshold, the vector information stored in the first storage space is adjusted and the vector information stored in the second storage space is adjusted.

[0137] Optionally, in this embodiment, the vector parameter may be, but is not limited to, used to indicate the degree of correlation between the vector corresponding to the vector information currently stored in the first storage space and the target vector, and the time parameter may be, but is not limited to, used to indicate the storage time of the vector information currently stored in the first storage space. The higher the degree of correlation between the vector and the target vector, the more important its vector information is; the later the vector information is stored, the more important it is. It is possible, but not limited to, to filter out relatively less important vector information from the vector information currently stored in the first storage space based on the vector parameter and the time parameter and unload it to the second storage space of the processor. The importance of the vector information filtered out from the first storage space may be, but is not limited to, lower than the importance of the vector information still retained in the first storage space after filtering.

[0138] Through the above steps, all relevant information of historical vectors is no longer saved on the accelerator card. Instead, the vector information stored in the first storage space and the vector information stored in the second storage space are adjusted according to the usage parameters, vector parameters and time parameters of the first storage space, effectively reducing the storage space occupied on the accelerator card.

[0139] As an optional solution, it is possible but not limited to the following method to adjust the vector information stored in the first storage space to obtain a first storage space storing the first vector information, and the vector information stored in the second storage space to obtain a second storage space storing the second vector information according to the usage parameters, vector parameters and time parameters of the first storage space: compare the usage parameters and the usage parameter threshold; when the usage parameters are greater than or equal to the usage parameter threshold, transfer the vector information of some vectors in the multiple second vectors to the second storage space according to the vector parameters and time parameters, wherein the second vector is the vector in which the current vector information is stored in the first storage space.

[0140] Optionally, in this embodiment, when the usage parameter is less than the usage parameter threshold, the vector information stored in the first storage space and the vector information stored in the second storage space can be maintained, but is not limited to being maintained, that is, the vector information stored in the first storage space is not adjusted and the vector information stored in the second storage space is not adjusted.

[0141] Optionally, in this embodiment, it is possible but not limited to comparing the usage parameters and the usage parameter threshold to determine whether there is insufficient usage space in the first storage space after the new target vector information is currently stored, and then determine whether the vector information stored in the first storage space needs to be adjusted.

[0142] Optionally, in this embodiment, the second vector may be, but is not limited to, a vector whose current vector information is stored in the first storage space, and the second vector may include, but is not limited to, a target vector.

[0143] Optionally, in this embodiment, it is possible but not limited to that when the usage parameter is greater than or equal to the usage parameter threshold, that is, when there is a risk of insufficient available space in the first storage space, some vectors are filtered out from the multiple second vectors based on the vector parameter and the time parameter, and the vector information of these some vectors is transferred to the second storage space (that is, stored in the second storage space and deleted from the first storage space).

[0144] Through the above content, when there is a risk of insufficient available space in the first storage space, part of the vector information stored in the first storage space is transferred to the second storage space. Compared with the solution of directly deleting part of the vector information from the first storage space, it can ensure that the historical vector information will not be directly lost and can still be used when needed.

[0145] As an optional solution, the vector information of some vectors in multiple second vectors can be transferred to the second storage space according to the vector parameters and time parameters, but is not limited to the following methods: the correlation between each second vector and the target vector is calculated according to the query vector of the target vector and the key vector of each second vector to obtain the vector parameter, wherein the vector information of the second vector includes the key vector of the second vector, and the target vector information includes the query vector of the target vector; the cumulative importance of each second vector is updated according to the vector parameter, wherein the cumulative importance is the cumulative value of the correlation between each second vector and each second vector itself and the correlation between each second vector and the input vector received after each second vector; according to each second vector The influence parameter of each second vector is calculated based on the cumulative importance and time parameter, wherein the influence parameter is used to indicate the influence of each second vector on the target vector; according to the influence parameter of each second vector, a vector to be transferred is screened out from multiple second vectors, wherein the multiple second vectors include a vector to be transferred and a non-transferred vector, the influence parameter of the vector to be transferred is less than or equal to the influence parameter of the non-transferred vector, and the occupation degree of the first storage space by the vector information of the non-transferred vector is less than the occupation degree of the first storage space indicated by the parameter threshold; after generating a first sub-vector of the target vector according to the first vector information and obtaining the second sub-vector generated by the processor of the inference device, the vector information of the vector to be transferred is transferred to the second storage space.

[0146] Optionally, in this embodiment, the query vector of the target vector can be obtained by, but is not limited to, the following formula: and the key vectors of each second vector Calculate the correlation between each second vector and the target vector to obtain the vector parameters:

[0147] ;

[0148] Wherein, t can be but not limited to the number of the second vector, max(c) can be but not limited to the case where x belongs to [1, t], c x The maximum value of . This formula can be used, but is not limited to, to represent the following: for each historical position j, the query is calculated. and key The dot product of the t-th token is the attention calculation of the j-th token, and the dot product is divided by the scaling factor To avoid gradient problems caused by excessive values. Because exponential operations may cause softmax to overflow, you can, but are not limited to, set each dot product to subtract the maximum value of the dot product in all historical positions. . Use softmax to normalize the results of all historical positions and get the attention weight for each position. Then Perform attention calculation for all historical tokens (i.e., the second vector) to obtain the attention weight:

[0149] ;

[0150] As we know above, normalization can make all historical tokens from 1 to t The sum equals 1.

[0151] Optionally, in this embodiment, since the step of calculating the degree of correlation between each second vector and the target vector based on the query vector of the target vector and the key vectors of each second vector is also used in calculating the first sub-vector, it can be, but is not limited to, storing the target vector information of the target vector in the first storage space after receiving the target vector, and then calculating the degree of correlation between each second vector and the target vector based on the query vector of the target vector and the key vectors of each second vector. On the one hand, the original thread calculates the first sub-vector based on the degree of correlation, and on the other hand, starts an asynchronous thread to perform the following operations: comparing the usage parameter and the usage parameter threshold, and when the usage parameter is greater than or equal to the usage parameter threshold, updating the cumulative importance of each second vector based on the vector parameter, calculating the influence parameter of each second vector based on the cumulative importance of each second vector and the time parameter, and filtering the vector to be transferred from multiple second vectors based on the influence parameter of each second vector. After the original thread generates the first sub-vector of the target vector based on the first vector information and obtains the second sub-vector generated by the processor of the inference device, the asynchronous thread transfers the vector information to be transferred to the second storage space.

[0152] Optionally, in this embodiment, an asynchronous thread may be started but is not limited to execute the following operations: maintaining a cumulative weighted average attention weight (i.e., cumulative importance) for each KV cache. The value can be, but is not limited to, updating the cumulative importance of each second vector according to the vector parameter by the following formula: :

[0153] ;

[0154] That is, for the jth historical token (i.e., the second vector) of the i-th attention head in the n-th layer, this is a weighted average of the historical average attention weight (i.e., the cumulative importance of the history) and the current newly calculated attention weight (i.e., the vector parameter), where , It can be used to adjust the cumulative importance of historical and current attention weights. The larger the value, the more importance the current token places on the cumulative importance, while the importance of historical tokens decays more rapidly. Compared to simple averaging, weighted averaging introduces time ordering, which can be adjusted to focus more on recently generated KV caches (i.e., vector information), thus better matching the sequential nature of autoregressive generation in the LLM model.

[0155] Optionally, in this embodiment, the offloading measurement index (i.e., influence parameter) can be calculated for the j-th historical token of the i-th attention head of the n-th layer, but is not limited to:

[0156] ;

[0157] in, represents the KV cache generation time (i.e., time parameter) of the j-th historical token of the i-th attention head in the n-th layer, It can be, but is not limited to, an integer greater than 1. Keep the scale consistent, you can but not limited to scale the generation time (i.e. time parameter) so that it is in Then, the ratio of the cumulative weighted average attention weight (i.e., cumulative importance) and the generation time is used as the Score value (i.e., influence parameter). This formula comprehensively considers the newness and cumulative attention weight of the token. When the cumulative weighted average attention weights are the same, the KV cache with a long generation time (i.e., a large time parameter) will be unloaded first. When the generation time is the same, the KV cache with a small cumulative weighted average attention weight will be unloaded first.

[0158] Optionally, in this embodiment, the Score value of the KV cache for all attention heads of the nth layer can be calculated in the above manner, but is not limited to. Then, the Score values ​​of all KV caches are sorted from large to small, and the excess KV cache is selected and unloaded to the CPU according to the preset cache block free ratio T. When unloading, the cumulative weighted average attention weight (i.e., the cumulative importance) of the corresponding KV cache is also unloaded to the CPU. The cumulative importance on the GPU side is unloaded to the CPU side, which can, but is not limited to, become the initial value of the cumulative importance on the CPU side. For all structural layers, the above steps can be performed, but are not limited to, to complete the KV cache unloading of the attention head.

[0159] Through the above content, the vector information of the second vector with a small influence parameter is transferred from the first storage space to the second storage space, and the vector information of the second vector with a large influence parameter is retained in the first storage space. While reducing the storage space occupied by the accelerator card by historical vector information, important vector information is retained on the accelerator card, which facilitates the use of the accelerator card and ensures the inference efficiency of the model.

[0160] As an optional solution, the influence parameter of each second vector can be calculated based on the accumulated importance and time parameter of each second vector in the following manner, but is not limited to: the influence parameter is calculated using the following formula:

[0161] ,

[0162] Among them, Scorej is the influence parameter of the j-th second vector, Aj is the cumulative importance of the j-th second vector, Bj is the storage time of the vector information of the j-th second vector, t is the number of second vectors, B0 is the longest storage time among the storage times of the vector information of t second vectors, and the time parameters include Bj and B0.

[0163] Optionally, in this embodiment, , that is, the aforementioned ; Aj is , Bj , B0 is .

[0164] Through the above content, the influence parameters of each second vector are determined in combination with the time parameter and the cumulative importance, which fully considers the influence of multiple aspects. The determined influence parameters can better reflect the influence of each second vector on the target vector.

[0165] As an optional solution, the vectors to be transferred can be selected from multiple second vectors based on the influence parameters of each second vector in the following manner, but is not limited to: arranging the second vectors in descending order of the influence parameters to obtain a vector sequence; and selecting the vectors to be transferred from the vector sequence based on the difference between the usage parameter threshold and the usage parameter.

[0166] As an optional embodiment, the context vector generation method described in the present application can be implemented but is not limited to the following process: the accelerator card receives the target vector, generates target vector information of the target vector, stores the target vector information in the first storage space, and asynchronously sends the query vector of the target vector to the processor, calculates the correlation between the target vector and the second vector corresponding to the vector information stored in the first storage space based on the target vector information and the vector information stored in the first storage space, the main thread continues to generate the first sub-vector and the first fusion factor based on the calculated correlation, starts an asynchronous thread to calculate the influence parameters corresponding to each second vector, and selects the vector to be transferred from the multiple second vectors based on the influence parameters, waits for the main thread to generate the first sub-vector and the first fusion factor and obtains the second sub-vector and the second fusion factor, and then transfers the vector information and the accumulated importance of the vector to be transferred to the second storage space of the processor. After receiving the query vector, the processor selects a reference historical vector from the historical vectors corresponding to the vector information stored in the second storage space based on the query vector and the vector information stored in the second storage space. Based on the reference historical vector and the query vector, the processor generates a second fusion factor and a second sub-vector. The processor then provides the second fusion factor and the second sub-vector to the accelerator card, and then receives the vector information and accumulated importance of the vector to be transferred from the accelerator card. It should be noted that the generation of the first sub-vector and the first fusion factor on the accelerator card and the generation of the second sub-vector and the second fusion factor on the processor are performed in parallel.

[0167] Optionally, in this embodiment, in order to better understand the generation process of the above-mentioned context vector, the generation process of the above-mentioned context vector is described below in combination with an optional embodiment, but it is not used to limit the technical solution of the embodiment of this application.

[0168] This embodiment also provides a method for uninstalling a key-value cache. Figure 4 This is a schematic diagram of a process of uninstalling a key-value cache according to an embodiment of the present application, such as Figure 4 As shown, a method for uninstalling a key-value cache may include, but is not limited to, the following steps:

[0169] Step S1: The LLM reasoning system first loads and initializes the model;

[0170] Step S2: Then initialize a cache block and divide the buffer block into N blocks evenly (corresponding to N GPUs, obtaining N first storage spaces);

[0171] Step S3: Start inference. For the input token t, on the nth layer, calculate the KV cache, generate the q vector (i.e., query vector), k vector (i.e., key vector), and v vector (i.e., value vector), and asynchronously transfer the q vector to the CPU.

[0172] Step S4: On the GPU side, for the input token t, in each attention head of each layer, the attention weight is calculated based on the q vector and the k vector of the historical token;

[0173] Step S5: Calculate the cumulative weighted average attention weight (i.e., cumulative importance) for each historical token;

[0174] Step S6: Calculate the uninstallation metric Socre value (i.e., influence parameter) for each historical token;

[0175] Step S7: For all attention heads in the nth layer, sort the score values ​​of the KV cache from large to small, and select the excess KV cache to be unloaded to the CPU according to the preset cache block free ratio T.

[0176] Step S8: Process all layers according to the rules of steps S3 to S7;

[0177] Step S9: The asynchronous thread running on the CPU receives the q from the GPU and starts to evaluate the KV cache, updates its cumulative weighted average attention weight (i.e., cumulative importance), and adds part of the KV cache to the context set S (i.e., determines the reference history vector) based on the size of the cumulative weighted average attention weight.

[0178] Step S10: When calculating attention, in parallel, the GPU uses the nearest token and a few distant key tokens to perform full attention calculation, and the CPU uses the S set to perform sparse attention calculation;

[0179] Step S11: Finally, the CPU results LSEcpu (i.e., the second fusion factor) and ocpu (i.e., the second sub-vector) are transferred to the GPU, where the attention layer outputs are fused. The fused results are used to calculate the FFN of the subsequent layers.

[0180] Figure 4 Also shown are the KV cache changes in the second storage space of the central processing unit CPU and the KV cache changes in the first storage space of the accelerator card GPU.

[0181] In summary, this application's method uses a KV cache offload strategy to support longer context tasks, even with limited GPU memory, thereby supporting a wider range of model applications. A GPU-CPU parallel attention computation strategy effectively utilizes the CPU's computing power and large memory, avoiding the waste of CPU computing power and storage resources during traditional LLM model inference.

[0182] Because the importance of a small number of tokens changes dynamically during different decoding steps, tokens that are initially evicted may become important again during subsequent decoding steps. This application's KV offloading strategy does not delete the KV cache, but instead retains it on the CPU for re-evaluation and participation in attention calculations. This prevents degradation of model accuracy during inference.

[0183] Moreover, through the method of the present application, during parallel attention calculation, only the CPU output vector ocpu and a scalar LSEcpu need to be transmitted. Compared with the PCIe (Peripheral Component Interconnect Express) bandwidth pressure caused by transmitting the KV cache from the CPU to the GPU, the method of the present application greatly reduces the transmission amount. Therefore, the transmission will not cause a performance bottleneck, thereby keeping pace with the attention calculation on the GPU.

[0184] Furthermore, through the method of the present application, an equivalent fusion strategy is adopted, which can fuse the attention outputs calculated separately on the CPU and GPU to achieve a result equivalent to that of a unified calculation, thereby avoiding the introduction of additional errors.

[0185] It is worth noting that in the method of this application, KV unloading and CPU-side KV cache evaluation are asynchronous processes, so they will not affect the calculation of the main process, and when calculating attention, CPU-GPU parallelism is used to avoid increased delays in the inference process.

[0186] Finally, in the calculation of the cumulative weighted average attention, the method of this application introduces a time sequence compared to the simple average, which can be adjusted by The cumulative importance / cumulative importance is made more focused on the recently generated KV, so as to better fit the LLM model's autoregressive generation characteristics with a sequential order.

[0187] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0188] Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, or optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of this application.

[0189] In this embodiment, a device for generating a context vector is also provided, which is applied to an accelerator card of an inference device. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the details that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0190] Figure 5 is a structural block diagram of a device for generating a context vector according to an embodiment of the present application; Figure 5 Shown, including:

[0191] A receiving module 502 is configured to receive a target vector;

[0192] A first generation module 504 is configured to generate a first subvector of a target vector based on the first vector information, and obtain a second subvector generated by a processor of the inference device, wherein the first vector information is stored in a first storage space on the accelerator card, the first vector information is used to indicate a first historical vector received historically, the first subvector is used to indicate a relationship between the target vector and the first historical vector, the second subvector is used to indicate a relationship between the target vector and the second historical vector, the second vector information is stored in a second storage space on the processor, the second vector information is used to indicate a second historical vector received historically, and the influence of the first historical vector on the target vector is greater than the influence of the second historical vector on the target vector;

[0193] The second generating module 506 is configured to generate a context vector of the target vector according to the first sub-vector and the second sub-vector.

[0194] Through the above device, the accelerator card of the inference device receives the target vector, generates a first subvector of the target vector based on the first vector information stored in the first storage space of the accelerator card, and obtains the second subvector generated by the processor of the inference device, the first subvector is used to indicate the relationship between the target vector and the first history vector, and the second subvector is used to indicate the relationship between the target vector and the second history vector stored in the second storage space of the processor. The influence of the first history vector on the target vector is higher than the influence of the second history vector on the target vector. The historical vector information of the history vector no longer needs to be stored only on the accelerator card, which reduces the storage space occupied by the accelerator card. The accelerator card will eventually generate a context vector of the target vector based on the first subvector and the second subvector, thereby achieving the goal of maintaining the model's running accuracy as much as possible while reducing the storage space occupied by the accelerator card and avoiding the introduction of additional errors. Therefore, it can solve the technical problem in the related art that the historical information occupies a large amount of storage space of the accelerator card, and achieve the technical effect of reducing the storage space occupied by the historical information on the accelerator card while maintaining the model's running accuracy.

[0195] In an exemplary embodiment, the first generation module includes: a first generation unit, used to generate a first sub-vector based on the first vector information, and generate a first fusion factor of the first sub-vector based on the first vector information, wherein the first fusion factor is used to indicate the fusion ratio of the first sub-vector, and the first fusion factor is used to fuse the first sub-vector and the second sub-vector to generate a context vector.

[0196] In an exemplary embodiment, the first generation unit is further used to: calculate the dot product of the query vector of the target vector and the key vector of each first vector to obtain multiple first dot product values ​​corresponding one-to-one to each first vector, wherein the first vector includes the first history vector and the target vector, and the first vector information includes the key vector of the first history vector; calculate the quotient between each first dot product value and the square root of the vector dimension of the query vector to obtain multiple first intermediate values; and generate a first fusion factor based on each first intermediate value.

[0197] In an exemplary embodiment, the first generating unit is further configured to generate the first fusion factor by using the following formula:

[0198] ,

[0199] Wherein, LSE1 is the first fusion factor, t1 is the number of first vectors, C(x) is the first intermediate value corresponding to the x-th first vector, and M1 is the largest first intermediate value.

[0200] In an exemplary embodiment, the first generation module further includes: a first acquisition unit, used to acquire a second sub-vector generated by the processor of the inference device, and acquire a second fusion factor of the second sub-vector generated by the processor of the inference device, wherein the second fusion factor is used to indicate the fusion ratio of the second sub-vector, and the second fusion factor is used to fuse the first sub-vector and the second sub-vector to generate a context vector.

[0201] In an exemplary embodiment, the first acquisition unit is further used to: send a query vector of the target vector to a processor; receive a second sub-vector and a second fusion factor sent by the processor in response to the query vector, wherein the processor is used to filter out a reference historical vector from multiple second historical vectors based on the query vector and the second vector information, generate a second sub-vector based on the third vector information corresponding to the reference historical vector and the query vector, and generate a second fusion factor based on the third vector information corresponding to the reference historical vector and the query vector, wherein the cumulative importance of the reference historical vector is greater than the importance threshold, and the cumulative importance is the cumulative value of the correlation between each second historical vector and each second historical vector itself and the correlation between each second historical vector and the input vector received after each second historical vector, and the second vector information includes the third vector information.

[0202] In an exemplary embodiment, the first acquisition unit is further used to: calculate the dot product of the query vector and the key vector of each reference historical vector to obtain multiple second dot product values ​​corresponding one-to-one to each reference historical vector, wherein the third vector information includes the key vector of the reference historical vector; calculate the quotient between each second dot product value and the square root of the vector dimension of the query vector to obtain multiple second intermediate values; and generate a second fusion factor based on each second intermediate value.

[0203] In an exemplary embodiment, the first acquiring unit is further configured to generate the second fusion factor using the following formula:

[0204] ,

[0205] Among them, LSE2 is the second fusion factor, t2 is the number of reference history vectors, C(y) is the second intermediate value corresponding to the y-th reference history vector, and M2 is the largest second intermediate value.

[0206] In an exemplary embodiment, the first acquisition unit is further used to: calculate the degree of correlation between each second historical vector and the target vector based on the query vector and the key vector of each second historical vector, wherein the second vector information includes the key vector of the second historical vector; update the cumulative importance of each second historical vector based on the degree of correlation between each second historical vector and the target vector; and filter out the second historical vector whose cumulative importance is greater than the importance threshold from multiple second historical vectors to obtain a reference historical vector.

[0207] In an exemplary embodiment, the first acquisition unit is further used to: calculate the degree of correlation between each reference historical vector and the target vector based on the query vector and the key vector of each reference historical vector, wherein the third vector information includes the key vector of the reference historical vector and the value vector of the reference historical vector; calculate the degree of correlation between each reference historical vector and the target vector, and add the product of the product and the value vector of the corresponding reference historical vector to obtain the second sub-vector.

[0208] In an exemplary embodiment, the second generation module includes: a second acquisition unit, configured to acquire a first fusion factor of the first sub-vector and a second fusion factor of the second sub-vector, wherein the first fusion factor is used to indicate a fusion weight of the first sub-vector, and the second fusion factor is used to indicate a fusion weight of the second sub-vector; and a second generation unit, configured to generate a context vector using the following formula:

[0209] ,

[0210] Among them, O is the context vector, O1 is the first sub-vector, O2 is the second sub-vector, LSE1 is the first fusion factor, and LSE2 is the second fusion factor.

[0211] In an exemplary embodiment, the context vector generation device also includes: a storage unit for storing the target vector information of the target vector in a first storage space after receiving the target vector; an adjustment unit for adjusting the vector information stored in the first storage space to obtain a first storage space storing the first vector information, and the vector information stored in the second storage space to obtain a second storage space storing the second vector information according to the usage parameter, vector parameter and time parameter of the first storage space, wherein the usage parameter is used to indicate the occupancy degree of the first storage space, the vector parameter is used to indicate the degree of correlation between the vector corresponding to the vector information currently stored in the first storage space and the target vector, and the time parameter is used to indicate the storage time of the vector information currently stored in the first storage space.

[0212] In an exemplary embodiment, the adjustment unit is further used to: compare the usage parameter and the usage parameter threshold; when the usage parameter is greater than or equal to the usage parameter threshold, transfer the vector information of some vectors in the multiple second vectors to the second storage space according to the vector parameter and the time parameter, wherein the second vector is the vector in which the current vector information is stored in the first storage space.

[0213] In an exemplary embodiment, the adjustment unit is further used to: calculate the degree of correlation between each second vector and the target vector based on the query vector of the target vector and the key vector of each second vector to obtain a vector parameter, wherein the vector information of the second vector includes the key vector of the second vector, and the target vector information includes the query vector of the target vector; update the cumulative importance of each second vector based on the vector parameter, wherein the cumulative importance is the cumulative value of the degree of correlation between each second vector and each second vector itself and the degree of correlation between each second vector and the input vector received after each second vector; calculate the impact of each second vector based on the cumulative importance of each second vector and the time parameter Influence parameters, wherein the influence parameters are used to indicate the influence of each second vector on the target vector; according to the influence parameters of each second vector, a vector to be transferred is screened out from a plurality of second vectors, wherein the plurality of second vectors include a vector to be transferred and a vector not to be transferred, the influence parameter of the vector to be transferred is less than or equal to the influence parameter of the vector not to be transferred, and the degree to which the vector information of the vector not to be transferred occupies the first storage space is less than the degree to which the first storage space is occupied by the vector information of the first storage space indicated by the parameter threshold; after generating a first sub-vector of the target vector according to the first vector information and obtaining a second sub-vector generated by the processor of the inference device, the vector information of the vector to be transferred is transferred to the second storage space.

[0214] In an exemplary embodiment, the adjustment unit is further configured to calculate the influence parameter using the following formula:

[0215] ,

[0216] Among them, Scorej is the influence parameter of the j-th second vector, Aj is the cumulative importance of the j-th second vector, Bj is the storage time of the vector information of the j-th second vector, t is the number of second vectors, B0 is the longest storage time among the storage times of the vector information of t second vectors, and the time parameters include Bj and B0.

[0217] In an exemplary embodiment, the adjustment unit is further configured to: arrange the second vectors in descending order of influence parameters to obtain a vector sequence; and filter the vectors to be transferred from the vector sequence according to the difference between the usage parameter threshold and the usage parameter.

[0218] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0219] For the description of the features in the embodiment corresponding to the context vector generation device, please refer to the relevant description of the embodiment corresponding to the context vector generation method, and no further details will be given here.

[0220] An embodiment of the present application further provides an electronic device, Figure 6 is a schematic diagram of an electronic device according to an embodiment of the present application, such as Figure 6 As shown, the electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned embodiments of the method for generating a context vector.

[0221] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0222] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0223] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps of any of the above-mentioned context vector generation method embodiments when running.

[0224] In an exemplary embodiment, the computer-readable storage medium may be, but is not limited to, a non-volatile computer-readable storage medium.

[0225] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0226] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0227] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method in each embodiment of the present application; the computer program product also includes a non-volatile computer-readable storage medium, which stores the computer program, and, when executed by a processor, implements the steps of the method for generating a context vector in each embodiment of the present application.

[0228] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0229] The above is a detailed introduction to a method and device for generating a context vector provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only applicable to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for generating a context vector, characterized in that: An accelerator card applied to an inference device, the method comprising: Receive target vector; Generate a first subvector of the target vector based on first vector information, and obtain a second subvector generated by a processor of the inference device, wherein the first vector information is stored in a first storage space on the accelerator card, the first vector information is used to indicate a historically received first history vector, the first subvector is used to indicate a relationship between the target vector and the first history vector, the second subvector is used to indicate a relationship between the target vector and a second history vector, the second vector information is stored in a second storage space on the processor, the second vector information is used to indicate a historically received second history vector, and the influence of the first history vector on the target vector is greater than the influence of the second history vector on the target vector; A context vector of the target vector is generated according to the first sub-vector and the second sub-vector.

2. The method according to claim 1, characterized in that Generating the first sub-vector of the target vector according to the first vector information includes: The first sub-vector is generated according to the first vector information, and a first fusion factor of the first sub-vector is generated according to the first vector information, wherein the first fusion factor is used to indicate the fusion ratio of the first sub-vector, and the first fusion factor is used to fuse the first sub-vector and the second sub-vector to generate the context vector.

3. The method according to claim 2, characterized in that Generating a first fusion factor of the first sub-vector according to the first vector information includes: Calculating a dot product between a query vector of the target vector and a key vector of each first vector to obtain a plurality of first dot product values ​​corresponding one-to-one to each first vector, wherein the first vector includes the first history vector and the target vector, and the first vector information includes the key vector of the first history vector; Calculating a quotient between each of the first dot product values ​​and a square root of a vector dimension of the query vector to obtain a plurality of first intermediate values; The first fusion factor is generated according to each of the first intermediate values.

4. The method according to claim 3, characterized in that Generating the first fusion factor according to each of the first intermediate values ​​includes: The first fusion factor is generated by the following formula: ， Wherein, LSE1 is the first fusion factor, t1 is the number of the first vectors, C(x) is the first intermediate value corresponding to the x-th first vector, and M1 is the maximum first intermediate value.

5. The method according to claim 1, wherein The obtaining of the second sub-vector generated by the processor of the inference device includes: Obtain the second sub-vector generated by the processor of the inference device, and obtain a second fusion factor of the second sub-vector generated by the processor of the inference device, wherein the second fusion factor is used to indicate a fusion weight of the second sub-vector, and the second fusion factor is used to fuse the first sub-vector and the second sub-vector to generate the context vector.

6. The method according to claim 5, characterized in that The obtaining of the second sub-vector generated by the processor of the inference device and obtaining a second fusion factor of the second sub-vector generated by the processor of the inference device includes: sending a query vector of the target vector to the processor; Receive the second sub-vector and the second fusion factor sent by the processor in response to the query vector, wherein the processor is used to filter out a reference historical vector from multiple second historical vectors based on the query vector and the second vector information, generate the second sub-vector based on the third vector information corresponding to the reference historical vector and the query vector, and generate the second fusion factor based on the third vector information corresponding to the reference historical vector and the query vector, wherein the cumulative importance of the reference historical vector is greater than the importance threshold, the cumulative importance is the cumulative value of the correlation between each second historical vector and each second historical vector itself and the correlation between each second historical vector and the input vector received after each second historical vector, and the second vector information includes the third vector information.

7. The method according to claim 6, characterized in that The generating the second fusion factor according to the third vector information corresponding to the reference historical vector and the query vector includes: calculating a dot product between the query vector and a key vector of each of the reference history vectors to obtain a plurality of second dot product values ​​corresponding one-to-one to each of the reference history vectors, wherein the third vector information includes the key vector of the reference history vector; Calculating a quotient between each of the second dot product values ​​and a square root of the vector dimension of the query vector to obtain a plurality of second intermediate values; The second fusion factor is generated according to each of the second intermediate values.

8. The method according to claim 7, characterized in that Generating the second fusion factor according to each of the second intermediate values ​​includes: The second fusion factor is generated by the following formula: ， Among them, LSE2 is the second fusion factor, t2 is the number of the reference historical vectors, C(y) is the second intermediate value corresponding to the y-th reference historical vector, and M2 is the maximum second intermediate value.

9. The method according to claim 6, characterized in that The step of selecting a reference historical vector from a plurality of second historical vectors according to the query vector and the second vector information includes: calculating the relevance between each of the second history vectors and the target vector based on the query vector and the key vector of each of the second history vectors, wherein the second vector information includes the key vector of the second history vector; updating the accumulated importance of each second history vector according to the correlation between each second history vector and the target vector; The second history vectors whose cumulative importance is greater than the importance threshold are screened out from the plurality of second history vectors to obtain the reference history vector.

10. The method according to claim 6, characterized in that The generating the second sub-vector according to the third vector information corresponding to the reference history vector and the query vector includes: calculating the degree of relevance between each of the reference history vectors and the target vector based on the query vector and a key vector of each reference history vector, wherein the third vector information includes the key vector of the reference history vector and the value vector of the reference history vector; The second sub-vector is obtained by calculating the cumulative sum of the products of the correlation degree between each reference history vector and the target vector and the value vector of the corresponding reference history vector.

11. The method according to claim 1, characterized in that Generating a context vector of the target vector according to the first sub-vector and the second sub-vector includes: Obtaining a first fusion factor of the first sub-vector and a second fusion factor of the second sub-vector, wherein the first fusion factor is used to indicate a fusion weight of the first sub-vector, and the second fusion factor is used to indicate a fusion weight of the second sub-vector; The context vector is generated by the following formula: ， Among them, O is the context vector, O1 is the first sub-vector, O2 is the second sub-vector, LSE1 is the first fusion factor, and LSE2 is the second fusion factor.

12. The method according to claim 1, characterized in that After receiving the target vector, the method further includes: Storing target vector information of the target vector in the first storage space; According to the usage parameters, vector parameters and time parameters of the first storage space, the vector information stored in the first storage space is adjusted to obtain the first storage space storing the first vector information, and the vector information stored in the second storage space is adjusted to obtain the second storage space storing the second vector information, wherein the usage parameters are used to indicate the occupancy degree of the first storage space, the vector parameters are used to indicate the degree of correlation between the vector corresponding to the vector information currently stored in the first storage space and the target vector, and the time parameters are used to indicate the storage time of the vector information currently stored in the first storage space.

13. The method according to claim 12, characterized in that The adjusting, according to the usage parameter, vector parameter, and time parameter of the first storage space, the vector information stored in the first storage space to obtain the first storage space storing the first vector information, and the vector information stored in the second storage space to obtain the second storage space storing the second vector information, includes: comparing the usage parameter to a usage parameter threshold; When the usage parameter is greater than or equal to the usage parameter threshold, vector information of some vectors in multiple second vectors is transferred to the second storage space according to the vector parameter and the time parameter, wherein the second vector is the vector in which the current vector information is stored in the first storage space.

14. The method according to claim 13, wherein: The transferring the vector information of some of the plurality of second vectors to the second storage space according to the vector parameter and the time parameter includes: calculating the degree of correlation between each second vector and the target vector based on the query vector of the target vector and the key vector of each second vector to obtain the vector parameter, wherein the vector information of the second vector includes the key vector of the second vector, and the target vector information includes the query vector of the target vector; updating a cumulative importance of each second vector according to the vector parameter, wherein the cumulative importance is a cumulative value of a correlation between each second vector and itself and a correlation between each second vector and an input vector received after the second vector; Calculating an influence parameter of each second vector according to the accumulated importance of each second vector and the time parameter, wherein the influence parameter is used to indicate the influence of each second vector on the target vector; Filtering a to-be-transferred vector from a plurality of second vectors based on the influence parameter of each second vector, wherein the plurality of second vectors includes the to-be-transferred vector and a non-transferred vector, the influence parameter of the to-be-transferred vector is less than or equal to the influence parameter of the non-transferred vector, and the degree of occupation of the first storage space by vector information of the non-transferred vector is less than the degree of occupation of the first storage space indicated by the usage parameter threshold; After generating the first sub-vector of the target vector according to the first vector information and acquiring the second sub-vector generated by the processor of the inference device, the vector information of the vector to be transferred is transferred to the second storage space.

15. The method according to claim 14, characterized in that The calculating the influence parameter of each second vector according to the accumulated importance of each second vector and the time parameter includes: The influence parameter is calculated by the following formula: ， Among them, Scorej is the influence parameter of the j-th second vector, Aj is the cumulative importance of the j-th second vector, Bj is the storage time of the vector information of the j-th second vector, t is the number of second vectors, B0 is the longest storage time among the storage times of the vector information of t second vectors, and the time parameters include Bj and B0.

16. The method according to claim 14, characterized in that The step of selecting a vector to be transferred from a plurality of second vectors according to the influence parameter of each second vector includes: Arrange the second vectors in descending order of the influence parameters to obtain a vector sequence; The vector to be transferred is selected from the vector sequence according to a difference between the usage parameter threshold and the usage parameter.

17. A device for generating a context vector, characterized in that: An accelerator card for an inference device, the device comprising: A receiving module, configured to receive a target vector; a first generating module, configured to generate a first subvector of the target vector based on first vector information, and obtain a second subvector generated by a processor of the inference device, wherein the first vector information is stored in a first storage space on the accelerator card, the first vector information is used to indicate a historically received first historical vector, the first subvector is used to indicate a relationship between the target vector and the first historical vector, the second subvector is used to indicate a relationship between the target vector and a second historical vector, the second vector information is stored in a second storage space on the processor, the second vector information is used to indicate a historically received second historical vector, and the influence of the first historical vector on the target vector is greater than the influence of the second historical vector on the target vector; A second generating module is configured to generate a context vector of the target vector according to the first sub-vector and the second sub-vector.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 16 are implemented.

19. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 16 are implemented.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

Images