Method and apparatus for generating context vector

Abstract

The application discloses a context vector generation method and device, relates to the technical field of machine learning, and comprises the following steps: receiving a target vector; generating a first subvector of the target vector according to first vector information, and obtaining a second subvector generated by a processor of an inference device, wherein the first vector information is stored in a first storage space on an acceleration card, the first subvector is used for indicating the relationship between the target vector and a first historical vector, the second subvector is used for indicating the relationship between the target vector and a second historical vector, and second vector information is stored in a second storage space on the processor; and generating a context vector of the target vector according to the first subvector and the second subvector. Through the application, the technical problems that historical information occupies more storage space of an acceleration card are solved, and the technical effect that the storage space of the acceleration card occupied by the historical information is reduced while the model operation precision is maintained is achieved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of machine learning, and in particular, to a method and apparatus for generating context vectors. BACKGROUND

[0002] With the continuous development of modern technology, it becomes more and more common to use models to solve problems. When the current model performs reasoning, in order to avoid repeated processing of input tokens, the relevant information of the tokens input in the early stage is generally stored in an acceleration card for use in the subsequent reasoning process. However, as the size of the model becomes larger and larger, the cache overhead of historical token information also becomes larger and larger, becoming a key bottleneck when deploying the model. SUMMARY

[0003] Embodiments of the present application provide a method and apparatus for generating context vectors to at least solve the technical problem that, in related technologies, historical information occupies a large amount of storage space on an acceleration card.

[0004] According to an embodiment of the present application, a method for generating context vectors is provided, applied to an acceleration card of a reasoning device, and the method comprises: receiving a target vector; generating a first sub-vector of the target vector according to first vector information, and obtaining a second sub-vector generated by a processor of the reasoning device, wherein the first vector information is stored in a first storage space on the acceleration card, the first vector information is used to indicate a first historical vector received historically, the first sub-vector is used to indicate a relationship between the target vector and the first historical vector, the second sub-vector is used to indicate a relationship between the target vector and a second historical vector, the second vector information is stored in a second storage space on the processor, the second vector information is used to indicate the second historical vector received historically, and the influence of the first historical vector on the target vector is higher than the influence of the second historical vector on the target vector; and generating a context vector of the target vector according to the first sub-vector and the second sub-vector.

[0005] According to another embodiment of the embodiment of the application, a context vector generation apparatus applied to an acceleration card of an inference device is further provided, and the apparatus comprises: a receiving module configured to receive a target vector; a first generation module configured to generate a first sub-vector of the target vector according to first vector information, and acquire a second sub-vector generated by a processor of the inference device, wherein the first vector information is stored in a first storage space on the acceleration card, the first vector information is used to indicate a first historical vector received historically, the first sub-vector is used to indicate a relationship between the target vector and the first historical vector, the second sub-vector is used to indicate a relationship between the target vector and a second historical vector, the second vector information is stored in a second storage space on the processor, the second vector information is used to indicate the second historical vector received historically, and an influence of the first historical vector on the target vector is higher than an influence of the second historical vector on the target vector; and a second generation module configured to generate a context vector of the target vector according to the first sub-vector and the second sub-vector.

[0006] The application further provides an electronic device, comprising: a memory configured to store a computer program; and a processor configured to execute the computer program to implement the steps of any of the context vector generation methods.

[0007] The application further provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of any of the context vector generation methods.

[0008] The application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of any of the context vector generation methods.

[0009] According to the application, the acceleration card of the inference device receives a target vector, generates a first sub-vector of the target vector according to first vector information stored in a first storage space on the acceleration card, and acquires a second sub-vector generated by a processor of the inference device, the first sub-vector is used to indicate a relationship between the target vector and a first historical vector, the second sub-vector is used to indicate a relationship between the target vector and a second historical vector stored in a second storage space of the processor, the influence of the first historical vector on the target vector is higher than the influence of the second historical vector on the target vector, the historical vector information of the historical vector does not need to be stored only on the acceleration card, the occupation of the storage space of the acceleration card is reduced, and finally the acceleration card generates a context vector of the target vector according to the first sub-vector and the second sub-vector, the running precision of the model is maintained as much as possible under the condition of reducing the occupation of the storage space of the acceleration card, and additional errors are avoided. Therefore, the technical problems such as the high occupation of the storage space of the acceleration card by the historical information in the related art are solved, and the technical effect of reducing the occupation of the storage space of the acceleration card by the historical information while maintaining the running precision of the model is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0011] Figure 1 is a hardware structure block diagram of a computer device of a context vector generation method according to an embodiment of the present application;

[0012] Figure 2 is a flow chart of a context vector generation method according to an embodiment of the present application;

[0013] Figure 3 is a schematic diagram of a context vector generation method according to an embodiment of the present application;

[0014] Figure 4 is a schematic diagram of a flow of a key-value cache offloading method according to an embodiment of the present application;

[0015] Figure 5 is a structure block diagram of a context vector generation device according to an embodiment of the present application;

[0016] Figure 6 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort fall within the protection scope of the present application.

[0018] It should be noted that, in the description of the present application, the terms “include”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0019] In order to make the skilled in the art better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0020] The method embodiments provided in the embodiments of the present application can be executed in a server device or similar computing device. Taking the case of running on a server device, Figure 1 is a hardware structure block diagram of a computer device of a context vector generation method according to an embodiment of the present application. As shown in Figure 1 , the server device can include one or more (only one is shown in Figure 1 ) processor 102 (the processor 102 can include but not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned server device can further include a transmission device 106 for communication function and an input and output device 108. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, which does not limit the structure of the above-mentioned server device. For example, the server device can further include more or less components than those shown in Figure 1 , or have a different configuration from Figure 1 .

[0021] The memory 104 can be used to store computer programs, for example, software programs of application software and modules, such as the computer program corresponding to the context vector generation method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implements the above-mentioned method. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, which can be connected to the server device through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0022] The transmission device 106 is used to receive or send data via a network. The specific examples of the above-mentioned network can include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet in a wireless manner.

[0023] The application provides a context vector generation method. Before describing optional embodiments of the application, in order to better understand the inventive concept of the application and the creativity of the scheme, first, the related art is described.

[0024] The inference process of a model such as an LLM (Large Language Model) is to generate tokens step by step, which is called autoregressive generation. First, the text input by the user is segmented into a series of tokens and embedded into word vectors to be input into the model. The model predicts the next most likely token under the current context through forward calculation. Then, the model samples a token according to the predicted probability distribution and adds it to the existing sequence as new input for inference again. This process is repeated continuously, with only one token generated each time, until a termination symbol is generated or the maximum length is reached. The entire generation process relies on the previous tokens to gradually build coherent text.

[0025] In the model inference process, a KV (Key-Value) caching mechanism can be used to improve efficiency, which helps the model avoid recalculating the token information in advance during runtime, thereby accelerating the inference process. The following briefly describes the implementation of the mechanism:

[0026] In the self-attention mechanism of the model, each input token generates three vectors, namely query (Query), key vector (Key), and value vector (Value). In the standard inference process, all inputs need to be recalculated for these three vectors every time they are generated. When using KV caching, the model caches the calculated key and value vectors, so that when generating the next token, the model can directly use these cached keys and values without recalculating all inputs. This caching mechanism is particularly efficient for long text generation, reducing the amount of calculation.

[0027] The prefilling stage in the model inference process refers to processing the input sequence in the user's request through the model once to generate KV cache for use in the decoding stage. Decoding is based on the context of the prefilling stage, and the model generates new tokens step by step until the target length or end symbol is reached. Decoding uses KV cache to avoid repeated calculation and improve efficiency.

[0028] The current popular model adopts a structure of multiple layers, each layer containing a multi-head attention mechanism and a feed-forward neural network (FFN), and having an embedding layer and an MLP layer (Multi-Layer Perceptron) at the beginning and end of the model. The multi-head attention mechanism is composed of multiple self-attention mechanisms, so the size of the KV cache increases linearly with the number of heads of the multi-head attention and the number of model layers.

[0029] The cost of LLM deployment is extremely high, and usually needs to rely on multiple high-performance GPUs (Graphics Processing Unit) or other hardware accelerators to meet the dual requirements of computing speed and storage space capacity during inference. As the length of the context increases, the storage of the KV cache stored in the GPU quickly becomes a new bottleneck of the system. Therefore, how to effectively reduce the storage space occupation and bandwidth pressure of the KV cache without affecting the model accuracy and inference quality has become one of the core problems to be solved in the current LLM deployment.

[0030] In the present embodiment, a context vector generation method is provided, which is applied to an acceleration card of an inference device. Figure 2 is a flowchart of a context vector generation method according to an embodiment of the present application, as Figure 2 shown, the flow includes the following steps:

[0031] Step S202, receiving a target vector;

[0032] Step S204, generating a first sub-vector of the target vector according to first vector information, and obtaining a second sub-vector generated by a processor of the inference device, wherein the first vector information is stored in a first storage space on the acceleration card, the first vector information is used to indicate a first historical vector received historically, the first sub-vector is used to indicate a relationship between the target vector and the first historical vector, the second sub-vector is used to indicate a relationship between the target vector and a second historical vector, the second vector information is stored in a second storage space on the processor, the second vector information is used to indicate the second historical vector received historically, and the influence of the first historical vector on the target vector is higher than the influence of the second historical vector on the target vector;

[0033] Step S206, generating a context vector of the target vector according to the first sub-vector and the second sub-vector.

[0034] Through the above steps, the acceleration card of the inference device receives a target vector, generates a first sub-vector of the target vector according to first vector information stored in a first storage space on the acceleration card, and acquires a second sub-vector generated by a processor of the inference device, the first sub-vector is used to indicate a relationship between the target vector and a first historical vector, the second sub-vector is used to indicate a relationship between the target vector and a second historical vector stored in a second storage space of the processor, an influence of the first historical vector on the target vector is higher than an influence of the second historical vector on the target vector, and historical vector information of the historical vector is not stored only on the acceleration card, thereby reducing occupation of the storage space of the acceleration card, and the acceleration card finally generates a context vector of the target vector according to the first sub-vector and the second sub-vector, thereby realizing the running accuracy of the model as much as possible in the case of reducing the occupation of the storage space of the acceleration card, and avoiding introducing additional errors. Therefore, the technical problem that historical information occupies a large storage space of an acceleration card in the related art can be solved, and the technical effect that the occupation of the storage space of the acceleration card by historical information is reduced while the running accuracy of the model is maintained can be achieved.

[0035] Optionally, in the present embodiment, the method can be applied to, but is not limited to, an acceleration card of an inference device. The inference device can refer to, but is not limited to, a hardware device used to perform machine learning model inference (or inference) tasks. Such devices are not limited to traditional CPUs (Central Processing Unit), but also include various hardware accelerators specially designed to accelerate deep learning or machine learning operations. For example, the inference device can include, but is not limited to, servers, desktop computers or workstations, embedded devices (such as Internet of Things devices, smart cameras, smart speakers, etc.), mobile devices (such as smartphones and tablets, which are built-in APU (AI Processing Unit) or dedicated AI acceleration unit for real-time processing of AI tasks, such as image recognition or voice assistants), ASIC (Application-Specific Integrated Circuit), etc. Optionally, the acceleration card can be, but is not limited to, a plug-in hardware device that can be used to expand the computing power of a host (such as a server or a workstation). Specifically, the acceleration card can include, but is not limited to, a GPU, an FPGA (Field-Programmable Gate Array), an ASIC, a DSP (Digital Signal Processor), etc. Optionally, in addition to the acceleration card, the inference device can include, but is not limited to, a processor. The processor can refer to, but is not limited to, a CPU, which is the brain of a computer system and is responsible for interpreting and executing software instructions. The processor can be connected to one or more acceleration cards, and the method for generating a context vector described in the present application can be independently run on each acceleration card.

[0036] Optionally, in the present embodiment, a large language model including a multi-layer structure and a multi-head attention mechanism can be deployed in a system including a processor and one or more acceleration cards. In the multi-head attention mechanism of the LLM, the key vectors (hereinafter referred to as keys) and value vectors (hereinafter referred to as values) calculated in this process are cached each time a token is input into the model. In this way, when generating each token, the model can directly use these cached keys and values, without the need to recalculate all inputs. Specifically, in the multi-head attention mechanism, for the input vector of the t-th token , first pass through three linear transformation matrices to obtain the query (Query), key (Key), and value (Value) vectors:

[0037] ;

[0038] ;

[0039] ;

[0040] where, the embedding dimension of the input vector (e.g., the embedding of a token) is denoted as , the number of heads in multi-head attention is denoted as , the dimension of each attention head is denoted as , the input vector of the t-th token in the attention layer is denoted as . , i.e., the overall representation of the multiple heads. In order to parallelize the calculation of multiple attention heads, the above vector can be, but is not limited to, cut into sub-vectors: ; ; . Each of the sub-vectors corresponds to the i-th attention head. The calculation of the output of each attention head is as follows:

[0041] ;

[0042] Because the exponential operation is involved in the softmax function, in actual engineering use, in order to avoid the overflow and underflow problems of softmax, the following formula can be, but is not limited to, used:

[0043] ;

[0044] i.e., subtracting the maximum value in , thereby avoiding the overflow problem of the softmax function. The outputs of all attention heads are spliced together, and then a output mapping matrix is used for linear transformation to obtain the final multi-head attention output (i.e., the generated context vector):

[0045] ;

[0046] The dimension of this vector is the same as the input The same, all d, can continue to input to the subsequent layer. The method for generating the context vector described in the present application can be applied in each of the plurality of attention heads described above, but is not limited thereto. Each attention head does not affect other attention heads or other structural layers when applying the method. That is, each attention head can independently use the method provided in the present application, and each structural layer can independently apply the method provided in the present application according to the layer relationship between the structural layers. Specifically, the plurality of attention heads in each structural layer can independently apply the method for generating the context vector described in the present application, but are not limited thereto. The structural layers can independently apply the method for generating the context vector described in the present application according to the order of the structural layers, but are not limited thereto.

[0047] Optionally, in the present embodiment, the context vector plays a core role in natural language processing and deep learning, especially in recurrent neural networks, long short-term memory networks, gated recurrent units, and modern Transformer models (an encoder-decoder architecture). The context vector can represent a summary of information in a certain sequence or context, reflecting the model's understanding of the current input and the relationship between the current input and historical input. As described above, the context vector is particularly important in LLM inference. The context vector can capture historical context information of the text sequence, helping the model generate more coherent and meaningful output. Specifically, when processing each input token, the model generates a vector by considering all previous tokens to form a prediction for the current token. This vector integrates the features of previous tokens, helping the model make decisions based on more comprehensive historical information when generating the next token.

[0048] It should be noted that the method for generating the context vector provided in the present application can be applied to large language models, or can also be applied to other models that need to generate context vectors, such as video action recognition models and audio recognition and synthesis models, etc. When applied to a video action recognition model, the video action recognition model processes a sequence of video frames and can generate a context vector to understand the continuity of actions and changes in scenes. When applied to an audio recognition and synthesis model, the context vector can be generated by processing a time sequence of audio signals to recognize or generate syllables, melodies, etc.

[0049] Optionally, in the embodiment provided in step S202, the target vector can be, but is not limited to, a formatted representation of the base inference information received by the model. For example, the target vector can be, but is not limited to, an embedding vector converted from the input token, as previously discussed, whenever a new token t is input into the model, it is converted into a corresponding vector form (i.e., input vector ), which lays the foundation for subsequent inference processes.

[0050] Optionally, in the embodiment provided in step S204, the first vector information can be, but is not limited to, used to indicate a first historical vector received historically, and the second vector information can be, but is not limited to, used to indicate a second historical vector received historically. It should be noted that the first historical vector and the second historical vector are different historical vectors, and the influence of the first historical vector on the target vector can be, but is not limited to, higher than the influence of the second historical vector on the target vector.

[0051] Optionally, in this embodiment, the influence of each historical vector on the currently input target vector can be, but is not limited to, the strength of the correlation or the degree of information contribution between them. This influence evaluation can be, but is not limited to, based on the weight or score of the historical vector and the target vector in a specific calculation process, such as in time series analysis, the correlation between historical data points and current data points can be measured by the similarity of the latest trend, the degree of trend matching or statistical correlation; in video action recognition, the influence of historical frames on the current frame can be evaluated by the similarity of motion patterns or the continuity of features.

[0052] Optionally, in this embodiment, the influence of each historical vector on the currently input target vector can be, but is not limited to, determined by comparing the cumulative importance of each historical vector to the last input vector of the target vector.

[0053] Optionally, in this embodiment, the first vector information can be, but is not limited to, stored in a first storage space on the accelerator card, for example, the first vector information can be, but is not limited to, stored in the video memory on the GPU.

[0054] Optionally, in this embodiment, the second vector information can be, but is not limited to, stored in a second storage space on the processor, for example, the second vector information can be, but is not limited to, stored in the memory on the CPU.

[0055] Optionally, in the embodiment, the first sub-vector can be, but is not limited to, used to indicate the relationship between the newly input target vector and the first historical vector, and the second sub-vector can be, but is not limited to, used to indicate the relationship between the newly input target vector and the second historical vector. The nature and calculation method of such relationship can be, but is not limited to, varied according to the type of data processed by the model and the task requirement. For example, in a time series prediction model, such relationship can be, but is not limited to, embodied as the trend continuity or periodic correlation between the historical data points and the current data points; in a video analysis model, such relationship can be, but is not limited to, represented as the continuity of the previous frame and the current frame in object recognition or motion trajectory. In all different application backgrounds, the first sub-vector and the second sub-vector can be, but are not limited to, used to help the model more accurately understand the position and role of the current input in the sequence by capturing and quantifying such relationship, so as to optimize the subsequent inference process.

[0056] Optionally, in the embodiment, the second sub-vector generated by the processor of the inference device can be, but is not limited to, obtained by waiting for receiving the second sub-vector sent by the processor of the inference device within a target time period before and after the generation of the first sub-vector, or can be, but is not limited to, obtained by reading the second sub-vector from the fixed memory area of the processor of the inference device after the generation of the first sub-vector, and the like.

[0057] Optionally, in the embodiment, the second sub-vector generated by the processor of the inference device can be, but is not limited to, the second sub-vector of the target vector generated by the processor of the inference device according to the second vector information stored in the second storage space, or the second sub-vector of the target vector generated by the processor of the inference device according to part of the vector information in the second vector information stored in the second storage space, and the like.

[0058] Optionally, in the embodiment provided in step S206, the context vector of the target vector can be, but is not limited to, generated according to the first sub-vector and the second sub-vector after the first sub-vector is generated by the accelerator according to the first vector information and the second sub-vector generated by the processor of the inference device is obtained. The context vector here can be, but is not limited to, the output of each attention head as described above. .

[0059] Optionally, in the embodiment, the context vector of the target vector generated according to the first sub-vector and the second sub-vector can be, but is not limited to, obtained by fusing the first sub-vector and the second sub-vector according to a default first fusion proportion (for example, 70%) and a default second fusion proportion (for example, 30%) to obtain the context vector. The default first fusion proportion and the default second fusion proportion can be, but are not limited to, flexibly set according to experience.

[0060] Optionally, in the embodiment, the context vector of the target vector generated according to the first sub-vector and the second sub-vector can also but not limited to include generating a first fusion factor according to the first vector information while generating the first sub-vector, and obtaining a second fusion factor generated by the processor while obtaining the second sub-vector generated by the processor, fusing the first sub-vector and the second sub-vector according to the first fusion factor and the second fusion factor to obtain the context vector.

[0061] Optionally, Figure 3 is a schematic diagram of a context vector generation method according to an embodiment of the present application. As shown in Figure 3 , a large language model including an N-layer structure can be but not limited to deployed, wherein each layer structure includes a multi-head attention mechanism and a feed-forward neural network, and there are embedding layers and MLP layers at the beginning and end of the large language model. It is found through research that in attention calculation, the nearest token and a few key tokens far away are more likely to be paid attention to by the current token, and when GPU memory resources are scarce, tokens far away and with less influence can be but not limited to evicted to reduce memory occupation. Since the importance of a few tokens is dynamically changing, those tokens that are initially evicted may become important again in the subsequent decoding process. Therefore, the KV cache of tokens far away and with less influence can be but not limited to evicted from the GPU memory and offloaded to the CPU memory. Subsequently, at each decoding iteration, in addition to using the KV cache in the GPU memory for dense attention calculation, important tokens can also but not limited to be retrieved from the KV cache in the CPU for sparsity attention calculation. After the sparsity attention is calculated on the CPU, an independent calculation strategy is adopted to calculate the attention head output of the CPU part, which is transmitted to the GPU and combined with the same attention head output in the GPU. By using this kind of attention head input calculation method (i.e., the context vector generation method), the occupation of the KV cache to the GPU memory can be reduced while ensuring the running accuracy and inference quality of the large language model.

[0062] As an optional solution, the first sub-vector of the target vector generated according to the first vector information can be but not limited to generated by the following way: generating a first sub-vector according to the first vector information, and generating a first fusion factor of the first sub-vector according to the first vector information, wherein the first fusion factor is used to indicate the fusion proportion of the first sub-vector, and the first fusion factor is used to fuse the first sub-vector and the second sub-vector to generate the context vector.

[0063] Optionally, in the embodiment, the first fusion factor of the first sub-vector according to the first vector information can be generated at the same time according to the first sub-vector, but is not limited to. Compared with the scheme of directly using the default fusion factor, the first fusion factor calculated according to the first vector information can better fuse the first sub-vector in combination with the distribution of the first historical vector, and reduce the model running precision deviation caused by the generation of the context vector in the vector fusion manner.

[0064] Optionally, in the embodiment, the first sub-vector according to the first vector information can be generated by the following manner: the correlation degree of each first vector with the target vector is calculated according to the query vector and the key vector of each first vector, wherein the first vector information includes the key vector of the first historical vector and the value vector of the first historical vector, and the first vector includes the first historical vector and the target vector; the correlation degree of each first vector with the target vector is calculated, and the cumulative sum of the product of the value vector of the corresponding first vector is obtained to obtain the first sub-vector.

[0065] Optionally, in the embodiment, the first sub-vector can be calculated by the following formula (i.e. O1):

[0066] ;

[0067] wherein, is the correlation degree of each first vector with the target vector, is the value vector of each first vector. Each first vector can be traversed to calculate the correlation degree of each first vector with the target vector, the product of each correlation degree and the value vector of the corresponding first vector is calculated, and finally the cumulative sum of each product is calculated to obtain the first sub-vector.

[0068] Optionally, in the embodiment, the correlation degree of each first vector with the target vector can be calculated according to the query vector and the key vector of each first vector by the following formula : wherein, t1 is the number of first vectors, max(c) can be but not limited to the maximum value of , , is the query vector, is the key vector of each first vector, is the vector dimension of the query vector.

[0069] As an optional solution, the first fusion factor of the first sub-vector according to the first vector information can be generated in the following manner, but is not limited to: calculating the dot product of the key vector of each first vector and the query vector of the target vector, to obtain a plurality of first dot product values corresponding to each first vector, wherein the first vector includes the first historical vector and the target vector, and the first vector information includes the key vector of the first historical vector; calculating the quotient between each first dot product value and the square root of the vector dimension of the query vector, to obtain a plurality of first intermediate values; and generating the first fusion factor according to each first intermediate value.

[0070] Optionally, in the embodiment, the target vector can exist in the form of a column vector, and the dot product of the query vector of the target vector and the key vector of each first vector can be calculated in the following manner, but is not limited to: . , wherein x belongs to [1, t1], and t1 is the number of the first vectors.

[0071] Optionally, in the embodiment, the target vector can also exist in the form of a row vector, and the dot product of the query vector of the target vector and the key vector of each first vector can be calculated in the following manner, but is not limited to: . , wherein x belongs to [1, t1], and t1 is the number of the first vectors. Here, a plurality of forms of the target vector have been shown, and the processes after this are all exemplified in the form of a column vector. The process represented in the form of a row vector is similar to the process represented in the form of a column vector, and will not be described hereafter.

[0072] Optionally, in the embodiment, the quotient between each first dot product value and the square root of the vector dimension of the query vector can be calculated in the following manner, but is not limited to: , to obtain a plurality of first intermediate values C(x), C(x) = , wherein x belongs to [1, t1].

[0073] As an optional solution, the first fusion factor of the first sub-vector according to the first vector information can be generated in the following manner, but is not limited to:

[0074] The first fusion factor is generated in the following formula:

[0075] ,

[0076] , wherein LSE1 is the first fusion factor, t1 is the number of the first vectors, C(x) is the first intermediate value corresponding to the xth first vector, and M1 is the largest first intermediate value.

[0077] Optionally, in the embodiment, ​​The first vector can be traversed, and the sum of the first vector can be calculated. The sum of the first vector can be calculated.

[0078] As an optional solution, the second sub-vector generated by the processor of the inference device can be obtained by the following method: obtaining the second sub-vector generated by the processor of the inference device, and obtaining a second fusion factor of the second sub-vector generated by the processor of the inference device, wherein the second fusion factor is used to indicate the fusion proportion of the second sub-vector, and the second fusion factor is used to fuse the first sub-vector and the second sub-vector to generate the context vector.

[0079] Optionally, in this embodiment, the second fusion factor of the second sub-vector generated by the processor of the inference device can be obtained at the same time as the second sub-vector generated by the processor of the inference device. Compared with the solution of directly using a default fusion factor, the generated second fusion factor is more flexible, can better fuse the second sub-vector in combination with the distribution of the second historical vector, and reduces the model running precision deviation caused by the vector fusion method for generating the context vector.

[0080] Optionally, in this embodiment, the processor of the inference device can generate the second sub-vector according to the second vector information and generate the second fusion factor of the second sub-vector according to the second vector information.

[0081] As an optional solution, the second sub-vector generated by the processor of the inference device can be obtained by the following method: sending a query vector of a target vector to the processor; receiving the second sub-vector and the second fusion factor sent by the processor in response to the query vector, wherein the processor is configured to filter a reference historical vector from a plurality of second historical vectors according to the query vector and second vector information, generate the second sub-vector according to third vector information corresponding to the reference historical vector and the query vector, and generate the second fusion factor according to the third vector information corresponding to the reference historical vector and the query vector, wherein the cumulative importance degree of the reference historical vector is greater than an importance degree threshold, the cumulative importance degree is a cumulative value of the correlation degree of each second historical vector with itself and the correlation degree of each second historical vector with an input vector received after each second historical vector, and the second vector information includes the third vector information.

[0082] ​Optionally, in the embodiment, while the first fusion factor and the first sub-vector are calculated on the acceleration card side, the processor side can but not limited to have an asynchronous sparsification thread to start evaluating the vector information (e.g. KV cache) on the processor in parallel with the inference on the acceleration card, and add part of the vector information to the context set S (i.e. determine the reference history vector), and then generate the second sub-vector and the second fusion factor according to the third vector information of the reference history vector.

[0083] Optionally, in the embodiment, the query vector of the target vector can but not limited to be sent to the processor, and the second sub-vector and the second fusion factor are generated according to the second vector information by means of the computing power of the processor, thereby reducing the computing power burden of the acceleration card.

[0084] As an optional solution, the second fusion factor can but not limited to be generated according to the third vector information corresponding to the reference history vector and the query vector by the following method: calculating the dot product of the query vector and the key vector of each reference history vector to obtain a plurality of second dot product values corresponding to each reference history vector one by one, wherein the third vector information includes the key vector of the reference history vector; calculating the quotient between each second dot product value and the square root of the vector dimension of the query vector to obtain a plurality of second intermediate values; and generating the second fusion factor according to each second intermediate value.

[0085] Optionally, in the embodiment, the target vector can but not limited to exist in the form of a column vector, and the query vector of the target vector can but not limited to be calculated as follows: the dot product of the query vector and the key vector of each reference history vector: wherein y belongs to [1, t2], and t2 is the number of reference history vectors.

[0086] Optionally, in the embodiment, the quotient between each second dot product value and the square root of the vector dimension of the query vector can but not limited to be calculated to obtain a plurality of second intermediate values C(y), C(y) = wherein y belongs to [1, t2].

[0087] As an optional solution, the second fusion factor can but not limited to be generated according to each second intermediate value by the following method: the second fusion factor is generated by the following formula:

[0088] ,

[0089] wherein LSE2 is the second fusion factor, t2 is the number of reference history vectors, C(y) is the second intermediate value corresponding to the yth reference history vector, and M2 is the largest second intermediate value.

[0090] Optionally, in the embodiment,​​ It can, but is not limited to, representing traversing each reference history vector and calculating the value of each reference history vector. Finally, the values ​​of each reference history vector are calculated. The sum of.

[0091] As an optional approach, the following methods can be used to select reference historical vectors from multiple second historical vectors based on query vectors and second vector information: calculate the relevance of each second historical vector to the target vector based on the query vector and the key vectors of each second historical vector, wherein the second vector information includes the key vectors of the second historical vectors; update the cumulative importance of each second historical vector based on the relevance of each second historical vector to the target vector; and select second historical vectors from multiple second historical vectors whose cumulative importance is greater than the importance threshold to obtain reference historical vectors.

[0092] Optionally, in this embodiment, generally speaking, the influence of the second historical vector on the target vector is lower than that of the first historical vector. Using all the second historical vectors stored in the second storage space in the generation process of the target vector's context vector is not very efficient. Therefore, it is possible, but not limited to, selecting a portion of the more important reference historical vectors from the second historical vectors, and using the third vector information of the selected reference historical vectors as the basis for generating the second sub-vector. By selecting reference historical vectors, the processor's computing power can be utilized more efficiently, avoiding unnecessary computations that waste processor power and model inference time.

[0093] Optionally, in this embodiment, a cumulative weighted average attention weight (i.e., cumulative importance) may be maintained for each KV cache (or for each second history vector), but not limited to. The value can be defined, but is not limited to, as:

[0094] ;

[0095] in, t3 represents the number of the second history vectors, and max(c) can refer to, but is not limited to, the number of the second history vectors. hour, The maximum value, That is, the query vector. That is, the key vectors of each second history vector. That is, the vector dimension of the query vector.

[0096] That is, for the j-th historical token (i.e., the second historical vector) of the i-th attention head in the n-th layer structure, this is a weighted average of the historical average attention weight (i.e., the cumulative importance of the past) and the newly calculated attention weight (i.e., the relevance), where , The cumulative weighted average attention weight (i.e., the cumulative importance) can be but is not limited to used to accumulate the importance of the history, current attention weight, when The larger the , the more attention is paid to the current token, and the faster the importance of the history token decays. Compared with the simple average, the weighted average introduces the time sequence, which can be adjusted by to make it pay more attention to the recently generated KV cache (or the second history vector), so as to better fit the characteristics of the LLM model autoregressive generation with the front and rear order.

[0097] Optionally, in the present embodiment, after the (i.e., the query vector) is transmitted from the GPU to the CPU (i.e., the processor) asynchronously, the processor can but is not limited to perform attention calculation on the KV cache (i.e., the key vector of the second history vector) of the history token j (i.e., the second history vector), and update the cumulative weighted average attention weight of the attention head i. The processor can but is not limited to normalize the cumulative weighted average attention weight of the KV cache corresponding to the attention head i, and then construct the context set S (i.e., select the reference history vector), where the normalization is to facilitate the importance evaluation in the subsequent process. For the second history vector (i.e., the importance threshold), it can but is not limited to be added to S (i.e., it is determined as the reference history vector), which can but is not limited to represent the number of KV caches (i.e., the number of second history vectors) in the processor under the attention head, The meaning of the formula is that the importance of the KV cache is higher than the average importance by times. The threshold parameter can but is not limited to used to control the degree of sparsification, The larger the , the more strict the filtering is, and the less the number of KV caches is reserved (i.e., the number of final determined reference history vectors is less), which can reduce the CPU computing overhead, but may sacrifice the accuracy; The smaller the , the more the number of KV caches is reserved, and the more attention is paid to the accuracy but the larger the overhead is. The mechanism can but is not limited to dynamically run per attention head, and adaptively adjust the attention distribution: the head with a concentrated distribution reserves less KV cache (i.e., determines a larger , to finally obtain less reference history vector), and the head with a flat distribution reserves more KV cache (i.e., determines a smaller , to finally obtain more reference history vector). which can but is not limited to set to 1, is a relatively balanced setting, when When set to be less than 1, more KV cache is reserved with priority of accuracy, and when set to be greater than 1, CPU calculation is reduced with priority of efficiency. When set to be greater than 1, CPU calculation is reduced with priority of efficiency. For all historical tokens of the i-th attention head (i.e., all second historical vectors), the cumulative weighted average attention weight thereof can be but not limited to calculated and determined whether to be added to S. For all attention heads of all layers, the above steps can be but not limited to performed to complete the processing of all attention heads.

[0098] As an optional solution, the second sub-vector can be but not limited to generated according to the third vector information corresponding to the reference historical vector and the query vector in the following manner: the correlation degree of each reference historical vector and the target vector is calculated according to the key vector of the query vector and each reference historical vector, wherein the third vector information comprises the key vector of the reference historical vector and the value vector of the reference historical vector; and the second sub-vector is obtained by calculating the cumulative sum of the product of the correlation degree of each reference historical vector and the target vector and the value vector of the corresponding reference historical vector.

[0099] Optionally, in the embodiment, the second sub-vector can be but not limited to calculated according to the following formula (i.e., O2):

[0100] ;

[0101] wherein, is the correlation degree of each reference historical vector and the target vector, is the value vector of each reference historical vector. Each reference historical vector can be but not limited to traversed, the correlation degree of each reference historical vector and the target vector is calculated, the product of each correlation degree and the value vector of the corresponding reference historical vector is calculated, and finally the cumulative sum of each product is calculated to obtain the second sub-vector.

[0102] Optionally, in the embodiment, the correlation degree of each reference historical vector and the target vector can be but not limited to calculated according to the following formula : wherein, t2 is the number of reference historical vectors, and max(c) can be but not limited to the maximum value of , , is the query vector, is the key vector of each reference historical vector, is the vector dimension of the query vector.

[0103] As an optional solution, the context vector generated according to the first sub-vector and the second sub-vector can be generated by the following method, but is not limited to: obtaining a first fusion factor of the first sub-vector and a second fusion factor of the second sub-vector, wherein the first fusion factor is used to indicate the fusion proportion of the first sub-vector, and the second fusion factor is used to indicate the fusion proportion of the second sub-vector; and generating the context vector by the following formula:

[0104] ,

[0105] wherein O is the context vector, O1 is the first sub-vector, O2 is the second sub-vector, LSE1 is the first fusion factor, and LSE2 is the second fusion factor.

[0106] Optionally, in the embodiment, the attention calculation (i.e. the generation of the context vector) can include two parts, but is not limited to: using the nearest token and a small number of key tokens far away to perform full attention calculation on the GPU (i.e. the accelerator); and performing sparse attention calculation on the CPU (i.e. the processor), i.e. no longer using all the KV caches, but using the re-evaluated token S set (i.e. the reference history vector) retrieved from the unloaded KV cache (i.e. the second history vector) to perform sparse attention calculation. In this way, the computational load imbalance between the GPU and the CPU is reduced, the GPU performs small-range high-density calculation, the CPU processes a light sparse subset, and either side is prevented from becoming a bottleneck; and the GPU and the CPU are concurrently run through efficient cooperative parallelism, the sparse attention calculation is processed in parallel by the multi-core CPU, efficient data structures are used, and the delay is reduced through zero-copy communication optimization.

[0107] Optionally, for the i-th attention head of the n-th layer, the following formula is used: First, the GPU (i.e. the accelerator) side is calculated as follows: , wherein t1 represents that the number of KV caches reserved by the attention head on the GPU is t1. While calculating the attention on the GPU side, the following is transmitted to the CPU (i.e. the processor) side: The context set S (i.e. the reference history vector) is used to perform CPU-side sparse attention calculation in parallel: , wherein t2 represents that the number of KV caches reserved by the attention head in the set S is t2. When fusing and , it should be noted that, as can be seen from the calculation formula of each , when calculating the attention weight (i.e. calculating the correlation degree), whether on the CPU side or the GPU side, the maximum value of the in the KV cache saved by itself and the sum of the attention weights of all the history tokens in the KV cache saved by itself is 1, i.e. the following is calculated: and When, respectively, only the part of the CPU or the part of the GPU is processed with numerical stability and normalization, the following strategy is needed to process the fusion operation with numerical stability and normalization: it is known that, It can be but is not limited to recording Here, the value of K is t1 or t2. It can be but is not limited to recording The maximum value is The maximum value obtained on the GPU is The maximum value is The overall maximum value on the CPU and the GPU is . In order to obtain the required attention head output It can be but is not limited to the need to convert , fusion into It can be but is not limited to the following strategy for fusion:

[0108] ;

[0109] From the above analysis, it can be seen that recording , , , , and can restore , and further can be obtained.

[0110] Further, it can be but is not limited to further optimization on this basis, and it can be but is not limited to recording , ,

[0111] Because , then,

[0112] ,

[0113] then,

[0114] ,

[0115] and ,

[0116] and ,

[0117] then,

[0118] ,

[0119] So the fusion formula is:

[0120] (i.e. );

[0121] That is, but not limited to, only record , , can restore .

[0122] Since the final fusion operation is performed on the GPU, it can be but not limited to only need to pass (i.e. LSE2, the second fusion factor) and (i.e. O2, the second sub-vector) to the GPU, and the GPU calculates (i.e. O, the context vector) to get it.

[0123] Optionally, pass into the GPU, and when transmitting, it can be but not limited to using zero-copy memory access to make the GPU directly read the CPU data, bypassing the CPU to GPU memory copy step, thereby improving access speed and accelerating the generation speed of the context vector. It can be but not limited to performing fusion operation according to the formula (i.e. ) to ensure that the next layer of feedforward network can directly use the output. It can be but not limited to performing the fusion operation of the first sub-vector and the second sub-vector for all attention heads of all layers, thereby completing all attention fusion calculations.

[0124] Through the above fusion strategy, the separated attention output can be fused to obtain a result equivalent to unified calculation and maintain numerical stability.

[0125] As an optional solution, after receiving the target vector, the vector information stored in the first storage space and the vector information stored in the second storage space can be but not limited to adjusted in the following way: storing the target vector information of the target vector into the first storage space; adjusting the vector information stored in the first storage space according to the use parameter, the vector parameter and the time parameter of the first storage space to obtain the first storage space storing the first vector information, and the vector information stored in the second storage space to obtain the second storage space storing the second vector information, wherein the use parameter is used to indicate the occupation degree of the first storage space, the vector parameter is used to indicate the correlation degree of the vector corresponding to the vector information currently stored in the first storage space and the target vector, and the time parameter is used to indicate the storage time of the vector information currently stored in the first storage space.

[0126] Optionally, in this embodiment, a cache pool can be initialized during model initialization, taking the LLM model as an example. A contiguous cache block is allocated in the GPU for the LLM model with an N-layer network structure, and this cache block is divided into N equal parts to store the key-value (KV) cache for each layer (i.e., the first storage space corresponding to multiple attention heads in each layer). An idle ratio T is set for each layer's cache block. When the stored KV cache exceeds the ratio T, an unloading operation is performed. This hierarchical division not only fits the model structure but also allows for independent cache management for each layer—when unloading is performed, each layer unloads independently, so that unloading the KV cache of one layer does not affect other layers.

[0127] Optionally, in this embodiment, during inference, for the input token t (corresponding to the target vector), a KV cache is calculated at the nth layer to generate a cache (i.e., target vector information):

[0128] ;

[0129] ;

[0130] ;

[0131] Each subvector here This corresponds to the i-th attention head.

[0132] Optionally, in this embodiment, after receiving the target vector, the target vector information can be stored in the first storage space. After new vector information is stored in the first storage space, the vector information stored in the first storage space is adjusted according to the usage parameters, vector parameters, and time parameters of the first storage space, and the vector information stored in the second storage space is also adjusted.

[0133] Optionally, in this embodiment, adjusting the vector information stored in the first storage space and the vector information stored in the second storage space may, but is not limited to, be mutually corresponding. For example, it may include deleting some of the vector information stored in the first storage space from the first storage space and storing it in the second storage space.

[0134] Optionally, in this embodiment, adjusting the vector information stored in the first storage space and the vector information stored in the second storage space can also be independent of each other. For example, it can include deleting some of the vector information stored in the first storage space from the first storage space, and extracting zero, one or more vector information from the deleted vector information in the first storage space and storing them in the second storage space. The extraction criteria can be, but are not limited to, the correlation between the corresponding historical vector and the target vector or the storage time of the vector information, etc.

[0135] Optionally, in the embodiment, the target vector information may, but is not limited to, include a query vector, a key vector and a value vector of the target vector.

[0136] Optionally, in the embodiment, the usage parameter may, but is not limited to, be used to indicate the usage degree of the first storage space, and a usage parameter threshold value may, but is not limited to, be set for the first storage space. In a case where the usage parameter is greater than or equal to the usage parameter threshold value, the vector information stored in the first storage space is adjusted and the vector information stored in the second storage space is adjusted.

[0137] Optionally, in the embodiment, the vector parameter may, but is not limited to, be used to indicate the correlation degree between the vector corresponding to the currently stored vector information in the first storage space and the target vector, and the time parameter may, but is not limited to, be used to indicate the storage time of the currently stored vector information in the first storage space. The more important the vector information is, the higher the correlation degree between the vector corresponding to the currently stored vector information in the first storage space and the target vector is, and the more important the vector information is, the later the storage time of the currently stored vector information in the first storage space is. The relatively less important vector information in the currently stored vector information in the first storage space may, but is not limited to, be unloaded to the second storage space of the processor according to the vector parameter and the time parameter. The importance degree of the vector information filtered out from the first storage space may, but is not limited to, be lower than the importance degree of the vector information remaining in the first storage space after filtering.

[0138] Through the above steps, the related information of all the historical vectors is no longer saved on the accelerator card, but the vector information stored in the first storage space and the vector information stored in the second storage space are adjusted according to the usage parameter, the vector parameter and the time parameter of the first storage space, thereby effectively reducing the usage of the storage space on the accelerator card.

[0139] As an optional solution, the first storage space storing the first vector information and the second storage space storing the second vector information may, but are not limited to, be obtained by adjusting the vector information stored in the first storage space according to the usage parameter, the vector parameter and the time parameter of the first storage space in the following manner: comparing the usage parameter with the usage parameter threshold value; and in a case where the usage parameter is greater than or equal to the usage parameter threshold value, transferring the vector information of part of the vectors in a plurality of second vectors to the second storage space according to the vector parameter and the time parameter, wherein the second vector is a vector in which the current vector information is stored in the first storage space.

[0140] Optionally, in the embodiment, the vector information stored in the first storage space and the vector information stored in the second storage space may, but are not limited to, be maintained in a case where the usage parameter is less than the usage parameter threshold value, that is, the vector information stored in the first storage space is not adjusted and the vector information stored in the second storage space is not adjusted.

[0141] Optionally, in the embodiment, whether the first storage space is in the situation of insufficient available space after the new target vector information is stored in the first storage space can be determined by comparing the usage parameter and the usage parameter threshold, and whether the vector information stored in the first storage space needs to be adjusted can be determined.

[0142] Optionally, in the embodiment, the second vector can be but is not limited to the vector whose vector information is stored in the first storage space, and the second vector can include but is not limited to the target vector.

[0143] Optionally, in the embodiment, when the usage parameter is greater than or equal to the usage parameter threshold, that is, when the first storage space is in the situation of insufficient available space, part of the second vectors can be selected according to the vector parameter and the time parameter, and the vector information of the part of the second vectors can be stored in the second storage space (that is, stored in the second storage space and deleted from the first storage space).

[0144] According to the above, when the first storage space is in the situation of insufficient available space, part of the vector information stored in the first storage space is stored in the second storage space, compared with the scheme of directly deleting part of the vector information from the first storage space, the historical vector information will not be directly lost, and can still be used when needed.

[0145] As an optional solution, the transfer of vector information of a subset of multiple second vectors to a second storage space based on vector parameters and time parameters can be achieved, but is not limited to, in the following manner: Calculate the correlation between each second vector and the target vector based on the query vector of the target vector and the key vectors of each second vector to obtain vector parameters, where the vector information of the second vectors includes the key vectors of the second vectors, and the target vector information includes the query vectors of the target vectors; update the cumulative importance of each second vector based on the vector parameters, where the cumulative importance is the cumulative value of the correlation between each second vector and itself, and the correlation between each second vector and the input vectors received after each second vector; based on each second vector… The cumulative importance and time parameters are used to calculate the influence parameters of each second vector, where the influence parameters indicate the influence of each second vector on the target vector. Based on the influence parameters of each second vector, vectors to be transferred are selected from multiple second vectors, where the multiple second vectors include vectors to be transferred and non-transferable vectors. The influence parameters of the vectors to be transferred are less than or equal to the influence parameters of the non-transferable vectors, and the occupancy of the vector information of the non-transferable vectors in the first storage space is less than the occupancy of the first storage space indicated by the parameter threshold. After generating the first sub-vector of the target vector based on the first vector information and obtaining the second sub-vector generated by the processor of the inference device, the vector information of the vectors to be transferred is transferred to the second storage space.

[0146] Optionally, in this embodiment, the query vector of the target vector can be used, but is not limited to, by the following formula. and the key vectors of each second vector Calculate the correlation between each second vector and the target vector to obtain the vector parameters:

[0147] ;

[0148] Where t can be, but is not limited to, the number of the second vector, and max(c) can be, but is not limited to, the case where x belongs to [1,t], and c x The maximum value. This represents the correlation between the j-th second vector and the target vector. This formula can, but is not limited to, represent calculating the query result for each historical position j. s and keys The dot product is the attention computation of the t-th token relative to the j-th token, divided by the scaling factor. To avoid gradient problems caused by excessively large values. Because exponential operations can lead to softmax overflow issues, it is possible, but not limited to, setting each dot product to subtract the maximum value of the dot products at all historical positions. . The results for all historical positions are normalized using softmax to get the attention weight for each position. Then The attention calculation is performed for all historical tokens (i.e., the second vector) to obtain the attention weight:

[0149] ;

[0150] As can be seen from the foregoing, the normalization can make the historical token from 1 to t, and all Add up to 1.

[0151] Optionally, in this embodiment, since the step of calculating the relevance of each second vector to the target vector according to the query vector of the target vector and the key vector of each second vector is also used in calculating the first sub-vector, the target vector information of the target vector can be but not limited to stored to the first storage space after receiving the target vector, and then the relevance of each second vector to the target vector is calculated according to the query vector of the target vector and the key vector of each second vector. On the one hand, the original thread calculates the first sub-vector according to the relevance, and on the other hand, an asynchronous thread is started to perform the following operations: comparing the use parameter and the use parameter threshold value, in the case that the use parameter is greater than or equal to the use parameter threshold value, updating the cumulative importance degree of each second vector according to the vector parameter, calculating the influence parameter of each second vector according to the cumulative importance degree of each second vector and the time parameter, selecting the to-be-stored vector from the plurality of second vectors according to the influence parameter of each second vector, after the original thread generates the first sub-vector of the target vector according to the first vector information, and obtains the second sub-vector generated by the processor of the inference device, the asynchronous thread stores the vector information to be stored to the second storage space.

[0152] Optionally, in this embodiment, an asynchronous thread can be but not limited to started to perform the following operations: maintaining a cumulative weighted average attention weight (i.e., cumulative importance degree) value for each KV cache, which can but not limited to update the cumulative importance degree of each second vector according to the vector parameter through the following formula :

[0153] ;

[0154] That is, for the jth historical token (i.e., the second vector) of the ith attention head of the nth layer, it is a weighted average of a historical average attention weight (i.e., a historical cumulative importance degree) and a newly calculated attention weight (i.e., a vector parameter), where , which can be used to adjust the importance of the cumulative importance degree to the historical and current attention weight, when The greater the cumulative importance, the more attention is paid to the current token, and the faster the importance of the historical token decays. Compared with simple averaging, weighted averaging introduces a time sequence, which can be adjusted to focus more on recently generated KV cache (i.e., vector information), thus better fitting the LLM model's autoregressive generation with a before-after sequence.

[0155] Optionally, in this embodiment, the unloading measurement index (i.e., influence parameter) of the jth historical token of the ith attention head of the nth layer can be calculated, but is not limited to:

[0156]

[0157] wherein, represents the KV cache generation time (i.e., time parameter) of the jth historical token of the ith attention head of the nth layer, may be but is not limited to an integer greater than 1, in order to be consistent with In order to maintain the same scale, the generation time (i.e., time parameter) can be but is not limited to scaled to be within the range of , and then the ratio of the cumulative weighted average attention weight (i.e., cumulative importance) and the generation time is used as the Score value (i.e., influence parameter). This formula takes into account both the age of the token and the cumulative attention weight. When the cumulative weighted average attention weight is the same, the KV cache with a longer generation time (i.e., time parameter) is unloaded first. When the generation time is the same, the KV cache with a smaller cumulative weighted average attention weight is unloaded first.

[0158] Optionally, in this embodiment, the Score value of the KV cache for all attention heads of the nth layer can be but is not limited to calculated in the above manner. Then, all Score values of the KV cache are sorted in descending order, and the KV cache that exceeds the preset cache block idle ratio T is selected to be unloaded to the CPU. The cumulative weighted average attention weight (i.e., cumulative importance) corresponding to the KV cache is also unloaded to the CPU at the same time. The cumulative importance on the GPU side unloaded to the CPU side can be but is not limited to become the initial value of the cumulative importance on the CPU side. For all structural layers, the above steps can be but are not limited to performed to complete the KV cache unloading of the attention head.

[0159] Through the above, the vector information of the second vector with a small influence parameter is transferred from the first storage space to the second storage space, and the vector information of the second vector with a large influence parameter is left in the first storage space. This reduces the storage space occupied by historical vector information on the accelerator while keeping important vector information on the accelerator, facilitating the use of the accelerator and ensuring the inference efficiency of the model.

[0160] ​As an optional solution, the influence parameter of each second vector can be calculated according to the accumulated importance degree and the time parameter of each second vector, but is not limited to the following manner: the influence parameter is calculated by the following formula:

[0161] ,

[0162] wherein Scorej is the influence parameter of the jth second vector, Aj is the accumulated importance degree of the jth second vector, Bj is the storage time of the vector information of the jth second vector, t is the number of second vectors, B0 is the longest storage time among the storage times of the vector information of the t second vectors, and the time parameter includes Bj and B0.

[0163] Optionally, in the embodiment, , i.e., the aforementioned ; Aj is , Bj is , and B0 is .

[0164] According to the above, the influence parameter of each second vector is determined in combination with the time parameter and the accumulated importance degree, and the influence parameter determined fully considers various influences and can better reflect the influence of each second vector on the target vector.

[0165] As an optional solution, the second vector to be stored can be selected from the plurality of second vectors according to the influence parameter of each second vector, but is not limited to the following manner: each second vector is arranged in descending order of the influence parameter to obtain a vector sequence; and the second vector to be stored is selected from the vector sequence according to the difference between the use parameter threshold and the use parameter.

[0166] As an optional implementation, the context vector generation method described in this application can be implemented through, but is not limited to, the following process: the accelerator card receives the target vector, generates target vector information of the target vector, stores the target vector information in the first storage space, and asynchronously sends the query vector of the target vector to the processor. Based on the target vector information and the vector information stored in the first storage space, the correlation between the target vector and the second vector corresponding to the vector information stored in the first storage space is calculated. The main thread continues to generate the first sub-vector and the first fusion factor based on the calculated correlation. An asynchronous thread is started to calculate the influence parameter corresponding to each second vector. Based on the influence parameter, the vector to be transferred is selected from multiple second vectors. After the main thread generates the first sub-vector and the first fusion factor and obtains the second sub-vector and the second fusion factor, the vector information and cumulative importance of the vector to be transferred are transferred to the processor's second storage space. After receiving the query vector, the processor selects a reference historical vector from the historical vectors corresponding to the vector information stored in the second storage space, based on the query vector and the vector information stored in the second storage space. It then generates a second fusion factor and a second sub-vector based on the reference historical vector and the query vector, and provides the second fusion factor and the second sub-vector to the accelerator card. Finally, it receives the vector information and cumulative importance of the vectors to be transferred from the accelerator card. It is important to note that the processes of generating the first sub-vector and the first fusion factor on the accelerator card side and the processes of generating the second sub-vector and the second fusion factor on the processor side are performed in parallel.

[0167] Optionally, in this embodiment, in order to better understand the above-mentioned context vector generation process, the above-mentioned context vector generation process will be described in conjunction with optional embodiments below, but it is not intended to limit the technical solution of the embodiments of this application.

[0168] This embodiment also provides a method for unloading key-value cache. Figure 4 This is a schematic diagram of the process of unloading a key-value cache according to an embodiment of this application, as shown below. Figure 4 As shown, a method for unloading a key-value cache may include, but is not limited to, the following steps:

[0169] Step S1: The LLM inference system first loads and initializes the model;

[0170] Step S2: Then initialize a cache block and divide the cache block into N blocks (corresponding to N GPUs, resulting in N first storage spaces);

[0171] Step S3: Start reasoning. For the input token t, at the nth level, calculate the KV cache and generate the q vector (i.e., query vector), k vector (i.e., key vector), and v vector (i.e., value vector). Asynchronously transmit the q vector to the CPU.

[0172] Step S4: On the GPU side, for each input token t, at each attention head of each layer, the attention weight is calculated according to the q vector and the k vector of the historical token;

[0173] Step S5: Calculate the cumulative weighted average attention weight (i.e., cumulative importance) for each historical token;

[0174] Step S6: Calculate the unloading measurement indicator Score value (i.e., influence parameter) for each historical token;

[0175] Step S7: For all attention heads of the nth layer, sort the Score values of the KV cache from large to small, and select the excess KV cache to be unloaded to the CPU according to the preset cache block idle ratio T.

[0176] Step S8: Process all layers according to the rules of steps S3 to S7;

[0177] Step S9: The thread running asynchronously on the CPU starts to evaluate the KV cache after receiving the q input from the GPU, updates the cumulative weighted average attention weight (i.e., cumulative importance) thereof, and according to the size of the cumulative weighted average attention weight, adds part of the KV cache to the context set S (i.e., determines the reference historical vector);

[0178] Step S10: In parallel during the calculation of attention, the GPU performs complete attention calculation using the most recent token and a small number of key tokens that are far away, and the CPU performs sparse attention calculation using the S set.

[0179] Step S11: Finally, the results LSEcpu (i.e., the second fusion factor) and ocpu (i.e., the second sub-vector) executed on the CPU are transmitted to the GPU, and the fusion of the attention layer output is performed on the GPU. Using the fused results, the calculation of the subsequent layer FFN is performed.

[0180] Figure 4 The changes in the KV cache in the second storage space of the central processing unit CPU and the changes in the KV cache in the first storage space of the accelerator GPU are also shown.

[0181] In summary, by using the KV cache unloading strategy, the method of the present application can support longer context tasks and thus more model applications in the case of limited GPU video memory. By using the GPU-CPU parallel attention calculation strategy, the computing power and large memory of the CPU can be effectively utilized, and the waste of computing power and storage resources on the CPU during the inference of traditional LLM models can be avoided.

[0182] Because in different decoding steps, the importance of a small part of tokens is dynamically changing. Those tokens that are initially excluded may become important again in subsequent decoding processes. Through the KV offloading strategy of the present application, the KV cache is not deleted, but is retained on the CPU for reevaluation and participation in attention calculation, so that during inference, the degradation of model accuracy can be avoided.

[0183] Furthermore, through the method of the present application, only the output vector ocpu of the CPU and a scalar LSEcpu need to be transmitted during parallel attention calculation. Compared with the PCIe bandwidth pressure caused by transmitting the KV cache from the CPU to the GPU, the method of the present application greatly reduces the transmission amount, so that the transmission does not cause a performance bottleneck, thereby keeping pace with the attention calculation on the GPU.

[0184] Further, through the method of the present application, the equivalent fusion strategy can make the attention outputs calculated separately on the CPU and the GPU fused to complete a result equivalent to unified calculation, thereby avoiding the introduction of additional errors.

[0185] It is worth noting that in the method of the present application, KV offloading and CPU-side KV cache evaluation are asynchronous processes, so they do not affect the main process calculation, and during attention calculation, CPU-GPU parallelism is adopted, thereby avoiding an increase in inference process delay.

[0186] Finally, in the cumulative weighted average attention calculation of the method of the present application, compared with simple averaging, weighted averaging introduces a time sequence, which can be adjusted by adjusting the weight parameter w to make the cumulative importance pay more attention to recently generated KVs, thereby better fitting the characteristics of the LLM model autoregressive generation with a front-back order.

[0187] From the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software and the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is a better embodiment.

[0188] Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk) and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of the various embodiments of the present application.

[0189] ​An apparatus for generating a context vector is also provided in the embodiment. The apparatus is applied to an acceleration card of an inference device and is used to implement the above-described embodiments and preferred embodiments, which have been described above. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementation in hardware or a combination of software and hardware is also possible and contemplated.

[0190] Figure 5 is a structural block diagram of an apparatus for generating a context vector according to an embodiment of the present application. As shown in Figure 5 , the apparatus comprises:

[0191] a receiving module 502 configured to receive a target vector;

[0192] a first generating module 504 configured to generate a first sub-vector of the target vector according to first vector information and to obtain a second sub-vector generated by a processor of the inference device, wherein the first vector information is stored in a first storage space on the acceleration card, the first vector information is used to indicate a first historical vector that is historically received, the first sub-vector is used to indicate a relationship between the target vector and the first historical vector, the second sub-vector is used to indicate a relationship between the target vector and a second historical vector, the second vector information is stored in a second storage space on the processor, the second vector information is used to indicate the second historical vector that is historically received, and an influence of the first historical vector on the target vector is higher than an influence of the second historical vector on the target vector;

[0193] a second generating module 506 configured to generate a context vector of the target vector according to the first sub-vector and the second sub-vector.

[0194] With the above apparatus, the acceleration card of the inference device receives the target vector, generates the first sub-vector of the target vector according to the first vector information stored in the first storage space on the acceleration card, and obtains the second sub-vector generated by the processor of the inference device. The first sub-vector is used to indicate the relationship between the target vector and the first historical vector, the second sub-vector is used to indicate the relationship between the target vector and the second historical vector stored in the second storage space of the processor, the influence of the first historical vector on the target vector is higher than the influence of the second historical vector on the target vector, the historical vector information of the historical vector does not need to be stored only on the acceleration card, the occupation of the storage space of the acceleration card is reduced, and the acceleration card finally generates the context vector of the target vector according to the first sub-vector and the second sub-vector, which realizes the running accuracy of the model as much as possible while reducing the occupation of the storage space of the acceleration card and avoids introducing additional errors. Therefore, the technical problems such as the high occupation of the storage space of the acceleration card by the historical information in the related art are solved, and the technical effect of reducing the occupation of the storage space of the acceleration card by the historical information while maintaining the running accuracy of the model is achieved.

[0195] In an example embodiment, the first generating module comprises: a first generating unit configured to generate a first sub-vector according to the first vector information, and generate a first fusion factor of the first sub-vector according to the first vector information, wherein the first fusion factor is used to indicate a fusion proportion of the first sub-vector, and the first fusion factor is used to fuse the first sub-vector and a second sub-vector to generate the context vector.

[0196] In an example embodiment, the first generating unit is further configured to: calculate a dot product of a query vector of the target vector and a key vector of each first vector to obtain a plurality of first dot product values corresponding to each first vector, wherein the first vectors comprise the first historical vectors and the target vector, and the first vector information comprises the key vectors of the first historical vectors; calculate a quotient between each first dot product value and a square root of a vector dimension of the query vector to obtain a plurality of first intermediate values; and generate the first fusion factor according to each first intermediate value.

[0197] In an example embodiment, the first generating unit is further configured to generate the first fusion factor by the following formula:

[0198]

[0199] wherein LSE1 is the first fusion factor, t1 is the number of the first vectors, C(x) is the first intermediate value corresponding to the xth first vector, and M1 is the largest first intermediate value.

[0200] In an example embodiment, the first generating module further comprises: a first obtaining unit configured to obtain a second sub-vector generated by a processor of the inference device, and obtain a second fusion factor of the second sub-vector generated by the processor of the inference device, wherein the second fusion factor is used to indicate a fusion proportion of the second sub-vector, and the second fusion factor is used to fuse the first sub-vector and the second sub-vector to generate the context vector.

[0201] In an example embodiment, the first obtaining unit is further configured to: send the query vector of the target vector to the processor; and receive the second sub-vector and the second fusion factor sent by the processor in response to the query vector, wherein the processor is configured to: select a reference historical vector from a plurality of second historical vectors according to the query vector and second vector information, generate the second sub-vector according to third vector information corresponding to the reference historical vector and the query vector, and generate the second fusion factor according to the third vector information corresponding to the reference historical vector and the query vector, wherein the cumulative importance degree of the reference historical vector is greater than an importance degree threshold, the cumulative importance degree is a cumulative value of a correlation degree of each second historical vector with itself and a correlation degree of each second historical vector with an input vector received after each second historical vector, and the second vector information comprises the third vector information. ​

[0202] In an example embodiment, the first obtaining unit is further configured to: calculate dot products of the query vector and key vectors of the respective reference historical vectors to obtain a plurality of second dot product values corresponding to the respective reference historical vectors, wherein the third vector information comprises the key vectors of the reference historical vectors; calculate quotients between the respective second dot product values and square roots of vector dimensions of the query vector to obtain a plurality of second intermediate values; and generate the second fusion factor according to the respective second intermediate values.

[0203] In an example embodiment, the first obtaining unit is further configured to generate the second fusion factor according to the following formula:

[0204] ,

[0205] wherein LSE2 is the second fusion factor, t2 is the number of the reference historical vectors, C(y) is the second intermediate value corresponding to the yth reference historical vector, and M2 is the largest second intermediate value.

[0206] In an example embodiment, the first obtaining unit is further configured to: calculate correlation degrees of the respective second historical vectors and the target vector according to the query vector and key vectors of the respective second historical vectors, wherein the second vector information comprises the key vectors of the second historical vectors; update cumulative importance degrees of the respective second historical vectors according to the correlation degrees of the respective second historical vectors and the target vector; and select, from the plurality of second historical vectors, second historical vectors with cumulative importance degrees greater than the importance degree threshold to obtain the reference historical vectors.

[0207] In an example embodiment, the first obtaining unit is further configured to: calculate correlation degrees of the respective reference historical vectors and the target vector according to the query vector and key vectors of the respective reference historical vectors, wherein the third vector information comprises the key vectors of the reference historical vectors and value vectors of the reference historical vectors; and calculate an accumulated sum of the correlation degrees of the respective reference historical vectors and products of the value vectors of the corresponding reference historical vectors to obtain a second sub-vector.

[0208] In an example embodiment, the second generating module comprises: a second obtaining unit configured to obtain a first fusion factor of the first sub-vector and a second fusion factor of the second sub-vector, wherein the first fusion factor is used to indicate a fusion proportion of the first sub-vector, and the second fusion factor is used to indicate a fusion proportion of the second sub-vector; and a second generating unit configured to generate the context vector according to the following formula:

[0209] ,

[0210] wherein O is the context vector, O1 is the first sub-vector, O2 is the second sub-vector, LSE1 is the first fusion factor, and LSE2 is the second fusion factor.

[0211] In an example embodiment, the generation apparatus of the context vector further comprises: a storage unit, configured to store target vector information of the target vector into a first storage space after receiving the target vector; and an adjustment unit, configured to adjust the vector information stored in the first storage space according to a usage parameter, a vector parameter and a time parameter of the first storage space, to obtain the first storage space storing the first vector information and the second storage space storing the second vector information, wherein the usage parameter is used to indicate an occupation degree of the first storage space, the vector parameter is used to indicate a correlation degree between a vector corresponding to the current stored vector information in the first storage space and the target vector, and the time parameter is used to indicate a storage time of the current stored vector information in the first storage space.

[0212] In an example embodiment, the adjustment unit is further configured to: compare the usage parameter with a usage parameter threshold; and in a case where the usage parameter is greater than or equal to the usage parameter threshold, transfer the vector information of part of the second vectors in the plurality of second vectors to the second storage space according to the vector parameter and the time parameter, wherein the second vector is a vector in which the current vector information is stored in the first storage space.

[0213] In an example embodiment, the adjustment unit is further configured to: calculate the correlation degree between each second vector and the target vector according to a key vector of each second vector and a query vector of the target vector, to obtain the vector parameter, wherein the vector information of the second vector comprises the key vector of the second vector, and the target vector information comprises the query vector of the target vector; update an accumulated importance degree of each second vector according to the vector parameter, wherein the accumulated importance degree is a cumulative value of the correlation degree between each second vector and itself and the correlation degree between each second vector and an input vector received after each second vector; calculate an influence parameter of each second vector according to the accumulated importance degree of each second vector and the time parameter, wherein the influence parameter is used to indicate the influence of each second vector on the target vector; select a to-be-transferred vector from the plurality of second vectors according to the influence parameter of each second vector, wherein the plurality of second vectors comprise the to-be-transferred vector and a non-transferred vector, the influence parameter of the to-be-transferred vector is less than or equal to the influence parameter of the non-transferred vector, and the occupation degree of the first storage space by the vector information of the non-transferred vector is less than the occupation degree of the first storage space indicated by the usage parameter threshold; and transfer the vector information of the to-be-transferred vector to the second storage space after generating a first sub-vector of the target vector according to the first vector information and obtaining a second sub-vector generated by a processor of the inference device.

[0214] In an example embodiment, the adjustment unit is further configured to: calculate the influence parameter by the following formula:

[0215] ,

[0216] wherein, Scorej is the influence parameter of the jth second vector, Aj is the cumulative importance degree of the jth second vector, Bj is the storage time of the vector information of the jth second vector, t is the number of second vectors, B0 is the longest storage time among the storage times of the vector information of the t second vectors, and the time parameter includes Bj and B0.

[0217] In an example embodiment, the adjusting unit is further configured to arrange the second vectors in descending order of the influence parameters to obtain a vector sequence, and filter the second vectors to be stored from the vector sequence according to a difference between the usage parameter threshold and the usage parameter.

[0218] It should be noted that the above modules can be implemented by software or hardware, and for the latter, the following implementation manners can be used, but are not limited thereto: all the modules are located in the same processor; or the modules are located in different processors in any combination.

[0219] The features of the embodiments of the context vector generation apparatus can be referred to the related descriptions of the embodiments of the context vector generation method, which will not be repeated here.

[0220] Embodiments of the present application also provide an electronic device, Figure 6 is a schematic diagram of the electronic device according to the embodiments of the present application, as Figure 6 shown, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above context vector generation method embodiments.

[0221] In an example embodiment, the electronic device can further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0222] The specific examples in the embodiments can refer to the examples described in the above embodiments and example embodiments, which will not be repeated here.

[0223] Embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in any of the above context vector generation method embodiments when running.

[0224] In an example embodiment, the above computer readable storage medium can be, but is not limited to, a non-volatile computer readable storage medium.

[0225] In an example embodiment, the computer readable storage medium can include, but is not limited to, a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media capable of storing computer programs.

[0226] In an example embodiment, the computer readable storage medium can include, but is not limited to, a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media capable of storing computer programs.

[0227] Embodiments of the present application also provide a computer program product, including a computer program, the computer program being executed by a processor to implement the steps of the method in the embodiments of the present application; the computer program product further includes a non-volatile computer readable storage medium, the non-volatile computer readable storage medium storing the computer program, the computer program being executed by the processor to implement the steps of the method for generating a context vector in the embodiments of the present application.

[0228] The skilled person can further realize that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0229] The above describes in detail a method and device for generating a context vector provided by the present application. The principles and implementation modes of the present application are described by applying specific examples. The above description of the examples is only applicable to help understand the method and core idea of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A method for generating context vectors, characterized in that, An accelerator card applied to an inference device, the method comprising: Receive the target vector; A first sub-vector of the target vector is generated based on the first vector information, and a second sub-vector generated by the processor of the inference device is obtained. The first vector information is stored in a first storage space on the accelerator card. The first vector information is used to indicate a first historical vector received in the past. The first sub-vector is used to indicate the relationship between the target vector and the first historical vector. The second sub-vector is used to indicate the relationship between the target vector and the second historical vector. The second vector information is stored in a second storage space on the processor. The second vector information is used to indicate a second historical vector received in the past. The influence of the first historical vector on the target vector is higher than the influence of the second historical vector on the target vector. Generate the context vector of the target vector based on the first sub-vector and the second sub-vector.

2. The method according to claim 1, characterized in that, The step of generating the first sub-vector of the target vector based on the first vector information includes: The first sub-vector is generated based on the first vector information, and a first fusion factor of the first sub-vector is generated based on the first vector information. The first fusion factor is used to indicate the fusion weight of the first sub-vector and to fuse the first sub-vector and the second sub-vector to generate the context vector.

3. The method according to claim 2, characterized in that, The step of generating the first fusion factor of the first sub-vector based on the first vector information includes: Calculate the dot product between the query vector of the target vector and the key vector of each first vector to obtain multiple first dot product values ​​that correspond one-to-one with each first vector. The first vector includes the first historical vector and the target vector, and the first vector information includes the key vector of the first historical vector. Calculate the quotient between each of the first dot product values ​​and the square root of the vector dimension of the query vector to obtain multiple first intermediate values; The first fusion factor is generated based on each of the first intermediate values.

4. The method according to claim 3, characterized in that, The step of generating the first fusion factor based on each of the first intermediate values ​​includes: The first fusion factor is generated using the following formula: ， Where LSE1 is the first fusion factor, t1 is the number of the first vectors, C(x) is the first intermediate value corresponding to the xth first vector, and M1 is the largest first intermediate value.

5. The method according to claim 1, characterized in that, The step of obtaining the second sub-vector generated by the processor of the inference device includes: Obtain the second sub-vector generated by the processor of the inference device, and obtain the second fusion factor of the second sub-vector generated by the processor of the inference device, wherein the second fusion factor is used to indicate the fusion weight of the second sub-vector, and the second fusion factor is used to fuse the first sub-vector and the second sub-vector to generate the context vector.

6. The method according to claim 5, characterized in that, The step of obtaining the second sub-vector generated by the processor of the inference device and obtaining the second fusion factor of the second sub-vector generated by the processor of the inference device includes: Send the query vector of the target vector to the processor; The processor receives the second sub-vector and the second fusion factor sent by the processor in response to the query vector. The processor is configured to filter a reference historical vector from multiple second historical vectors based on the query vector and the second vector information, generate the second sub-vector based on the third vector information corresponding to the reference historical vector and the query vector, and generate the second fusion factor based on the third vector information corresponding to the reference historical vector and the query vector. The cumulative importance of the reference historical vector is greater than an importance threshold. The cumulative importance is the cumulative value of the correlation between each second historical vector and itself, and the correlation between each second historical vector and the input vector received after each second historical vector. The second vector information includes the third vector information.

7. The method according to claim 6, characterized in that, The step of generating the second fusion factor based on the third vector information corresponding to the reference historical vector and the query vector includes: Calculate the dot product of the query vector and the key vector of each of the reference history vectors to obtain a plurality of second dot product values ​​that correspond one-to-one with each of the reference history vectors, wherein the third vector information includes the key vector of the reference history vector; Calculate the quotient between each of the second dot product values ​​and the square root of the vector dimension of the query vector to obtain multiple second intermediate values; The second fusion factor is generated based on each of the second intermediate values.

8. The method according to claim 7, characterized in that, The step of generating the second fusion factor based on each of the second intermediate values ​​includes: The second fusion factor is generated using the following formula: ， Wherein, LSE2 is the second fusion factor, t2 is the number of reference history vectors, C(y) is the second intermediate value corresponding to the y-th reference history vector, and M2 is the largest second intermediate value.

9. The method according to claim 6, characterized in that, The step of filtering reference historical vectors from multiple second historical vectors based on the query vector and the second vector information includes: The relevance of each second historical vector to the target vector is calculated based on the query vector and the key vectors of each second historical vector, wherein the second vector information includes the key vectors of the second historical vectors; The cumulative importance of each second historical vector is updated based on the correlation between each second historical vector and the target vector; The reference historical vector is obtained by filtering out the second historical vectors whose cumulative importance is greater than the importance threshold from a plurality of second historical vectors.

10. The method according to claim 6, characterized in that, The step of generating the second sub-vector based on the third vector information corresponding to the reference historical vector and the query vector includes: The correlation degree between each reference historical vector and the target vector is calculated based on the query vector and the key vector of each reference historical vector, wherein the third vector information includes the key vector and the value vector of the reference historical vector; The second sub-vector is obtained by summing the products of the correlation degree between each of the reference historical vectors and the target vector and the corresponding value vector of the reference historical vector.

11. The method according to claim 1, characterized in that, The step of generating the context vector of the target vector based on the first sub-vector and the second sub-vector includes: Obtain a first fusion factor for the first sub-vector and a second fusion factor for the second sub-vector, wherein the first fusion factor is used to indicate the fusion weight of the first sub-vector and the second fusion factor is used to indicate the fusion weight of the second sub-vector; The context vector is generated using the following formula: ， Where O is the context vector, O1 is the first sub-vector, O2 is the second sub-vector, LSE1 is the first fusion factor, and LSE2 is the second fusion factor.

12. The method according to claim 1, characterized in that, After receiving the target vector, the method further includes: Store the target vector information of the target vector into the first storage space; Based on the usage parameters, vector parameters, and time parameters of the first storage space, the vector information stored in the first storage space is adjusted to obtain the first storage space storing the first vector information, and the vector information stored in the second storage space is obtained to obtain the second storage space storing the second vector information. The usage parameters are used to indicate the occupancy level of the first storage space, the vector parameters are used to indicate the correlation between the vector corresponding to the vector information currently stored in the first storage space and the target vector, and the time parameters are used to indicate the storage time of the vector information currently stored in the first storage space.

13. The method according to claim 12, characterized in that, The step of adjusting the vector information stored in the first storage space to obtain a first storage space storing the first vector information, and adjusting the vector information stored in the second storage space to obtain a second storage space storing the second vector information, based on the usage parameters, vector parameters, and time parameters of the first storage space, includes: Compare the usage parameters with the usage parameter thresholds; When the usage parameter is greater than or equal to the usage parameter threshold, the vector information of a portion of the multiple second vectors is transferred to the second storage space according to the vector parameter and the time parameter, wherein the second vector is the vector whose current vector information is stored in the first storage space.

14. The method according to claim 13, characterized in that, The step of transferring vector information of a portion of multiple second vectors to the second storage space according to the vector parameters and the time parameters includes: The relevance between each second vector and the target vector is calculated based on the query vector of the target vector and the key vectors of each second vector to obtain the vector parameters, wherein the vector information of the second vector includes the key vector of the second vector, and the target vector information includes the query vector of the target vector; The cumulative importance of each second vector is updated according to the vector parameters, wherein the cumulative importance is the cumulative value of the correlation between each second vector and itself and the correlation between each second vector and the input vector received after each second vector; The influence parameter of each second vector is calculated based on the cumulative importance of each second vector and the time parameter, wherein the influence parameter is used to indicate the influence of each second vector on the target vector; Vectors to be transferred are selected from a plurality of second vectors based on the influence parameters of each second vector, wherein the plurality of second vectors include the vectors to be transferred and non-transfer vectors, the influence parameters of the vectors to be transferred are less than or equal to the influence parameters of the non-transfer vectors, and the degree of occupancy of the vector information of the non-transfer vectors on the first storage space is less than the degree of occupancy of the first storage space indicated by the usage parameter threshold. After generating the first sub-vector of the target vector based on the first vector information and obtaining the second sub-vector generated by the processor of the inference device, the vector information of the vector to be transferred is transferred to the second storage space.

15. The method according to claim 14, characterized in that, The step of calculating the influence parameter of each second vector based on the cumulative importance of each second vector and the time parameter includes: The influence parameter is calculated using the following formula: ， Wherein, Scorej is the influence parameter of the j-th second vector, Aj is the cumulative importance of the j-th second vector, Bj is the storage time of the vector information of the j-th second vector, t is the number of second vectors, and B0 is the longest storage time among the storage times of the vector information of t second vectors. The time parameters include Bj and B0.

16. The method according to claim 14, characterized in that, The step of selecting the vector to be transferred from multiple second vectors based on the influence parameter of each second vector includes: Arrange the second vectors in descending order of the influence parameters to obtain a vector sequence; The vectors to be transferred are selected from the vector sequence based on the difference between the usage parameter threshold and the usage parameter.

17. A context vector generation apparatus, characterized in that, An accelerator card for use in inference devices, the device comprising: The receiving module is used to receive the target vector; A first generation module is configured to generate a first sub-vector of the target vector based on first vector information, and obtain a second sub-vector generated by the processor of the inference device. The first vector information is stored in a first storage space on the accelerator card, indicating a first historical vector received previously. The first sub-vector indicates the relationship between the target vector and the first historical vector. The second sub-vector indicates the relationship between the target vector and a second historical vector. The second vector information is stored in a second storage space on the processor, indicating a second historical vector received previously. The influence of the first historical vector on the target vector is higher than the influence of the second historical vector on the target vector. The second generation module is used to generate a context vector of the target vector based on the first sub-vector and the second sub-vector.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 16.

19. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 16.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 16.

Images