Model inference method, electronic device, storage medium, and program product

Abstract

Embodiments of the present application provide a model inference method, an electronic device, a storage medium and a program product. The method comprises: a first processor performing first attention calculation on a current token by using first key-value pair data of at least one first historical token to generate a first calculation result; a second processor performing second attention calculation on the current token by using second key-value pair data of at least one second historical token to generate a second calculation result; and inferring target context information of the current token according to the first calculation result and the second calculation result. The technical solution of the embodiments of the present application cooperates two heterogeneous processors to perform attention calculation, thereby accelerating the attention calculation to quickly determine the context information that the current token should pay attention to, and further improve the model inference speed.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model reasoning method, electronic device, storage medium, and program product. Background Technology

[0002] Large Language Models (LLMs), often shortened to large models, are deep learning models trained on massive amounts of text data. Currently, LLMs have made significant progress across various fields, and their supported context windows (context lengths) are constantly expanding. For example, existing LLMs can support context lengths of 128K or even 200K. Larger context windows allow models to retain more text in their historical memory, thus better tracking important moments and details in long dialogues, lengthy documents, or complex coding tasks. While longer context support is widely used in fields such as text summarization, it also brings many challenges to the inference process. For example, longer contexts can lead to a significant increase in model inference time, thus degrading the user experience. Summary of the Invention

[0003] This application provides a model reasoning method, an electronic device, a computer-readable storage medium, and a computer program product to alleviate or solve one or more technical problems existing in the prior art.

[0004] In a first aspect, embodiments of this application provide a model inference method applied to a first processor. The method includes: performing a first attention calculation on a current word using first key-value pair data of at least one first historical word to generate a first calculation result; obtaining a second calculation result, wherein the second calculation result is generated by the second processor performing a second attention calculation on the current word using second key-value pair data of at least one second historical word; and inferring target context information of the current word based on the first calculation result and the second calculation result.

[0005] Secondly, embodiments of this application provide a model inference method applied to a second processor. The method includes: performing a second attention calculation on a current word using second key-value pair data of at least one second historical word, generating a second calculation result; and sending the second calculation result to a first processor so that the first processor can infer the target context information of the current word based on the first calculation result and the second calculation result, wherein the first calculation result is generated by the first processor performing a first attention calculation on the current word using first key-value pair data of at least one first historical word.

[0006] Thirdly, embodiments of this application provide a model inference method, comprising: obtaining a first calculation result and a second calculation result, wherein the first calculation result is generated by a first processor performing attention calculation on the current word using first key-value pair data of at least one first historical word, and the second calculation result is generated by a second processor performing attention calculation on the current word using second key-value pair data of at least one second historical word; and inferring the target context information of the current word based on the first calculation result and the second calculation result.

[0007] Fourthly, embodiments of this application provide a model reasoning method, including: reasoning about the current word based on target context information of the current word, wherein the target context information is determined according to any one of the methods of embodiments of this application.

[0008] Fifthly, embodiments of this application provide an electronic device, including: a first processor for executing the method of any one of the first aspects or the method of any one of the fourth aspects; and a second processor for executing the method of any one of the second aspects.

[0009] Sixthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0010] In a seventh aspect, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0011] Eighthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements any of the methods described in the embodiments of this application.

[0012] According to the technical solution of this application embodiment, among the candidate historical words to be processed by the first processor, a portion of words (second historical words) are selected and unloaded to the second processor. The first processor processes a portion of the words (first historical words), and the second processor processes another portion of the words (second historical words). The first processor compensates and merges the attention calculation results of the two portions (first calculation result and second calculation result) to obtain the target context information that the current word needs to pay attention to. This facilitates subsequent prediction and inference of the current word based on the target context information, thereby reducing the bandwidth bottleneck in the model decoding stage, accelerating attention calculation, quickly determining the most important context information for the current word, and improving the model inference speed.

[0013] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0014] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0015] Figure 1 An architecture diagram of an electronic device for implementing the model reasoning method of Embodiment 1 of this application is shown;

[0016] Figure 2 A flowchart illustrating the model reasoning method of Embodiment 1 of this application is shown;

[0017] Figure 3 A flowchart illustrating the model reasoning method of Embodiment 1 of this application is shown;

[0018] Figure 4A and Figure 4B This diagram illustrates an application example of the model reasoning method according to Embodiment 1 of this application.

[0019] Figure 5 A flowchart illustrating the model reasoning method of Embodiment 2 of this application is shown;

[0020] Figure 6 A flowchart illustrating the model reasoning method of Embodiment 3 of this application is shown;

[0021] Figure 7 A block diagram of an electronic device provided in Embodiment 4 of this application is shown. Detailed Implementation

[0022] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0023] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0024] Application scenarios

[0025] In large models and Natural Language Processing (NLP), tokens are the basic units that make up text. These units can be words, phrases, words, symbols, or any other form of text fragment. The definition of a token may vary in different scenarios, but they are generally the fundamental units for text analysis and processing.

[0026] Large models based on the autoregressive (Transformer) architecture are neural network architectures based on the attention mechanism. Attention computation in the Transformer architecture mainly includes: transforming the input sequence into query vectors, key vectors, and value vectors through a linear transformation; calculating the dot product of the query vector and all key vectors to generate an attention score matrix P; normalizing the attention score matrix P using a softmax function to generate an attention weight matrix S; and finally, weighted summing the attention weight matrix S with all value vectors to obtain the output vector. To reduce computational operations, large models cache the generated key and value vectors, i.e., a key-value cache.

[0027] The large model in the Transformer architecture mainly consists of two stages during inference: the prefill stage and the decoding stage. The prefill stage refers to the stage where the large model prefills or prepares contextual information before generating text. This includes introducing specific cues, background information, or data to enable the model to more effectively generate content relevant to that context. For example, the large model performs attention calculations based on the input token sequence to obtain the probability distribution of each token. The key-value data (KV data, composed of key vectors and value vectors) of each token generated during the attention calculation process is cached for use in the decoding stage. The decoding stage is the text generation stage. The large model generates coherent and grammatically correct text based on the existing contextual information. For example, the large model obtains the KV data of historical tokens (i.e., already generated tokens) and performs attention calculations on the current token to determine the contextual representation of the current token, i.e., the most important contextual information for the current token, and further predicts the probability distribution of the current token.

[0028] When large models process inference with long sequences of tokens (such as thousands or even tens of thousands of tokens), the amount of key-value cache data that needs to be stored will also increase accordingly. This will affect the speed of attention computation, leading to a decrease in inference speed and consuming a large amount of memory.

[0029] In the decoding phase, due to the need to retrieve KV cache data and the significant time spent waiting for data to be loaded from memory, the performance of the computational task in this phase is primarily limited by memory access speed. In other words, the latency in the decoding phase mainly consists of memory access time. Current popular large-scale model architectures, when applied to long-context scenarios, show that for a single query attention model, the proportion of matrix multiplication (linear) time in the decoding phase is approximately 10.8%, and the proportion of attention computation time is approximately 89.2%; for another single query attention model, the proportion of matrix multiplication time in the decoding phase is approximately 47.1%, and the proportion of attention computation time is approximately 52.9%. It is evident that in long-context scenarios, the proportion of attention computation time in the decoding phase is relatively high. This is because the matrix multiplication time remains constant as historical tokens increase, while the attention computation time increases linearly. Therefore, optimizing attention computation time is crucial for accelerating large-scale model inference in long-context scenarios.

[0030] The embodiments of this application aim to provide a model inference method for determining contextual information. By using two heterogeneous processors (a first processor and a second processor) to collaboratively perform attention calculation, attention calculation is accelerated to quickly determine the contextual information that the current token should pay the most attention to, thereby improving the model inference speed.

[0031] This technical solution can be applied to model inference scenarios, such as document summarization based on large models, code analysis scenarios, multi-turn dialogue scenarios, machine translation based on Transformer machine translation models, automatic speech recognition based on Transformer speech recognition models, and image caption generation based on Transformer multimodal models.

[0032] It should be noted that the application scenarios or examples provided in the embodiments of this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solutions. In addition, all information and data involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0033] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0034] Embodiment one

[0035] Figure 1 An architectural diagram of an electronic device for implementing the model inference method of embodiments of this application is shown. Figure 1 As shown, the electronic device includes a first processor 101 and a second processor 102.

[0036] For example, the first processor 101 and the second processor 102 can work collaboratively through an interactive process. The first processor 101 may be a processor capable of executing a large number of computational tasks in parallel, used to perform most of the computational tasks in the model inference process. The first processor 101 is, for example, a graphics processing unit (GPU) or an application-specific integrated circuit (ASIC). The second processor 102 may be a processor of a different type from the first processor 101. The second processor 102 is responsible for managing the operating system, processing model input requests and model output requests, and performing tasks related to the user interface. The second processor 102 is, for example, a central processing unit (CPU) or other general-purpose processor.

[0037] Taking the first processor 101 as a GPU and the second processor 102 as a CPU as an example, the CPU performs computational processing based on CPU memory, which provides memory space for the CPU and can be Random Access Memory (RAM). The GPU performs computational processing based on GPU memory, which provides memory space for the GPU and can be dedicated memory, such as a graphics card.

[0038] In this embodiment, the first processor 101 and the second processor 102 are used to collaboratively perform attention calculations during the model inference process. For example, in the pre-filling stage of model inference, the first processor 101 caches the key-value pair data of the generated tokens, for example, in its memory space. Further, the first processor 101 selects at least one token from the candidate historical tokens (i.e., the generated tokens) as a second historical token, and asynchronously transmits the key-value pair data of the second historical token (i.e., the second key-value pair data) to the second processor 102. The remaining candidate historical tokens can be used as the first historical tokens. In the encoding stage of model inference, the first processor 101 uses the key-value pair data of the first historical token (i.e., the first key-value pair data) to perform a first attention calculation on the current token, generating a first calculation result; the second processor 102 uses the second key-value pair data to perform a second attention calculation on the current token, generating a second calculation result. The first and second attention calculations are executed in parallel, thereby accelerating the attention calculation process. Furthermore, the first processor 101 merges the first calculation result and the second calculation result to generate the context representation of the current token, that is, the context information (target context information) that the current token should focus on. The first processor 101 uses the target context information of the current token to infer the probability distribution of the current token and outputs the predicted current token.

[0039] The following is a detailed introduction. Figure 2 A flowchart illustrating a model inference method according to an embodiment of this application is provided. This model inference method can be applied to a first processor, for example, by... Figure 1 The first processor 101 shown executes. (As shown) Figure 2 As shown, the method includes:

[0040] Step S201: Use the first key-value pair data of at least one first historical token to perform the first attention calculation on the current token and generate the first calculation result;

[0041] Step S202: Obtain the second calculation result, wherein the second calculation result is generated by the second processor using the second key-value pair data of at least one second historical token to perform a second attention calculation on the current token;

[0042] Step S203: Based on the first calculation result and the second calculation result, infer the target context information of the current token.

[0043] During the pre-filling phase of model inference, the first processor 101 caches the key-value pair data of the generated tokens, for example, by caching it in the memory space of the first processor 101. Both the first historical token and the second historical token are selected from the candidate historical tokens, i.e., the generated tokens. In one example, all candidate historical tokens can be divided into a set 1 of first historical tokens and a set 2 of second historical tokens. In another example, a sparsity operation can be performed on all candidate historical tokens, i.e., a portion of the candidate historical tokens can be deleted, and then the remaining candidate historical tokens can be divided into set 1 and set 2.

[0044] In this embodiment, the key-value pairs of each first historical token in set 1 are referred to as the first key-value pairs, which include the key vector and value vector of each first historical token; the key-value pairs of each second historical token in set 2 are referred to as the second key-value pairs, which include the key vector and value vector of each second historical token. It should be noted that this application does not impose a specific limit on the number of first historical tokens in set 1 or second historical tokens in set 2.

[0045] For example, the Query vector, Key vector, and Value vector of an attention head in the model's attention layer are represented as Q∈R. 1×n K∈R t×n and V∈R t×n Here, R represents the set of real numbers, t represents the number of candidate historical tokens, and n represents the dimension of the attention head. The number of first historical tokens in set 1 is t_g, where "g" represents the portion of the first attention calculation performed by the first processor. The number of second historical tokens in set 2 is t_c, where "c" represents the portion of the second attention calculation performed by the second processor. Taking the absence of sparse operations as an example, t = t_c + t_g.

[0046] During the decoding phase of model inference, each attention layer of the model uses cached key-value pair data to perform attention calculations on the current token, resulting in an output variable O. c,g The output variable O c,g This can represent the target context information that the current token needs to focus on from the candidate historical tokens, and then predict and infer the current token based on the target context information. Output variable O c,g The calculation formula is as follows:

[0047]

[0048] S c,g =[S c ,Sg ],O c,g =S c,g ·V c,g ∈R 1×n

[0049]

[0050] Where Q represents the Query vector of the current token; This represents the total key vector matrix of candidate historical tokens, including the key vector matrix from the first key-value pair data and the key vector matrix from the second key-value pair data; V c,g This represents the total Value vector matrix of the candidate historical tokens, including the Value vector matrix V from the first key-value pair data. g Value vector matrix V in the second key-value pair data c .

[0051] P c,g The total attention score matrix representing the candidate historical tokens can be obtained by concatenating the first attention score matrix P. g Second attention score matrix P c We obtain the first attention score matrix P. g The second attention score matrix P is obtained during the first attention calculation performed by the first processor. c This is obtained during the second attention calculation performed by the second processor. The softmax function is used to evaluate the total attention score matrix P. c,g After normalization, the total attention weight matrix S is obtained. c,g S c,g The first attention weight matrix S, including the first attention calculation part g The second attention weight matrix S in the second attention calculation part c .

[0052] Since the first processor only possesses the first attention score matrix P obtained from the first attention calculation... g The second processor only possesses the second attention score matrix P obtained from the second attention calculation. c Therefore, the normalization step using the softmax function needs to be obtained separately in the first attention calculation and the second attention calculation to avoid additional communication overhead between the first and second processors. That is, the first processor obtains the first attention weight matrix S based on the first attention calculation. g The second processor calculates the second attention weight matrix S based on the second attention. c .

[0053] Wherein, the second attention weight matrix Sc The calculation formula is as follows:

[0054]

[0055] Total attention score matrix P c,g The maximum attention value P in max That is, the maximum value of the first attention score P. g,max The second attention score maximum P c,max The maximum value in the range. Among them, the maximum value of the first attention score P. g,max Represents the first attention score matrix P g The maximum value in the second attention score, P. c,max Represents the second attention score matrix P c The maximum value in sum. g The sum represents the first normalization factor, which is the sum of the exponents of the attention scores obtained from the first attention calculation. c This represents the second normalization factor, which is the sum of the exponents of the attention scores obtained from the second attention calculation.

[0056] Similarly, the first attention weight matrix S g The calculation formula is as follows:

[0057]

[0058] In Example 1, the first processor performs the first attention calculation and obtains the first calculation result O. g It can be represented as: O g =softmax(P g )·V g The second calculation result O obtained by the second processor performing the second attention calculation c It can be represented as: O c =softmax(P c )·V c The first calculation result is O. g The first context information, namely the context information provided by the first set of historical tokens 1 for the current token, is used to represent the second context information, namely the context information provided by the second set of historical tokens 2 for the current token.

[0059] Then, in step S203, the first processor needs to perform compensation calculations based on the first calculation result and the second calculation result, that is, respectively, the first calculation result O g Second calculation result O cThe attention score, which was not considered during the normalization calculation, is compensated to obtain the target context information of the current token. For example, the target context information is calculated based on the first calculation result and its corresponding first coefficient, and the second calculation result and its corresponding second coefficient.

[0060] Wherein, the first coefficient Xg is used to represent the first context information (the first calculation result O). g In the target context information O c,g The weight of the component is calculated using the following formula:

[0061]

[0062] The second coefficient Xc is used to represent the second contextual information (the first calculation result O). c In the target context information O c,g The weight of the component is calculated using the following formula:

[0063]

[0064] Furthermore, the target context information, i.e., the output variable O c,g The calculation can be expressed as:

[0065]

[0066] In Example 2, the first processor performs the first attention calculation and obtains the first calculation result O. g It can be represented as:

[0067]

[0068] The second processor performs the second attention calculation, resulting in the second calculation result O. c It can be represented as:

[0069]

[0070] Then, in step S203, based on the first calculation result and the second calculation result, the target context information O of the current token is inferred. c,g , can be represented as:

[0071]

[0072] According to the technical solution of this application embodiment, during attention calculation in the model decoding stage, some tokens (second historical tokens) are selected from the candidate historical tokens to be processed by the first processor and offloaded to the second processor. The first processor processes one part of the tokens (first historical tokens), and the second processor processes the other part of the tokens (second historical tokens). The first processor then compensates and merges the attention calculation results of the two parts (first calculation result and second calculation result) to obtain the target context information that the current token needs to focus on, which facilitates subsequent prediction and inference of the current token based on the target context information. Taking a GPU as the first processor and a CPU as the second processor as an example, although the CPU has a large gap in computing power compared to the GPU, the gap in memory bandwidth is much smaller. Therefore, the method of using the first processor and the second processor to perform attention calculation in collaboration in this application embodiment can reduce the bandwidth bottleneck in the model decoding stage, accelerate attention calculation, quickly determine the context information that the current token should focus on, and improve the model inference speed.

[0073] In one implementation, in step S201, a first attention calculation is performed on the current token using first key-value pair data of at least one first historical token to generate a first calculation result. This may include: determining query information for the current token; the query information may be used by a second processor to generate a second calculation result based on second key-value pair data and query information; and generating the first calculation result based on first key-value pair data and query information.

[0074] Here, the query information for the current token is its Query vector Q. For example, during the model's pre-filling stage, the first processor transforms the embedding vector of the current token through a linear projection transformation of the attention layer to generate the corresponding Query vector Q. The first processor is based on the first key-value pair data (including K... g and V g The query vector Q generates the first calculation result O. g .

[0075] In one example, the first processor can send the query information to the second processor, so that the second processor can generate a second calculation result based on the second key-value pair data and the query information. For example, during the model decoding phase, the first processor sends the query vector Q of the current token to the second processor. The second processor receives the query vector Q and, based on the second key-value pair data (including K... c and V c The second calculation result O is generated from the query vector Q. cIn other words, the second processor can utilize the Query vector Q calculated by the first processor during the pre-filling stage, without having to calculate it itself, thus avoiding a waste of computing resources.

[0076] In one implementation, the first calculation result represents first context information provided by at least one first historical token to the current token, and the second calculation result represents second context information provided by at least one second historical token to the current token. Further, in step S203, inferring the target context information of the current token based on the first and second calculation results may include: calculating the target context information based on the first context information and its corresponding first coefficient, and the second context information and its corresponding second coefficient, wherein the first coefficient represents the weight of the first context information in the target context information, and the second coefficient represents the weight of the second context information in the target context information.

[0077] For a specific example, refer to Example 2 above. In Example 2, the first processor and the second processor only calculate their respective parts of the context information. That is, the first processor generates a first calculation result, and the second processor generates a second calculation result. The second calculation result is then transmitted to the first processor, which compensates for the attention score portion not involved in the first attention calculation process for the first calculation result, and compensates for the attention score portion not considered in the second attention calculation process for the second calculation result. These results are then merged to obtain the target context information of the current token, thereby improving the accuracy of the context information and thus improving the accuracy of the model's inference results. For example, when the first processor is used as a computationally accelerated processor (e.g., GP), it has better computational advantages. Therefore, compensation and merging by the first processor can further accelerate attention calculation.

[0078] In one implementation, step S203, inferring the target context information of the current token based on the first calculation result and the second calculation result, may further include: calculating a first coefficient based on the maximum value of the first attention score and the first normalization factor in the first attention calculation; obtaining the maximum value of the second attention score and the second normalization factor in the second attention calculation; and calculating a second coefficient based on the maximum value of the second attention score and the second normalization factor.

[0079] In Example 2, the maximum value of the first attention score P g,max Represents the first attention score matrix P g The maximum value in the second attention score, P. c,max Represents the second attention score matrix P c The maximum value in sum; gThe sum represents the first normalization factor, which is the sum of the exponents of the attention scores obtained from the first attention calculation. c This represents the second normalization factor, which is the sum of the exponents of the attention scores obtained from the second attention calculation. The first coefficient Xg is used to represent the first contextual information (the first calculation result O). g In the target context information O c,g The weight of the first calculation result; the second coefficient Xc is used to represent the second context information (the first calculation result O). c In the target context information O c,g The weight it occupies in the middle.

[0080] During the collaborative computation between the first and second processors, the second processor only needs to provide the second computation result O. c The maximum value of the second attention score P c,max and the second normalization factor sum c The first processor compensates for and merges the output variables (the target context information O of the current token). c,g In the process of calculating the first coefficient Xg and the second coefficient Xc respectively, the collaborative computing capability can be further improved, thereby further accelerating the attention computing.

[0081] For example, during the model decoding phase, the first processor transmits the query vector Q of the current token to the second processor, and the second processor transmits the second calculation result O to the first processor. c The maximum value of the second attention score P c,max and the second normalization factor sum c In a specific LLM, the required transmission size for each LLM layer is approximately 20.31 kilobytes (KiB) per batch. The bandwidth for data transmission between the first and second processors is approximately 24–32 gigabytes per second (GB / s). Therefore, in this embodiment, the time required for data transmission between the first and second processors is only 0.84 microseconds (μs), which is negligible compared to the inference time of hundreds of microseconds to milliseconds for a single LLM layer. Thus, the time overhead of data transmission between the first and second processors does not affect the efficiency of attention computation. As an example, data transmission between the first and second processors can be performed in the background using asynchronous events, i.e., by creating streams and assigning transmission tasks to the corresponding streams.

[0082] In one embodiment, before step S201, the method of this application embodiment may further include: obtaining second key-value pair data from the memory space of the first processor; sending the second key-value pair data to the second processor; and after deleting the second key-value pair data from the memory space, obtaining first key-value pair data from the memory space of the first processor.

[0083] The first processor's memory space is used to cache key-value pair data, including key-value pair data (key vector and value vector) of generated tokens (candidate historical tokens). After determining the second set of historical tokens 2, the first processor transmits the key-value pair data corresponding to set 2 (i.e., the second key-value pair data) to the second processor. After the transmission, the second key-value pair data is deleted from the memory space, and what remains is the first key-value pair data.

[0084] In this embodiment of the application, before the encoding stage of the model begins, the first processor asynchronously transmits the second key-value pair data to the second processor. This can offload some of the attention calculations for the tokens to the second processor, which can accelerate the attention calculations on the one hand and reduce the memory space occupied by the first processor on the other.

[0085] In one embodiment, before step S201, the method of this application embodiment may further include: selecting a preset proportion of candidate historical tokens as second historical tokens from a plurality of candidate historical tokens of the current token, wherein the preset proportion is set based on the memory bandwidth of the first processor and the memory bandwidth of the second processor.

[0086] For example, the memory bandwidth of the first processor is A1, and the memory bandwidth of the second processor is A2. The preset ratio can be A2 / (A1+A2). As in the previous example, the number of candidate historical tokens is t, the number of second historical tokens in set 2 is tc, which is t*A2 / (A1+A2), and the number of first historical tokens in set 1 is tg, which is t-tc. Here, candidate historical tokens can be all tokens in the memory space, or a subset of tokens obtained by performing a sparse operation on all tokens in the memory space.

[0087] Based on this, the number of tokens processed by the first processor and the second processor can be allocated according to their memory bandwidth, thereby maximizing the utilization of their bandwidth resources, ensuring load balance, avoiding processing delays caused by insufficient bandwidth of one side, and achieving more efficient collaborative computing.

[0088] Figure 3 A flowchart illustrating a model inference method according to an embodiment of this application is provided. This model inference method can be applied to a second processor, such as one... Figure 1The second processor 102 shown executes the commands. For example... Figure 3 As shown, the method includes:

[0089] Step S301: Use the second key-value pair data of at least one second historical token to perform a second attention calculation on the current token, and generate a second calculation result;

[0090] Step S302: Send the second calculation result to the first processor so that the first processor can infer the target context information of the current token based on the first calculation result and the second calculation result, wherein the first calculation result is generated by the first processor using the first key-value pair data of at least one first historical token to perform a first attention calculation on the current token.

[0091] In one implementation, step S301 may include: obtaining query information of the current token from the first processor; and generating a second calculation result based on the second key-value pair data and the query information.

[0092] In one embodiment, the method of this application may further include: sending the maximum value P of the second attention score in the second attention calculation to the first processor. c,max Second normalization factor sum c .

[0093] For details, please refer to the relevant description of the execution process of the first processor, which will not be repeated here.

[0094] Figure 4A and Figure 4B This diagram illustrates an application example of the model inference method according to an embodiment of this application. In this application example, the first processor is exemplified as a GPU, and the second processor is exemplified as a CPU. Figure 4A The flow chart for the pre-filling stage is shown. Figure 4B This illustrates the process of the coding phase.

[0095] like Figure 4AAs shown, the pre-filling stage of the model includes: (1) Pre-filling stage start: The GPU performs initialization operations at the start of pre-filling; (2) QKV mapping (Projection) and Rotary Position Embedding (RoPE): The GPU generates QKV data of the input tokens (including the Query vector, Key vector and Value vector of each input token) through projection linear transformation, adds position encoding to each input token, and saves KV cache data at the same time; The GPU asynchronously transmits KV cache data to the CPU. That is, after the KV cache data is calculated, it first checks whether the number of second historical tokens unloaded to the CPU reaches the preset ratio. If the preset ratio is not reached, it asynchronously transmits KV cache data to the CPU. Furthermore, the GPU checks whether there are any transmitted second historical tokens in the CPU. If there are, it deletes the corresponding second historical tokens in the GPU memory space; The CPU copies KV cache data through memory operations; (3) Attention calculation: The GPU and CPU start collaborative attention calculation; (4) Output mapping and FFN (Feed-Forward) The network (feedforward neural network) maps the result of collaborative attention computation (target attention information) and inputs it into the FFN for subsequent token prediction. Then, the GPU performs the computation of the next attention layer, including: QKV mapping and position encoding, attention computation and output mapping, and FFN.

[0096] like Figure 4B As shown, the encoding stage of the model includes: (1) QKV mapping and positional encoding; (2) Attention calculation: the GPU performs the first attention calculation and transmits Q (i.e., the query vector of the current token) to the CPU; the CPU performs the second attention calculation, thereby performing CPU and GPU attention collaborative calculation; the calculation results are copied, that is, the CPU transmits the calculation results to the GPU, including the second calculation result O. c The maximum value of the second attention score P c,max and the second normalization factor sum c The GPU performs compensation and merging calculations, such as based on the first calculation result O. g Second calculation result O c The maximum value of the second attention score P c,max and the second normalization factor sum c Calculate target context information O c,g .

[0097] It should be noted that the inference process of the model includes a pre-filling stage and an encoding stage. The model is usually a multi-layered structure, such as including multiple attention layers. The figure shows the inference process for each attention layer. Therefore, it can be understood that the pre-filling stage and the encoding stage of different layers may overlap.

[0098] Embodiment two

[0099] Figure 5 A flowchart illustrating a model inference method according to an embodiment of this application is provided. This model inference method can be applied to a third processor, which is a processor distinct from the first and second processors, such as a CPU, GPU, or other type of processor. The method includes:

[0100] Step S501: Obtain a first calculation result and a second calculation result, wherein the first calculation result is generated by the first processor using the first key-value pair data of at least one first historical token to perform attention calculation on the current token, and the second calculation result is generated by the second processor using the second key-value pair data of at least one second historical token to perform attention calculation on the current token;

[0101] Step S502: Based on the first calculation result and the second calculation result, infer the target context information of the current token.

[0102] The generation methods of the first and second calculation results can be found in Embodiment 1. Step S502 can be executed in the same or similar manner as step S203, such as by a third processor: calculating target context information based on the first context information and its corresponding first coefficient, and the second context information and its corresponding second coefficient, wherein the first coefficient represents the weight of the first context information in the target context information, and the second coefficient represents the weight of the second context information in the target context information; or, by a third processor: calculating the first coefficient based on the maximum value of the first attention score and the first normalization factor in the first attention calculation; obtaining the maximum value of the second attention score and the second normalization factor in the second attention calculation; and calculating the second coefficient based on the maximum value of the second attention score and the second normalization factor.

[0103] For example, before step S502, the third processor may also perform: obtaining the second key-value pair data from the memory space of the first processor; sending the second key-value pair data to the second processor; and after deleting the second key-value pair data from the memory space, obtaining the first key-value pair data from the memory space.

[0104] For example, before step S502, the third processor may also perform the following: select a preset proportion of candidate historical tokens from a plurality of candidate historical tokens of the current token as the second historical token, wherein the preset proportion is set based on the memory bandwidth of the first processor and the memory bandwidth of the second processor.

[0105] Embodiment three

[0106] Figure 6 A flowchart illustrating a model inference method according to an embodiment of this application is provided. This model inference method can be applied to a first processor, a second processor, or a third processor, and the method includes:

[0107] Step S601: Infer the current token based on the target context information of the current token, wherein the target context information is determined according to the method of either Embodiment 1 or Embodiment 2.

[0108] Embodiment four

[0109] Corresponding to the application scenarios and devices provided in the embodiments of this application, the embodiments of this application also provide a model inference device applied to a first processor. The device includes: a first calculation result generation module, used to perform a first attention calculation on the current word using first key-value pair data of at least one first historical word, to generate a first calculation result; a second calculation result acquisition module, used to acquire a second calculation result, wherein the second calculation result is generated by the second processor performing a second attention calculation on the current word using second key-value pair data of at least one second historical word; and a target context information inference module, used to infer the target context information of the current word based on the first calculation result and the second calculation result.

[0110] In one embodiment, the first calculation result generation module includes: a query information determination unit, configured to determine the query information of the current term; a query information sending unit, configured to send the query information to the second processor, so that the second processor can generate the second calculation result based on the second key-value pair data and the query information; and a first calculation result generation unit, configured to generate the first calculation result based on the first key-value pair data and the query information.

[0111] In one implementation, the first calculation result is used to represent the first context information provided by the at least one first historical word element to the current word element, and the second calculation result is used to represent the second context information provided by the at least one second historical word element to the current word element; the target context information reasoning module is specifically used to: calculate the target context information based on the first context information and its corresponding first coefficient and the second context information and its corresponding second coefficient, wherein the first coefficient is used to represent the weight of the first context information in the target context information, and the second coefficient is used to represent the weight of the second context information in the target context information.

[0112] In one implementation, the target context information reasoning module is specifically used to: calculate the first coefficient based on the first maximum attention score and the first normalization factor in the first attention calculation; obtain the second maximum attention score and the second normalization factor in the second attention calculation; and calculate the second coefficient based on the second maximum attention score and the second normalization factor.

[0113] In one embodiment, the apparatus further includes: a second key-value pair data acquisition module, configured to acquire the second key-value pair data from the memory space of the first processor before performing a first attention calculation on the current word using the first key-value pair data of at least one first historical word word to generate a first calculation result; a second key-value pair data sending module, configured to send the second key-value pair data to the second processor; and a first key-value pair data acquisition module, configured to acquire the first key-value pair data from the memory space after deleting the second key-value pair data from the memory space.

[0114] In one embodiment, the device further includes: a historical word filtering module, configured to select a preset proportion of candidate historical words as the second historical words from a plurality of candidate historical words of the current word before performing a first attention calculation on the current word using first key-value pair data of at least one first historical word and generating a first calculation result, wherein the preset proportion is set based on the memory bandwidth of the first processor and the memory bandwidth of the second processor.

[0115] Corresponding to the application scenarios and devices provided in the embodiments of this application, the embodiments of this application also provide a model inference device applied to a second processor. The device includes: a second calculation result generation module, used to perform a second attention calculation on the current word using second key-value pair data of at least one second historical word, to generate a second calculation result; and a second calculation result sending module, used to send the second calculation result to the first processor, so that the first processor can infer the target context information of the current word based on the first calculation result and the second calculation result, wherein the first calculation result is generated by the first processor performing a first attention calculation on the current word using first key-value pair data of at least one first historical word.

[0116] In one implementation, the second calculation result generation module is specifically used to: obtain query information of the current word from the first processor; and generate the second calculation result based on the second key-value pair data and the query information.

[0117] In one embodiment, the apparatus further includes a maximum value and a second normalization factor sending module, configured to send the maximum value of the second attention score and the second normalization factor in the second attention calculation to the first processor.

[0118] Corresponding to the application scenarios and devices provided in the embodiments of this application, the embodiments of this application also provide a model inference device, including: a first calculation result and a second calculation result acquisition module, used to acquire the first calculation result and the second calculation result, wherein the first calculation result is generated by a first processor performing attention calculation on the current word using first key-value pair data of at least one first historical word, and the second calculation result is generated by a second processor performing attention calculation on the current word using second key-value pair data of at least one second historical word; and a target context information determination module, used to infer the target context information of the current word based on the first calculation result and the second calculation result.

[0119] Corresponding to the application scenarios and devices provided in the embodiments of this application, the embodiments of this application also provide a model reasoning device, including: a current word element reasoning module, used to reason about the current word element based on the target context information of the current word element, wherein the target context information is determined according to any of the above-mentioned devices.

[0120] Figure 7 This is a block diagram of an electronic device used to implement embodiments of this application. Figure 7As shown, the electronic device includes a memory 701 and a processor 702. The memory 701 stores a computer program that can run on the processor 702. When the processor 702 executes the computer program, it implements the method described in the above embodiments. The number of memories 701 and processors 702 can be one or more. In a specific implementation, the electronic device may also include a communication interface 703 for communicating with external devices and exchanging and sending data.

[0121] In practical implementation, if the memory 701, processor 702, and communication interface 703 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0122] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0123] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0124] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0125] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0126] This application also provides a chip, including: an input interface, a transmission interface, a processor, and a memory. The input interface, the transmission interface, the processor, and the memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0127] It should be understood that the aforementioned processors can be CPUs, GPUs, other general-purpose processors, digital signal processors (DSPs), ASICs, field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Computing Machines (ARM) architecture.

[0128] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include RAM, which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0129] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another.

[0130] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0131] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0132] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0133] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0134] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0135] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0136] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model inference method applied to a first processor, the method comprising: Using the first key-value pair data of at least one first historical word, perform the first attention calculation on the current word and generate the first calculation result; Obtain a second calculation result, wherein the second calculation result is generated by the second processor using second key-value pair data of at least one second historical word to perform a second attention calculation on the current word; Based on the first calculation result and the second calculation result, the target context information of the current word element is inferred.

2. The method according to claim 1, wherein, The step of using at least one first key-value pair of first historical lexical units to perform a first attention calculation on the current lexical unit and generating a first calculation result includes: The query information of the current word element is determined, wherein the query information is used by the second processor to generate the second calculation result based on the second key-value pair data and the query information; The first calculation result is generated based on the first key-value pair data and the query information.

3. The method according to claim 1, wherein, The first calculation result is used to represent the first contextual information provided by the at least one first historical word element for the current word element, and the second calculation result is used to represent the second contextual information provided by the at least one second historical word element for the current word element; The step of reasoning the target context information of the current word element based on the first calculation result and the second calculation result includes: The target context information is calculated based on the first context information and its corresponding first coefficient, and the second context information and its corresponding second coefficient, wherein the first coefficient is used to represent the weight of the first context information in the target context information, and the second coefficient is used to represent the weight of the second context information in the target context information.

4. The method according to claim 3, wherein, The step of reasoning the target context information of the current word element based on the first calculation result and the second calculation result further includes: The first coefficient is calculated based on the maximum value of the first attention score and the first normalization factor in the first attention calculation; Obtain the maximum value of the second attention score and the second normalization factor in the second attention calculation; The second coefficient is calculated based on the maximum value of the second attention score and the second normalization factor.

5. The method according to any one of claims 1 to 4, wherein, Before generating the first calculation result by performing a first attention calculation on the current word using first key-value pairs of at least one first historical word, the method further includes: Retrieve the second key-value pair data from the memory space of the first processor; Send the second key-value pair data to the second processor; After deleting the second key-value pair data from the memory space, the first key-value pair data is retrieved from the memory space.

6. The method according to any one of claims 1 to 4, wherein, Before generating the first calculation result by performing a first attention calculation on the current word using first key-value pairs of at least one first historical word, the method further includes: From a plurality of candidate historical words of the current word word, a preset proportion of candidate historical words words are selected as the second historical word word word, wherein the preset proportion is set based on the memory bandwidth of the first processor and the memory bandwidth of the second processor.

7. A model inference method applied to a second processor, the method comprising: A second attention calculation is performed on the current word using the second key-value pair data of at least one second historical word, generating a second calculation result; The second calculation result is sent to the first processor so that the first processor can infer the target context information of the current word based on the first calculation result and the second calculation result, wherein the first calculation result is generated by the first processor using first key-value pair data of at least one first historical word to perform a first attention calculation on the current word.

8. The method according to claim 7, wherein the second attention calculation on the current word using the second key-value pair data of at least one second historical word element comprises: Obtain the query information of the current term from the first processor; The second calculation result is generated based on the second key-value pair data and the query information.

9. The method according to claim 7, wherein the second calculation result is used to represent the second contextual information provided by the at least one second historical lexical unit for the current lexical unit, the method further comprising: The first processor is sent the maximum second attention score and the second normalization factor from the second attention calculation, wherein the maximum second attention score and the second normalization factor are used to calculate the second coefficient, which represents the weight of the second context information in the target context information.

10. A model reasoning method, comprising: Obtain a first calculation result and a second calculation result, wherein the first calculation result is generated by the first processor using the first key-value pair data of at least one first historical word to perform attention calculation on the current word, and the second calculation result is generated by the second processor using the second key-value pair data of at least one second historical word to perform attention calculation on the current word; Based on the first calculation result and the second calculation result, the target context information of the current word element is inferred.

11. A model reasoning method, comprising: The current word is inferred based on the target context information of the current word, wherein the target context information is determined by the method according to any one of claims 1 to 10.

12. An electronic device, comprising: A first processor is configured to perform the method according to any one of claims 1 to 6 or claim 11; A second processor is configured to perform the method according to any one of claims 7 to 9.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 11.

14. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images