Computing system, multi-round session inference method, apparatus and computing device cluster

Abstract

The present application belongs to the technical field of cloud computing and provides a computing system, a multi-round session inference method, an apparatus and a computing device cluster. In the system, an external storage device stores a historical key value cache of a completed session. As the capacity of the external storage device is far greater than the capacity of an HBM in an accelerator, the hit rate of the historical key value cache can be increased, thereby avoiding recomputing the key value cache. In addition, when the accelerator processes a session, a host can preload a historical key value cache of a session to be processed in a task queue from the external storage device to an internal memory of the host; when performing i-th layer computing on the session, the accelerator can preload from the internal memory of the host the historical key value cache required for the (i+1)-th layer computing. As the computing process and the data loading process are synchronously performed, the computing process does not need to wait for the completion of data loading, such that the time overheads of the accelerator accessing the external storage device can be hidden, improving the efficiency of multi-round session inference.

Description

Computing system, multi-round conversation reasoning method, device and computing device cluster

[0001] This application claims priority to Chinese patent application No. 202410137574.7, filed on January 31, 2024, with the invention name “A method, device and other equipment for data processing”, and No. 202410223741.X, filed on February 28, 2024, with the invention name “Computing system, multi-round conversational reasoning method, device and computing device cluster”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of cloud computing technology, and in particular to a computing system, a multi-round conversation reasoning method, an apparatus, and a computing device cluster. Background Art

[0003] Generative large models based on the Transformer architecture can be applied to multi-round conversational reasoning scenarios. In a single round of conversation, the generative large model processes the input word sequence token[1:s] (also known as the question prompt), inferring the s+1th word token[s+1] to obtain a new word sequence token[1:s+1]. Token[1:s+1] is then input into the model, inferring token[s+2], and so on, until a complete word sequence is obtained or an inference terminator is encountered. The generative large model then outputs the inferred word sequence, which is also the answer, thus completing a round of conversation, where s is an integer greater than 1. When inferring a token, the generative large model generates corresponding intermediate data, key and value, when processing any input token. The generated intermediate data will also be used in the process of inferring subsequent tokens. In addition, when multiple rounds of conversations occur, each round of conversation will use the input and tokens generated by the previous rounds of conversations, and thus also use the intermediate data corresponding to the input and tokens generated by the previous rounds of conversations. Therefore, to save computing resources and improve inference efficiency, the intermediate data corresponding to the input and tokens generated by the previous rounds of conversations are stored in the high-bandwidth memory (HBM) inside the accelerator. That is, the HBM inside the accelerator is used to store the historical key-value cache (KV cache) of the session, so that the historical KV cache of the previous rounds of conversations can be directly reused for inference in subsequent sessions without recalculation.

[0004] However, the above method relies on HBM to store the KV cache, and HBM has limited capacity. When faced with insufficient HBM available storage space, the accelerator will delete the KV cache stored in HBM based on business needs, resulting in the inability to reuse the historical KV cache in subsequent sessions. The generative large model faces huge recalculation overhead on the historical KV cache, which still results in reduced inference efficiency and waste of computing resources.

[0005] Summary of the Invention

[0006] The present invention provides a computing system, a multi-round conversation reasoning method, an apparatus, and a computing device cluster that can avoid KV cache recalculation and improve the efficiency of multi-round conversation reasoning. The technical solution is as follows.

[0007] In a first aspect, a computing system is provided, the computing system including a host, an accelerator of the host, and an external storage device;

[0008] Among them, the external storage device is used to store the historical KV cache of completed sessions; the host is used to preload the historical KV cache of the pending sessions in the task queue from the external storage device into the host's memory while the accelerator is processing the session; the accelerator is used to preload the historical KV cache required for the i+1th layer calculation from the host's memory into the accelerator while performing the i-th layer calculation of the generative large model for the session.

[0009] In the above system, since the capacity of the external storage device is much larger than the capacity of the HBM in the accelerator, the external storage device can save a large amount of historical KV cache, thereby improving the hit rate of the historical KV cache, thereby avoiding KV cache recalculation, improving the efficiency of multi-round conversation reasoning, and saving computing resources; and since the calculation process is carried out synchronously with the data loading process, the calculation process does not need to wait for the data loading to be completed, thereby hiding the time overhead of the accelerator accessing the external storage device and improving the efficiency of multi-round conversation reasoning.

[0010] In some embodiments, the historical KV cache of the input of the first session at the (i+1) level is independent of the position encoding of the input of the first session.

[0011] Here, irrelevant means that the position encoding of the input of the first session has no impact on the historical KV cache of the first session.

[0012] In some embodiments, the number of word-grams included in the input of the first session is greater than or equal to the number of word-grams that can be accommodated by the context window of the generative large model;

[0013] The accelerator is configured to: when performing an i-th level calculation on the first session, load a first sub-input from the memory into the historical KV cache at the i+1th level, where the first sub-input is a sub-input of the input of the first session, and the number of tokens included in the first sub-input is smaller than the number of tokens that can be accommodated by the context window.

[0014] In the above system, the stored KV cache is decoupled from the position encoding. When the context window overflows, changes in the position encoding of the generative large model's input will not cause the stored KV cache to become invalid. Therefore, the accelerator can reuse the historical KV cache of the cropped input, thereby avoiding KV cache recalculation caused by context window overflow, improving inference efficiency and saving computing resources.

[0015] In some embodiments, the accelerator is further configured to:

[0016] The position code of the first sub-input is embedded in the historical KV cache of the first sub-input at the i+1th layer; and the i+1th layer calculation is performed on the first session based on the input of the first session and the historical KV cache of the first sub-input with the embedded position code at the i+1th layer.

[0017] The position encoding is relative positional encoding (RPE).

[0018] In some embodiments, the accelerator includes a first buffer and a second buffer, the first buffer being used to store a history KV cache of the first session, and the second buffer being used to store the history KV cache of the input of the second session in the task queue at the first layer, the second session being the first session to be processed after the first session in the task queue;

[0019] The accelerator is configured to: when performing layer i calculations on the first session, load the historical KV cache of the first session's input at layer i+1 from the memory to the first buffer; and when performing layer N calculations on the first session, load the historical KV cache of the second session's input at layer 1 from the memory to the second buffer.

[0020] In the above system, the HBM of the accelerator includes a first buffer and a second buffer. The first buffer is used to store the KV cache of the session currently being processed by the accelerator, and the second buffer is used to store the KV cache of the session to be processed by the accelerator. While the accelerator processes the first session based on the KV cache in the first buffer, it loads the KV cache required for the first layer calculation of the second session into the second buffer. When the accelerator completes processing of the first session, it processes the second session based on the KV cache in the second buffer. At this time, the second buffer becomes the execution buffer. While processing the second session, the accelerator releases the first buffer through an asynchronous thread, so that the computing thread of the accelerator can start processing the second session immediately after completing processing of the first session without waiting for the first buffer to be released. This hides the time gap between the two sessions caused by loading the data required for the next session, thereby improving the efficiency of inference.

[0021] In some embodiments, the accelerator is further configured to: when performing an i+1th layer calculation on the first session, write a KV cache generated by performing an i-th layer calculation on the first session into the memory.

[0022] In the above system, while the accelerator is performing the calculation of the i+1th layer, the KV cache generated by the i-th layer calculation is written back to the host's memory. Compared with writing the KV cache generated by a round of session inference after the current round is completed, this can hide the time gap between two adjacent rounds of sessions caused by writing back the data generated by the previous round of session, and can improve the efficiency of inference.

[0023] In some embodiments, the accelerator includes a third buffer, the third buffer being used to store the KV cache generated by the generative large model processing the first session;

[0024] The accelerator of the host is used to: if, when the Nth layer calculation of the first session is completed, there is a KV cache in the KV cache generated by the generative large model for processing the first session that has not been written to the memory, then write the KV cache that has not been written to the memory into the third buffer zone; when the second session in the task queue is processed by the generative large model, write the KV cache in the third buffer zone into the memory.

[0025] In the above system, the accelerator's HBM also includes a third buffer. After the accelerator completes processing of the first session, it writes the KV cache generated during the processing of the first session and not yet written back to the memory from the first buffer to the third buffer. Since the data copy efficiency between the first buffer and the third buffer is relatively high, the accelerator can quickly release the first buffer after completing processing of the first session, thereby loading the KV cache required for subsequent processing into the first buffer without waiting for the KV cache generated during the processing of the first session to be completely written into the memory before releasing the first buffer. This hides the time gap between the two sessions caused by waiting for the data of the previous session to be written back, thereby improving the efficiency of inference.

[0026] In some embodiments, the host is configured to: if the historical KV cache of the first to-be-processed session in the task queue does not exist in the memory, load the historical KV cache of the first to-be-processed session from the external storage device to the memory, where the first to-be-processed session is the first number of to-be-processed sessions at the head of the task queue.

[0027] In the above system, the host can preload the historical KV cache of the pending sessions in the task queue from the external storage device into the host's memory while the accelerator is processing the session. Since the calculation process and the data loading process are carried out synchronously, the calculation process does not need to wait for the data loading to be completed, thereby hiding the time overhead of the accelerator accessing the external storage device and improving the efficiency of multi-round conversation reasoning.

[0028] In some embodiments, the number of session rounds included in the first to-be-processed session is determined based on the capacity of the memory.

[0029] In some embodiments, the host is configured to: if the available storage space in the memory is sufficient, load the historical KV cache of the first pending session from the external storage device to the memory; if the available storage space in the memory is insufficient, write the historical KV cache of the second pending session in the task queue from the memory to the external storage device, and load the historical KV cache of the first pending session from the external storage device to the memory, where the second pending session is a session in the task queue other than the first pending session.

[0030] In some embodiments, the host is further configured to: if the available storage space of the external storage device is insufficient, delete the historical KV cache of the third pending session in the task queue from the external storage device, where the third pending session is the second-to-last pending session in the task queue.

[0031] In a second aspect, a multi-round conversation reasoning method is provided, which is applied to a computing system. The computing system includes a host, an accelerator of the host, and an external storage device. The external storage device is used to store a historical key-value cache (KV cache) of the session. The host is used to load the historical KV cache of the session to be processed in the task queue from the external storage device to the host's memory. The method includes:

[0032] The accelerator, based on the task queue, uses a generative large model to perform inference on a first session in the task queue, where the first session is one round of a multi-round session, and the generative large model includes N layers. When the accelerator performs layer i calculations on the first session, the input of the first session at layer i+1 of the historical KV cache is loaded from the memory into the accelerator, where i is an integer greater than or equal to 1 and less than N.

[0033] In some embodiments, the historical KV cache of the input of the first session at the (i+1) level is independent of the position encoding of the input of the first session.

[0034] In some embodiments, the number of word-grams included in the input of the first session is greater than or equal to the number of word-grams that can be accommodated by the context window of the generative large model;

[0035] When performing an i-th level calculation on the first session, loading the input of the first session at the i+1 level historical KV cache from the memory includes: when performing the i-th level calculation on the first session, loading a first sub-input from the memory at the i+1 level historical KV cache, where the first sub-input is a sub-input of the input of the first session, and the number of tokens included in the first sub-input is less than the number of tokens that can be accommodated by the context window.

[0036] In some embodiments, processing the first session in the task queue using the generative large model includes:

[0037] The position code of the first sub-input is embedded in the historical KV cache of the first sub-input at the i+1th layer; and the i+1th layer calculation is performed on the first session based on the input of the first session and the historical KV cache of the first sub-input with the embedded position code at the i+1th layer.

[0038] In some embodiments, the accelerator includes a first buffer and a second buffer, the first buffer being used to store a history KV cache of the first session, and the second buffer being used to store the history KV cache of the input of the second session in the task queue at the first layer, the second session being the first session to be processed after the first session in the task queue;

[0039] When performing an i-th layer calculation on the first session, loading the historical KV cache of the input of the first session at the i+1 layer from the memory includes: when the accelerator performs the i-th layer calculation on the first session, loading, by the accelerator, the historical KV cache of the input of the first session at the i+1 layer from the memory to the first buffer.

[0040] In some embodiments, the method further comprises:

[0041] When the accelerator performs the Nth layer calculation on the first session, the accelerator loads the historical KV cache of the input of the second session at the first layer from the memory to the second buffer.

[0042] In some embodiments, the method further comprises:

[0043] When the accelerator performs the (i+1)th layer calculation on the first session, the KV cache generated by performing the (i)th layer calculation on the first session is written into the memory by the accelerator.

[0044] In some embodiments, the accelerator includes a third buffer, the third buffer being used to store the KV cache generated by the generative large model processing the first session;

[0045] When performing the i+1th layer calculation on the first session, the KV cache generated by the i-th layer calculation on the first session is written into the memory, including: if, when the accelerator completes the N-th layer calculation on the first session, there is a KV cache in the KV cache generated by the generative large model processing the first session that has not been written to the memory, then the accelerator writes the KV cache that has not been written to the memory into the third buffer; when the accelerator processes the second session in the task queue through the generative large model, the accelerator writes the KV cache in the third buffer into the memory.

[0046] In some embodiments, the method further includes: if the historical KV cache of the first to-be-processed session in the task queue does not exist in the memory, loading the historical KV cache of the first to-be-processed session from the external storage device to the memory through the host, where the first to-be-processed session is the first number of to-be-processed sessions at the head of the task queue.

[0047] In some embodiments, the number of session rounds included in the first to-be-processed session is determined based on the capacity of the memory.

[0048] In some embodiments, if the historical KV cache of the first session to be processed in the task queue does not exist in the memory, loading the historical KV cache of the first session to be processed from the external storage device to the memory through the host includes:

[0049] If the available storage space in the memory is sufficient, the host loads the historical KV cache of the first pending session from the external storage device to the memory. If the available storage space in the memory is insufficient, the host writes the historical KV cache of the second pending session in the task queue from the memory to the external storage device, and loads the historical KV cache of the first pending session from the external storage device to the memory. The second pending session is the session in the task queue other than the first pending session.

[0050] In some embodiments, the method further comprises:

[0051] If the available storage space of the external storage device is insufficient, the host deletes the historical KV cache of the third pending session in the task queue from the external storage device, where the third pending session is the second-to-last pending session in the task queue.

[0052] In a third aspect, a multi-round conversation reasoning method is provided, which is applied to an accelerator of a host in a computing system. The computing system also includes a host and an external storage device, the external storage device is used to store a historical key-value cache (KV cache) of the session, and the host is used to load the historical KV cache of the session to be processed in the task queue from the external storage device to the host's memory. The method includes:

[0053] Based on the task queue, reasoning is performed on a first session in the task queue using a generative large model. The first session is one round of a multi-round session. The generative large model includes N layers. When performing an i-th layer calculation on the first session, the input of the first session is loaded from the memory into the historical KV cache at the i+1th layer, where i is an integer greater than or equal to 1 and less than N.

[0054] In some embodiments, the historical KV cache of the input of the first session at the (i+1) level is independent of the position encoding of the input of the first session.

[0055] In some embodiments, the number of word-grams included in the input of the first session is greater than or equal to the number of word-grams that can be accommodated by the context window of the generative large model;

[0056] When performing an i-th level calculation on the first session, loading the input of the first session at the i+1 level historical KV cache from the memory includes: when performing the i-th level calculation on the first session, loading a first sub-input from the memory at the i+1 level historical KV cache, where the first sub-input is a sub-input of the input of the first session, and the number of tokens included in the first sub-input is less than the number of tokens that can be accommodated by the context window.

[0057] In some embodiments, processing the first session in the task queue using the generative large model includes:

[0058] The position code of the first sub-input is embedded in the historical KV cache of the first sub-input at the i+1th layer; and the i+1th layer calculation is performed on the first session based on the input of the first session and the historical KV cache of the first sub-input with the embedded position code at the i+1th layer.

[0059] In some embodiments, the accelerator includes a first buffer and a second buffer, the first buffer being used to store a history KV cache of the first session, and the second buffer being used to store the history KV cache of the input of the second session in the task queue at the first layer, the second session being the first session to be processed after the first session in the task queue;

[0060] When performing the i-th layer calculation on the first session, loading the input of the first session at the i+1 layer historical KV cache from the memory includes: when performing the i-th layer calculation on the first session, loading the input of the first session at the i+1 layer historical KV cache from the memory to the first buffer.

[0061] In some embodiments, the method further includes: when performing the Nth layer calculation on the first session, loading the historical KV cache of the second session input at the first layer from the memory to the second buffer.

[0062] In some embodiments, the method further includes: when performing the (i+1)th layer calculation on the first session, writing the KV cache generated by performing the i-th layer calculation on the first session into the memory.

[0063] In some embodiments, the accelerator includes a third buffer, the third buffer being used to store the KV cache generated by the generative large model processing the first session;

[0064] When performing the i+1th layer calculation on the first session, the KV cache generated by the i-th layer calculation on the first session is written into the memory, including: if when the N-th layer calculation on the first session is completed, there is a KV cache in the KV cache generated by the generative large model for processing the first session that has not been written to the memory, then the KV cache that has not been written to the memory is written into the third buffer; when the second session in the task queue is processed by the generative large model, the KV cache in the third buffer is written into the memory.

[0065] In a fourth aspect, a multi-round conversation reasoning method is provided, which is applied to a host in a computing system. The computing system also includes an accelerator of the host and an external storage device. The external storage device is used to store a historical key-value cache (KV cache) of the conversation. The accelerator is used to perform reasoning on a first conversation in the task queue using a generative large model based on the task queue. The first conversation is one of the multi-round conversations. The generative large model includes N layers. When performing the i-th layer calculation on the first conversation, the input of the first conversation is loaded from the memory into the historical KV cache at the i+1th layer, where i is an integer greater than or equal to 1 and less than N.

[0066] The method includes:

[0067] The historical KV cache of the session to be processed in the task queue is loaded from the external storage device to the memory of the host; and the historical KV cache of the session to be processed is stored.

[0068] In some embodiments, loading the historical KV cache of the pending sessions in the task queue from the external storage device to the memory of the host includes:

[0069] If the historical KV cache of the first session to be processed in the task queue does not exist in the memory, the historical KV cache of the first session to be processed is loaded from the external storage device to the memory, where the first session to be processed is the first session to be processed at the head of the task queue.

[0070] In some embodiments, the number of session rounds included in the first to-be-processed session is determined based on the capacity of the memory.

[0071] In some embodiments, if the historical KV cache of the first session to be processed in the task queue does not exist in the memory, the host loads the historical KV cache of the first session to be processed from the external storage device to the memory, including:

[0072] If the available storage space in the memory is sufficient, the historical KV cache of the first pending session is loaded from the external storage device into the memory. If the available storage space in the memory is insufficient, the historical KV cache of the second pending session in the task queue is written from the memory to the external storage device, and the historical KV cache of the first pending session is loaded from the external storage device into the memory. The second pending session is the session in the task queue other than the first pending session.

[0073] In some embodiments, the method further comprises:

[0074] If the available storage space of the external storage device is insufficient, the historical KV cache of the third pending session in the task queue is deleted from the external storage device. The third pending session is the second-to-last pending session in the task queue.

[0075] In a fifth aspect, a multi-round conversation reasoning device is provided, which is applied to an accelerator of a host in a computing system. The device includes at least one functional module, which is used to execute the multi-round conversation reasoning method provided by the aforementioned third aspect or any possible implementation method of the third aspect.

[0076] In the sixth aspect, a multi-round conversation reasoning device is provided, which is applied to a host in a computing system. The device includes at least one functional module, and the at least one functional module is used to execute the multi-round conversation reasoning method provided by the aforementioned fourth aspect or any possible implementation method of the fourth aspect.

[0077] In the seventh aspect, an accelerator is provided, which includes a computing core and a memory. The accelerator is used to execute the multi-round conversational reasoning method provided by the aforementioned third aspect or any possible implementation of the third aspect, the memory is used to store computing data, and the computing core is used to perform computing operations on the computing data stored in the memory.

[0078] In an eighth aspect, a host is provided, comprising a processor and a memory, and configured to execute the multi-round conversational reasoning method provided in the aforementioned fourth aspect or any possible implementation of the fourth aspect.

[0079] In a ninth aspect, a computing device cluster is provided, comprising at least one computing device, each of the computing devices comprising a host and an accelerator of the host, an external storage device located on at least one computing device, the external storage device being configured to store a historical KV cache of a session, the host being configured to load the historical KV cache of a pending session in a task queue from the external storage device into a memory of the host;

[0080] The host in the computing device cluster is used to execute the multi-round conversation reasoning method provided by the aforementioned fourth aspect or any possible implementation of the fourth aspect;

[0081] The accelerator in the computing device cluster is used to execute the multi-round conversation reasoning method provided by the aforementioned third aspect or any possible implementation of the third aspect.

[0082] In the tenth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the multi-round conversation reasoning method provided in the aforementioned third aspect or any possible implementation of the third aspect, or executes the multi-round conversation reasoning method provided in the aforementioned fourth aspect or any possible implementation of the fourth aspect.

[0083] In the eleventh aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the multi-round conversation reasoning method provided in the aforementioned third aspect or any possible implementation of the third aspect, or executes the multi-round conversation reasoning method provided in the aforementioned fourth aspect or any possible implementation of the fourth aspect.

[0084] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] FIG1 is a schematic diagram of the structure of a computing system provided in an embodiment of the present application;

[0086] FIG2 is a functional schematic diagram of a computing system provided in an embodiment of the present application;

[0087] FIG3 is a flowchart of a multi-round conversation reasoning method provided by an embodiment of the present application;

[0088] FIG4 is a schematic diagram of directly trimming the KV cache after a context window overflow, provided by an embodiment of the present application;

[0089] FIG5 is a schematic diagram of reasoning based on a KV cache decoupled from position encoding provided by an embodiment of the present application;

[0090] FIG6 is a schematic diagram of a multi-round conversation reasoning method provided by an embodiment of the present application in which a calculation process and a data loading process are carried out in parallel;

[0091] FIG7 is a schematic diagram of a multi-round conversation reasoning method provided by an embodiment of the present application in which a calculation process and a data write-back process are carried out in parallel;

[0092] FIG8 is a schematic diagram of a data migration process between a host memory and an external storage device during a multi-round conversation reasoning process provided by an embodiment of the present application;

[0093] FIG9 is a schematic structural diagram of a multi-round conversation reasoning device provided in an embodiment of the present application;

[0094] FIG10 is a schematic diagram of the structure of a multi-round conversation reasoning device provided in an embodiment of the present application;

[0095] FIG11 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0096] FIG12 is a schematic diagram of a computing device cluster provided in an embodiment of the present application;

[0097] FIG13 is a schematic diagram of a possible implementation of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0099] First of all, this application involves the application of a generative large model based on the Transformer architecture in multi-round conversational reasoning. In order to facilitate understanding of the content of the embodiments of this application, several technical terms involved in the embodiments of this application are explained below.

[0100] Token: The smallest semantic unit represented by a vector, where the smallest semantic unit is, for example, a character, word, or phrase. A word sequence composed of multiple tokens serves as the input of the generative large model. In a multi-round conversational reasoning scenario, the word sequence input to the generative large model is also the question prompt of the conversation. The generative large model infers the input word sequence, and through multiple iterations, obtains the subsequent word elements of the question prompt one by one until a complete subsequent word sequence is obtained or an inference terminator appears. The generative large model outputs the generated word sequence, which is also the answer to the question prompt. For example, the input word sequence of the generative large model is X[1:s]. The generative large model performs the first iteration based on the word sequence X[1:s] and obtains the s+1th word token[s+1]. The generative large model performs the second iteration based on X[1:s] and token[s+1] and obtains the s+2th word token[s+2]. This process continues until a complete word sequence is obtained or an inference terminator appears. The generative large model outputs the inferred word sequence, that is, the answer, thereby completing the reasoning of a round of dialogue, where s is an integer greater than 1.

[0101] Generative large models based on the Transformer architecture: consisting of N Transformer layers, where N is an integer greater than 1. Each Transformer layer includes a self-attention mechanism and a feed-forward network (FFN).

[0102] One-round conversational reasoning: In each iteration, the N Transformer layers of the generative large model sequentially process the input word sequence of this iteration to obtain a word. The process of processing the input word sequence by the i-th layer of the generative large model includes: projecting each word in the input of this iteration onto the model parameters corresponding to the self-attention mechanism in the i-th layer, generating intermediate data corresponding to each word, namely, a key and value; performing a nonlinear transformation on the key and value, such as a residual connection and normalization, and passing the result of the nonlinear transformation to the feedforward neural network in the i-th layer; the feedforward neural network in the i-th layer processes the result of the nonlinear transformation to obtain the processing result of the i-th layer, and then passes the processing result of the i-th layer to the i+1 layer for processing. Here, i is an integer greater than or equal to 1 and less than N. The process of processing the input word sequence by the i-th layer of the generative large model to obtain the processing result of the i-th layer is also called the i-th layer calculation.

[0103] Multi-round conversation reasoning: To maintain semantic consistency within the conversation context and ensure accurate understanding of the current conversation input, the generative large model refers to the input and output of previous conversations with the same subject when generating answers for the current conversation. Specifically, the input of a conversation in a multi-round conversation consists of the word sequence input in the previous conversation, the word sequence output in the previous conversation, and the word sequence input by the subject in that conversation.

[0104] Key-value cache (KV cache): During an iteration of the generative large model (that is, during the process of generating a word), the generative large model generates corresponding intermediate data key and value when processing any word, and the generated intermediate data will be used in subsequent iterations. In addition, when multiple rounds of conversations occur, each round of conversation will use the word sequence input and output in the historical conversation, and thus also use the intermediate data corresponding to the word input and output in the historical conversation. Therefore, to save computing resources and improve reasoning efficiency, the intermediate data corresponding to the word input and output in the historical conversation is stored to form the historical KV cache of the conversation. When performing conversation reasoning, the historical KV cache of the conversation can be directly reused for reasoning without recalculation.

[0105] Pre-population phase: This phase is the first iteration of a conversational reasoning process. During this phase, the generative model performs the first iteration based on the word sequence X[1:s] input in the conversational round, obtaining token[s+1]. It then stores the key and value corresponding to each word in X[1:s] obtained in this iteration, forming a KV cache[1:s].

[0106] Decoding phase: This phase is the second to the last iteration of a round of conversational reasoning. During each iteration of the decoding phase, the generative model calculates the key and value corresponding to the token obtained in the previous iteration, reads the stored historical KV cache, and obtains the token for the current iteration based on the key and value calculated in the current iteration and the historical KV cache. The calculated key and value are then stored, effectively updating the historical KV cache.

[0107] Context window: The number of tokens that a large generative model can process simultaneously.

[0108] Accelerator stream: An abstract unit of parallel computing tasks performed by an accelerator.

[0109] Execution buffer: An area in high-bandwidth memory (HBM) used to store the KV cache that the accelerator's inference computation flow can currently process.

[0110] The above introduces several technical terms involved in the embodiments of the present application, and the following introduces the implementation environment of the embodiments of the present application.

[0111] Figure 1 is a structural diagram of a computing system provided in an embodiment of the present application. As shown in Figure 1, the computing system includes a host 101, an accelerator 102 of the host, and an external storage device 103, wherein the host 101, the accelerator 102, and the external storage device 103 communicate with each other via a wired network or a wireless network.

[0112] The computing system can be deployed on a computing device cluster, which includes at least one computing device, which can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0113] The external storage device 103 is, for example, a solid state disk (SSD) or a hard disk drive (HDD), or a cloud storage service such as an object storage service (OBS), an elastic volume service (EVS), or a scalable file service (SFS). This embodiment of the application does not limit the type of the external storage device 103. Schematically, in a multi-level storage system, the external storage device 103 is used to store the historical KV cache of the session.

[0114] The host 101 is, for example, a host of a computing device, or a main computing device in a computing device cluster for controlling and managing other computing devices. The memory of the host 101 is, for example, dynamic random access memory (DRAM). Schematically, a task scheduler runs on the host 101, which maintains a task queue for indicating pending sessions of the accelerator; the memory of the host 101 is used to store a historical KV cache of pending sessions in the task queue; and the host 101 is used to load the historical KV cache of pending sessions in the task queue from the external storage device 101 to the memory of the host 101 based on the task queue.

[0115] The accelerator 102 is, for example, a general-purpose central processing unit (CPU), a graphics processing unit (GPU), a switching module processor unit (SMPU), a network processor unit (NPU), a microprocessor, or one or more integrated circuits for implementing the solution of the present application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The above-mentioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The embodiment of the present application does not limit the type of accelerator. Schematically, the HBM in the accelerator 102 is used to store the historical KV cache of the first session. The accelerator 102 can reason about the first session in the task queue through a generative large model, that is, perform calculations at the i-th layer of the generative large model, thereby generating an answer to the first session. Among them, the first session is one round of multiple rounds of sessions. When the accelerator 102 performs the i-th layer calculation, it reuses the input of the first session in the HBM of the accelerator 102 in the i-th layer historical KV cache, and when performing the i-th layer calculation, it loads the historical KV cache of the first session at the i+1 layer from the memory into the HBM.

[0116] In some embodiments, each computing device in the computing device cluster includes a host 101, an accelerator 102, and an external storage device 103. That is, the storage device configured for each computing device is used as the external storage device 103. In other embodiments, each computing device in the computing device cluster includes the host 101 and the accelerator 102, and at least one computing device with storage capabilities in the computing cluster serves as the external storage device 103. This embodiment of the present application is not limited to this.

[0117] For example, FIG2 is a functional diagram of a computing system provided in an embodiment of the present application. As shown in FIG2 , the computing system includes a control system and a multi-level storage system. The multi-level storage system includes the memory of the host 101, the HBM in the accelerator 102, and the external storage device 103. The control system runs on the host 101 and the accelerator 102 and includes a task scheduler and a key-value cache management unit. The task scheduler maintains a task queue for indicating pending sessions of the accelerator; the key-value cache management unit is used to control data migration between the memory of the host 101 and the external storage device 103. Specifically, the key-value cache management subunit includes a key-value cache pulling subunit and a key-value cache placement subunit. The key-value cache pulling subunit is used to pre-pull the historical KV cache of the session to be processed in the task queue into the memory of the host 101, and to load the KV cache in the memory of the host 101 into the HBM of the accelerator; the key-value cache placement subunit is used to evict the KV cache from the memory of the host 101 to the external storage device 103 when the available storage space in the memory of the host 101 is insufficient, and to store (write back) the KV cache generated by the accelerator processing session into the memory of the host. It should be noted that the division of the functional modules of the computing system shown in Figure 2 is only exemplary, and the embodiments of the present application do not limit the division method of the functional modules of the computing system.

[0118] In some embodiments, the wireless network or wired network described above uses standard communication technologies and / or protocols. The network is typically a Transmission Control Protocol / Internet Protocol (TCP / IP) network in a data center network and an RDMA network, such as an RDMA over converged Ethernet (RoCE) network or an InfiniBand (IB) network, without limitation. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the above-mentioned data communication technologies.

[0119] The following describes a multi-round conversation reasoning method provided by an embodiment of the present application, which is applied to the above-mentioned computing system. In this method, an external storage device stores the historical KV cache of completed conversations. Since the capacity of the external storage device is much larger than the capacity of the HBM in the accelerator, the external storage device can save a large amount of historical KV cache, thereby improving the hit rate of the historical KV cache, thereby avoiding KV cache recalculation, and improving the efficiency of multi-round conversation reasoning and saving computing resources. At the same time, the host can pre-load the historical KV cache of the pending session in the task queue from the external storage device to the host's memory while the accelerator is processing the session. The accelerator can pre-load the historical KV cache required for the i+1th layer calculation from the host's memory to the accelerator while performing the i-th layer calculation of the generative large model for the session. Since the calculation process and the data loading process are carried out synchronously, the calculation process does not need to wait for the data loading to be completed, thereby hiding the time overhead of the accelerator accessing the external storage device and improving the efficiency of multi-round conversation reasoning.

[0120] The above method involves data migration between the accelerator and host memory, and between the host memory and external storage devices during multi-round conversational reasoning. The following describes these two data migration processes separately.

[0121] First, the data migration process between the accelerator and the host's memory is described. Figure 3 is a flowchart of a multi-round conversational reasoning method provided by an embodiment of the present application. As shown in Figure 3, the method is applied to a computing system including a host, an accelerator of the host, and an external storage device. The method includes the following steps 301 to 308.

[0122] Step 301: The accelerator loads the historical KV cache of the first layer of the generative large model of the input of the first session from the host memory to the first buffer of the accelerator. The generative large model includes N layers, where N is an integer greater than 1.

[0123] The first conversation is one of the multiple conversations initiated by the first object. The input of the first conversation includes the word sequence input in the historical conversation of the first conversation, the word sequence output in the historical conversation, and the word sequence input by the object in the first conversation. The multiple conversations initiated by the same object have the same conversation window identifier. The historical conversation of the first conversation is also a completed conversation with the same conversation window identifier as the first conversation. For example, the first object initiated three conversations, the first conversation and the second conversation have been completed, and the first conversation is the third conversation. The word sequence input in the first conversation is Q1, and the word sequence output is A1. The word sequence input in the second conversation is Q2, and the word sequence output is A2. The word sequence input in the third conversation is Q3, then the input of the first conversation is [Q1 A1 Q2 A2 Q3].

[0124] The historical KV cache of the first session's input in layer 1 is the KV cache in the historical KV cache of the first session that is associated with layer 1. The first buffer is the area in the accelerator's HBM used to store the KV cache currently processable by the accelerator, that is, the accelerator's execution buffer.

[0125] Before the accelerator performs the first-layer computation for the first iteration of the first session, the accelerator initiates a data reading thread. This data reading thread loads the first-layer historical KV cache of the first session's input from the host's memory into the first buffer. In this method, the data required for the first-layer computation is made available in the accelerator's first buffer by the data reading thread before the first-layer computation occurs. This avoids the time gap caused by loading data from the host's memory, thereby improving inference efficiency.

[0126] The historical KV cache is a multidimensional vector of variable length. The historical KV cache can be stored at different granularities. For example, in some embodiments, the historical KV cache is stored at the granularity of a single session. Specifically, the intermediate data keys and values ​​obtained by processing multiple sessions using the generative large model are stored as multiple KV caches per session, with each KV cache having a session identifier. For another example, in other embodiments, the historical KV cache is stored at the granularity of a single layer of the generative large model. Specifically, the intermediate data keys and values ​​obtained by processing multiple layers of the generative large model are stored as multiple KV caches per layer, with each KV cache having a session identifier and a layer identifier. For another example, in yet other embodiments, the historical KV cache is stored at the granularity of a group of multiple sessions. Specifically, the intermediate data keys and values ​​obtained by processing multiple sessions initiated by the generative large model for the same object are stored as a single KV cache. It should be noted that the above description of the storage granularity of the historical KV cache is merely exemplary, and the embodiments of this application do not limit the storage granularity of the KV cache.

[0127] The historical KV cache of the first session input in the first layer is independent of the position coding of the first session input, where independent means that the position coding of the first session input has no impact on the historical KV cache of the first session.

[0128] In some embodiments, the number of tokens included in the input of the first session is greater than the number of tokens that can be accommodated by the context window of the generative large model; the accelerator loads the historical KV cache of the first sub-input at the first layer from the host's memory, where the first sub-input is a sub-input of the input of the first session, and the number of tokens included in the first sub-input is less than the number of tokens that can be accommodated by the context window. This process can be understood as, when the input of the first session causes the context window to overflow, the input of the first session is trimmed to discard the first tokens in the input of the first session, and the first sub-input of the first session is used as the input of the generative large model. Correspondingly, the accelerator loads the historical KV cache corresponding to the first sub-input from the host's memory, that is, directly trimming the KV cache after the context window overflows. The above-mentioned loading process is illustrated below with reference to FIG4 . FIG4 is a schematic diagram of a method for directly pruning a KV cache after a context window overflow, as provided by an embodiment of the present application. As shown in FIG4 , the host's memory stores the input of the first session in the historical KV cache of the first layer, where the position code of the input of the first session is [0:2048]. When the input of the first session causes a context window overflow, the accelerator loads the first sub-input of the first session from the host's memory into the historical KV cache of the first layer. The position code of the first sub-input is [0:1536]. The accelerator embeds the position code of the first sub-input into the historical KV cache of the first layer, and performs subsequent reasoning based on the historical KV cache with the embedded position code. The accelerator embedding the position code into the historical KV cache refers to embedding the position code into a key in the historical KV cache.

[0129] In the above embodiment, the stored KV cache is decoupled from the position encoding. When the context window overflows, the change in the position encoding of the generative large model's input will not cause the stored KV cache to become invalid. Therefore, the accelerator can reuse the historical KV cache of the cropped input, thereby avoiding KV cache recalculation caused by context window overflow, improving inference efficiency, and saving computing resources.

[0130] It should be noted that the historical KV cache of the first session is stored in an external storage device. Before the accelerator performs inference on the first session using the generative large model, the host preloads the historical KV cache of the first session from the external storage device into the host's memory. The accelerator then loads the historical KV cache of the first session's input from the host's memory. The data migration process between the external storage device and the host's memory will be described in subsequent embodiments and will not be repeated here.

[0131] Step 302: The accelerator performs the first-layer calculation of the first session in the first iteration based on the input of the first session using the generative large model. At the same time, the accelerator loads the historical KV cache of the second layer of the input of the first session from the host's memory into the first buffer of the accelerator.

[0132] The accelerator starts a computing thread, and performs the first layer of computing of the first iteration on the first session through the computing thread.

[0133] When the number of tokens included in the input of the first session is smaller than the number of tokens that can be accommodated by the context window of the generative large model, that is, when the context window does not overflow, the accelerator performs the first-layer calculation of the first session in the first iteration using the generative large model, including: projecting the input token sequence of the first session onto the model parameters corresponding to the self-attention mechanism in the first layer to generate intermediate data keys and values ​​corresponding to each token in the input token sequence of the first session; updating the historical KV cache of the first session input in the first layer based on the keys and values ​​of the input token sequence of the first session in the first layer; embedding the position encoding of the input of the first session into the updated historical KV cache, where the position encoding is a relative positional encoding (RPE); performing a nonlinear transformation on the keys and values ​​in the historical KV cache after embedding the position encoding, where the nonlinear transformation includes residual connection and normalization, and passing the result of the nonlinear transformation to the feedforward neural network in the first layer; processing the result of the nonlinear transformation by the feedforward neural network in the first layer to obtain the calculation result of the first layer. The accelerator updates the historical KV cache of the first session's input at the first layer based on the key and value of the first session's input word sequence at the first layer, which means that each time the accelerator computing thread calculates an intermediate data key or value, it appends the intermediate data to the historical KV cache of the first session's input at the first layer in the first buffer.

[0134] The calculation process of the first layer is illustrated below with an example of FIG5 , which is a schematic diagram of reasoning based on a KV cache decoupled from position encoding provided by an embodiment of the present application. As shown in FIG5 , the accelerator compares the input word sequence (input) of the first session with the model parameters (W ) corresponding to the self-attention mechanism of the first layer. k 、W v) to generate intermediate data keys and values ​​corresponding to each word in the input word sequence of the first session; the accelerator updates the historical KV cache; the accelerator embeds the position code of the input of the first session into the key in the updated historical KV cache, and performs subsequent reasoning based on the KV cache with the embedded position code.

[0135] Among them, when the number of word units included in the input of the first session is greater than or equal to the number of word units that can be accommodated by the context window of the generative large model, that is, when the context window overflows, the accelerator performs the first layer of calculation of the first iteration for the first session through the generative large model based on the first sub-input of the first session. The calculation process is similar to the calculation process when the context window does not overflow. The difference is that when the context window overflows, the input of the generative large model and the historical KV cache loaded by the accelerator from the host's memory are pruned. The similarities are not repeated here.

[0136] The process of the accelerator loading the historical KV cache of the input of the first session in the second layer from the host memory to the first buffer is similar to the process of loading the historical KV cache of the first layer in step 301 and will not be repeated here.

[0137] Step 303: The accelerator performs the second-layer calculation of the first iteration for the first session. At the same time, the accelerator loads the historical KV cache of the third-layer input of the first session from the host's memory to the first buffer, and writes the KV cache obtained from the first-layer calculation of the first iteration to the host's memory.

[0138] The accelerator performs the second-layer calculation of the first iteration of the first session through the computing thread. At the same time, the accelerator loads the historical KV cache of the third layer of the input of the first session from the host's memory through the data reading thread. The accelerator writes the KV cache obtained from the first-layer calculation of the first iteration from the first buffer to the host's memory through the data write-back thread.

[0139] The accelerator performs the second layer calculation of the first iteration of the first session based on the calculation result of the first layer calculation of the first iteration of the first session through the computing thread. The calculation process of the second layer is the same as that of the first layer and is not repeated here.

[0140] The process of the accelerator loading the input of the first session in the third layer historical KV cache to the first buffer from the host's memory through the data reading thread is similar to the process of loading the first layer historical KV cache in step 301 and will not be repeated here.

[0141] In the above step 303, when the accelerator's computing thread is performing calculations on a certain layer, the accelerator's data reading thread simultaneously starts loading the KV cache of the subsequent layer into the first buffer. After the calculation of the current layer is completed, the computing thread can immediately start calculating the subsequent layer based on the loaded data, hiding the time gap between two adjacent layers caused by loading the data required for the next layer, thereby improving the efficiency of session reasoning; and, when the accelerator's computing thread is performing calculations on a certain layer, the accelerator's data write-back thread simultaneously writes the KV cache generated by the previous layer back to the host's memory. Compared with writing the KV cache generated by this iteration into the host's memory after one iteration is completed, this can hide the time gap between two adjacent iterations caused by writing back the data generated by the previous iteration, thereby improving the efficiency of reasoning.

[0142] Step 304: The accelerator performs the third to Nth layer calculations of the first iteration on the first conversation in the same manner as step 303, and infers the first word in the output of the first conversation.

[0143] Steps 301 to 304 above describe the first iteration of the first session. This first iteration is the pre-population phase of the inference process for the first session. During this pre-population phase, the first token in the output of the first session is inferred. The decoding phase of the inference process for the first session is described below, starting with step 305. This decoding phase includes at least one iteration, with each iteration inferring a token in the output of the first session.

[0144] Step 305: The accelerator performs iterations 2 to M on the first session in the same manner as steps 301 to 304 above, and infers the second to M-th word-grams in the output of the first session, where M is equal to the total number of iterations for the first session and is an integer greater than or equal to 2.

[0145] Among them, any iteration from the second iteration to the Mth iteration is similar to the first iteration. The difference is that in the first iteration, each layer calculates and generates the intermediate data key and value corresponding to each word in the word sequence of the first session input (the question prompt of the first session), while in any iteration from the second iteration to the Mth iteration, each layer calculates and generates the intermediate data key and value corresponding to the word obtained in the previous iteration. The similarities are not repeated here.

[0146] Step 306 : While the accelerator performs the Nth layer calculation of the Mth iteration on the first session, the accelerator loads the historical KV cache of the first layer of the second session input in the task queue from the host's memory into the second buffer of the accelerator.

[0147] The task queue is used to indicate pending sessions. The second session is the first pending session in the task queue. The second buffer is an area in the accelerator's HBM for storing the accelerator's pending KV cache, that is, a historical KV cache for storing the second session's input at layer 1.

[0148] The accelerator loads the input of the second session from the historical KV cache of the first layer in the memory of the host into the second buffer through a data reading thread.

[0149] In the above method, the accelerator's HBM includes a first buffer and a second buffer. The first buffer is used to store the KV cache of the session currently being processed by the accelerator, and the second buffer is used to store the KV cache of the session to be processed by the accelerator. While the accelerator processes the first session based on the KV cache in the first buffer, it loads the KV cache required for the first-layer calculation of the second session into the second buffer. When the accelerator completes processing of the first session, the accelerator processes the second session based on the KV cache in the second buffer. At this time, the second buffer becomes the execution buffer. While processing the second session, the accelerator releases the first buffer through an asynchronous thread, so that the accelerator's computing thread can immediately start processing the second session after completing processing of the first session without waiting for the first buffer to be released. This hides the time gap between the two sessions caused by loading the data required for the next session, thereby improving the efficiency of inference.

[0150] It should be noted that the above step 306 is an optional step. In some embodiments, the accelerator only includes a first buffer; after completing the N-th layer calculation of the M-th iteration of the first session, the accelerator releases the first buffer and then loads the historical KV cache of the second session input at the first layer into the first buffer. The embodiment of the present application is not limited to this. In the above embodiment, the HBM of the accelerator is all used as the first buffer, which can increase the capacity of the first buffer. When the amount of data in the historical KV cache of the session is large, it can accommodate the complete historical KV cache of the session, and can avoid the situation where the historical KV cache loaded later overwrites the historical KV cache loaded earlier due to insufficient capacity, thereby improving the hit rate of the historical KV cache in the first buffer and avoiding repeated loading of the historical KV cache.

[0151] Step 307: If, when the accelerator completes the N-th layer calculation of the M-th iteration of the first session, there is a KV cache in the KV cache generated by the generative large model for processing the first session that has not been written to the memory, the accelerator writes the KV cache that has not been written to the memory from the first buffer to the third buffer of the accelerator.

[0152] The KV cache generated by the generative large model during processing of the first session is stored in a first buffer. When the accelerator performs layer (i+1) calculations through a computing thread, the accelerator's data write-back thread writes the KV cache generated by the layer (i) calculations from the first buffer back to the host's memory, where i is an integer greater than or equal to 1 and less than N. The third buffer is an area in the accelerator's HBM for storing KV cache that has not yet been written back.

[0153] In the above method, the HBM of the accelerator also includes a third buffer. After the accelerator completes processing of the first session, the KV cache generated in the process of processing the first session and not yet written back to the memory is written from the first buffer to the third buffer. Since the data copy efficiency between the first buffer and the third buffer is relatively high, after the accelerator completes processing of the first session, the accelerator can quickly write the KV cache in the first buffer that has not yet been written back to the memory to the third buffer, and then can quickly release the first buffer, so that the KV cache required for subsequent processing can be loaded into the first buffer without waiting for the KV cache generated by processing the first session to be completely written to the memory before releasing the first buffer. This hides the time gap between the two sessions caused by waiting for the data of the previous session to be written back, thereby improving the efficiency of inference.

[0154] It should be noted that step 307 is optional. In some embodiments, the accelerator includes only a first buffer. After the accelerator completes the N-th layer calculation for the M-th iteration of the first session, the first buffer is released after all the KV cache generated by the first session to be processed is written from the first buffer to the host's memory. In the above embodiment, the accelerator's HBM is used entirely as the first buffer, which can increase the capacity of the first buffer. When the amount of historical KV cache data for a session is large, it can accommodate the entire historical KV cache of the session. This can prevent the situation where the later-loaded historical KV cache overwrites the earlier-loaded historical KV cache due to insufficient capacity, thereby improving the hit rate of the historical KV cache in the first buffer and avoiding repeated loading of the historical KV cache.

[0155] Step 308: The accelerator processes the second session in the same manner as steps 301 to 307 above, and writes the KV cache in the third buffer of the accelerator into the memory of the host.

[0156] In step 308, the process of the accelerator writing the KV cache in the third buffer into the host's memory is similar to the process of the accelerator writing the KV cache calculated at the first level of the first iteration into the host's memory in step 303, and will not be repeated here.

[0157] In the above method, the accelerator can preload the historical KV cache required for the i+1th layer calculation from the host's memory into the accelerator while performing the i-th layer calculation of the generative large model for the session. Since the calculation process is synchronized with the data loading process, the calculation process does not need to wait for the data loading to be completed, thereby hiding the time overhead of the accelerator accessing the external storage device and improving the efficiency of multi-round session reasoning; further, while the accelerator is performing the i+1th layer calculation, the KV cache generated by the i-th layer calculation is written back to the host's memory. Compared with writing the KV cache generated for the current round of session reasoning into the host's memory after the current round of session reasoning is completed, this can hide the time gap between two adjacent rounds of session caused by writing back the data generated by the previous round of session, thereby improving the efficiency of reasoning.

[0158] The following is an exemplary description of the process shown in steps 301 to 308 above using Figures 6 and 7. Figure 6 is a schematic diagram of the parallel calculation process and data loading process in a multi-round conversation reasoning method provided by an embodiment of the present application. As shown in Figure 6, the generative large model includes three layers (L1, L2 and L3). After the previous conversation is processed, the accelerator processes the current conversation through the generative large model. Before the calculation of the L1 layer occurs, the KV cache required for the layer is ready in the first buffer in the HBM of the accelerator. The data loading thread of the accelerator, that is, the KV cache read flow, loads the KV cache required for the L1 layer calculation into the first buffer; the calculation thread of the accelerator, that is, the execution flow, starts the calculation of the L1 layer; when the calculation thread of the accelerator is performing the calculation of the L1 layer, the data loading thread simultaneously starts to load the KV cache required for the L2 layer calculation into the first buffer, and so on, so that the time for the accelerator to load the KV cache from the host's memory overlaps with the calculation time, thereby hiding the time overhead of loading the KV cache from the host's memory and significantly improving the reasoning efficiency.

[0159] Figure 7 is a schematic diagram of the parallel calculation process and data write-back process in a multi-round conversational reasoning method provided by an embodiment of the present application. As shown in Figure 7, the generative large model includes three layers (L1, L2 and L3). In the pre-filling stage, the accelerator's computing thread, that is, the execution flow, performs L1 layer calculations; after the accelerator's computing thread completes the L1 layer calculations, the accelerator's computing thread performs L2 layer calculations. At the same time, the accelerator's data write-back thread, that is, the KV cache write-back flow, writes the KV cache generated by the L1 layer calculations back from the first buffer to the host's memory; after the accelerator's computing thread completes the L2 layer calculations, the accelerator's computing thread performs L3 layer calculations. At the same time, the accelerator's data write-back thread writes the KV cache generated by the L2 layer calculations back from the first buffer to the host's memory, and so on; the data write-back process in the decoding stage is the same as the data write-back process in the pre-filling stage and will not be repeated here. In the data write-back process shown in FIG7 , the time for the accelerator to write the generated KV cache back to the host's memory overlaps with the computation time, thereby hiding the time overhead of writing the generated KV cache back to the host's memory and significantly improving inference efficiency.

[0160] The above describes the data migration process between the accelerator and the host memory. The following describes the data migration process between the host memory and the external storage device.

[0161] The data migration process between the host's memory and the external storage device includes: if the historical KV cache of the first pending session in the task queue does not exist in the host's memory, the host loads the historical KV cache of the first pending session from the external storage device into the host's memory, where the first pending session is the first-numbered pending session at the head of the task queue.

[0162] Among them, the number of session rounds included in the first session to be processed is determined based on the capacity of the host's memory, that is, the first number is determined based on the capacity of the host's memory. For example, the first number = the available storage space of the host's memory ÷ the average data volume of the historical KV cache of the session. It should be noted that for the historical KV cache of the first session being processed by the accelerator, the accelerator writes the newly generated KV cache back to the host's memory while performing calculations to update the historical KV cache of the first session in the host's memory. The space occupied by the historical KV cache of the first session in the host's memory belongs to the system occupation, and the space cannot be released before the first session processing is completed. Therefore, the available storage space of the host's memory = the host's memory capacity - the capacity of the space occupied by the system.

[0163] In some embodiments, if the available storage space in the host's memory is sufficient, the host loads the historical KV cache of the first pending session from the external storage device to the host's memory; if the available storage space in the host's memory is insufficient, the host writes the historical KV cache of the second pending session in the task queue from the host's memory to the external storage device, and loads the historical KV cache of the first pending session from the external storage device to the host's memory, where the second pending session is the session in the task queue other than the first pending session.

[0164] Among them, those skilled in the art can determine how to judge whether the available storage space of the host's memory is sufficient according to actual needs. For example, in some embodiments, if the available storage space capacity of the host's memory is greater than or equal to the target threshold, it means that the available storage space of the host is sufficient; if the available storage space capacity of the host's memory is less than the target threshold, it means that the available storage space of the host is insufficient. For another example, in some other embodiments, if the available storage space capacity of the host's memory is greater than or equal to the amount of data in the historical KV cache of the first session to be processed, it means that the available storage space of the host is sufficient; if the available storage space capacity of the host's memory is less than the amount of data in the historical KV cache of the first session to be processed, it means that the available storage space of the host is insufficient. It should be noted that the above description of the method for judging whether the available storage space of the host's memory is sufficient is only exemplary, and the embodiments of the present application do not limit this judgment method.

[0165] In some embodiments, in addition to the available storage space in the host's memory, the host's memory further includes a fourth buffer zone, which is used to store historical KV caches loaded from an external storage device when the available storage space in the host's memory is insufficient. In this embodiment, the host's memory includes the fourth buffer zone, ensuring sufficient migration space when migrating the KV cache from the external storage device to the host's memory. Thus, when the host's available storage space is insufficient, the host can write the KV cache preloaded from the external storage device into the fourth buffer zone without having to wait for the KV cache in the host's memory to be evicted to the external storage device, thereby avoiding congestion caused by waiting for the host's memory to evict the KV cache when the host's available storage space is insufficient.

[0166] In some embodiments, if the available storage space on the external storage device is insufficient, the historical KV cache of the third pending session in the task queue is deleted from the external storage device. The third pending session is the second number of pending sessions at the tail of the task queue. The second number can be determined based on actual conditions, for example, 1 or 2, etc., and is not limited in this embodiment of the present application.

[0167] The following is an example of the data migration process between the host's memory and the external storage device using Figure 8. Figure 8 is a schematic diagram of the data migration process between the host's memory and the external storage device in a multi-round conversation reasoning process provided by an embodiment of the present application. As shown in Figure 8, the accelerator is processing Job1, which is also the first conversation; the task queue includes 8 pending conversations, from the head to the tail, Job2 to Job9; the host's memory includes a fourth buffer (the area indicated by "buf" in Figure 8), and the host's memory stores the historical KV caches of Job1, Job2, and Job4, which are KV1, KV2, and KV4, respectively; the external storage device is a disk, which stores the historical KV caches of Job9, Job8, Job7, and Job3, which are KV9, KV8, KV7, and KV3, respectively. The task scheduler in the host maintains a prefetching window, which is used to determine the first number of pending sessions at the head of the task queue, that is, the first pending session. In Figure 8, the size of the prefetching window is 2, indicating that the first pending session includes two rounds of sessions, that is, the first number is 2. As shown in Figure 8, the first pending sessions are Job2 and Job3. The task scheduler in the host also maintains an eviction window. When the available storage space of the external storage device is insufficient, the sessions indicated by the eviction window will be exempted from deletion from the external storage device. In Figure 8, the size of the eviction window is 6. As shown in Figure 8, in the first pending session (Job2 and Job3), the historical KV cache (KV2) of Job2 exists in the host's memory, the historical KV cache (KV3) of Job3 does not exist in the host's memory, and the available storage space in the host's memory is insufficient. The host then loads KV3 from the disk into the fourth buffer in the host's memory, and writes the historical KV cache of the second pending session in the task queue from the host's memory to the disk. The historical KV cache of the second pending session exists in the host's memory, and the second pending session is not the first pending session. The second pending session is not the first pending session because the second pending session is outside the front view window. In Figure 8, the second pending session is Job4, and the host writes the historical KV cache (KV4) of job4 from the host's memory to the disk. Due to insufficient available disk storage space, the host deletes the historical KV cache of the third pending session from the disk. The third pending session is the second-to-last session in the task queue outside the elimination exemption window. In Figure 8 , the second number is 1, so the third pending session is Job 9. The host deletes the historical KV cache of Job 9 (KV 9) from the disk.It should be noted that FIG8 is only an example of the data migration process between the host memory and the external storage device, and does not limit the present application.

[0168] In the above method, the external storage device stores the historical KV cache of completed sessions. Since the capacity of the external storage device is much larger than the capacity of the HBM in the accelerator, the external storage device can save a large amount of historical KV cache, thereby improving the hit rate of the historical KV cache, thereby avoiding KV cache recalculation, improving the efficiency of multi-round conversation reasoning, and saving computing resources; at the same time, the host can pre-load the historical KV cache of the pending sessions in the task queue from the external storage device to the host's memory while the accelerator is processing the session. Since the calculation process is carried out synchronously with the data loading process, the calculation process does not need to wait for the data loading to be completed, thereby hiding the time overhead of the accelerator accessing the external storage device and improving the efficiency of multi-round conversation reasoning.

[0169] It should be noted that, in order to facilitate the introduction of the multi-round conversation reasoning method provided in the embodiment of the present application, two embodiments are used to respectively introduce the data migration process between the accelerator and the host's memory and the data migration process between the host's memory and the external storage device. However, this does not mean that the above two embodiments are independent of each other and occur separately. That is, in the multi-round conversation reasoning method provided in the embodiment of the present application, while data migration is performed between the accelerator and the host's memory, data migration can also occur between the host's memory and the external storage device.

[0170] Figure 9 is a structural schematic diagram of a multi-round conversation reasoning device provided in an embodiment of the present application, which is an accelerator applied to a host in a computing system. The computing system also includes a host and an external storage device. The external storage device is used to store the historical key-value cache KV cache of the session. The host is used to load the historical KV cache of the session to be processed in the task queue from the external storage device to the host's memory. The device includes a processing module 901 and a loading module 902.

[0171] The processing module 901 is configured to perform reasoning on a first session in the task queue using a generative large model based on the task queue, where the first session is one of multiple rounds of sessions, and the generative large model includes N layers.

[0172] The loading module 902 is configured to load the input of the first session at the i+1th level historical KV cache from the memory when performing the i-th level calculation on the first session, where i is an integer greater than or equal to 1 and less than N.

[0173] In some embodiments, the historical KV cache of the input of the first session at the (i+1) level is independent of the position encoding of the input of the first session.

[0174] In some embodiments, the number of word-grams included in the input of the first session is greater than or equal to the number of word-grams that can be accommodated by the context window of the generative large model;

[0175] The loading module 902 is used to:

[0176] When performing the i-th level calculation on the first session, a first sub-input is loaded from the memory into the historical KV cache at the i+1 level, where the first sub-input is a sub-input of the input of the first session, and the number of tokens included in the first sub-input is less than the number of tokens that can be accommodated by the context window.

[0177] In some embodiments, the processing module 901 includes:

[0178] An embedding unit, configured to embed the position code of the first sub-input into the history KV cache of the first sub-input at the (i+1) layer;

[0179] The computing unit is configured to perform an i+1th level computing on the first session based on the input of the first session and the first sub-input after embedding position encoding in the i+1th level historical KV cache.

[0180] In some embodiments, the accelerator includes a first buffer and a second buffer, the first buffer being used to store a history KV cache of the first session, and the second buffer being used to store the history KV cache of the input of the second session in the task queue at the first layer, the second session being the first session to be processed after the first session in the task queue;

[0181] The loading module 902 is used to:

[0182] When performing the i-th layer calculation on the first session, the historical KV cache of the input of the first session at the i+1th layer is loaded from the memory into the first buffer.

[0183] In some embodiments, the loading module 902 is further configured to:

[0184] When performing the Nth layer calculation on the first session, the historical KV cache of the input of the second session at the first layer is loaded from the memory to the second buffer.

[0185] In some embodiments, the apparatus further comprises:

[0186] The writing module is configured to write the KV cache generated by performing the i-th layer calculation on the first session into the memory when performing the i+1-th layer calculation on the first session.

[0187] In some embodiments, the accelerator includes a third buffer, the third buffer being used to store the KV cache generated by the generative large model processing the first session;

[0188] The write module is used to:

[0189] If, upon completion of the Nth layer calculation of the first session, there is a KV cache in the KV cache generated by the generative large model for processing the first session that has not been written into the memory, the KV cache that has not been written into the memory is written into the third buffer;

[0190] When the second session in the task queue is processed by the generative large model, the KV cache in the third buffer is written into the memory.

[0191] Figure 10 is a structural schematic diagram of a multi-round conversation reasoning device provided by an embodiment of the present application, which is applied to a host in a computing system. The computing system also includes an accelerator of the host and an external storage device. The external storage device is used to store the historical key-value cache KV cache of the session. The accelerator is used to reason about the first conversation in the task queue based on the task queue through a generative large model. The first conversation is one round in the multi-round conversation. The generative large model includes N layers. When the i-th layer calculation is performed on the first conversation, the input of the first conversation is loaded from the memory into the historical KV cache at the i+1 layer, where i is an integer greater than or equal to 1 and less than N. The device includes a loading module 1001 and a storage module 1002.

[0192] The loading module 1001 is used to load the historical KV cache of the pending sessions in the task queue from the external storage device to the memory of the host;

[0193] The storage module 1002 is configured to store the history KV cache of the session to be processed.

[0194] In some embodiments, the loading module 1001 includes:

[0195] The loading unit is configured to load the historical KV cache of the first to-be-processed session in the task queue from the external storage device to the memory if the historical KV cache of the first to-be-processed session does not exist in the memory, where the first to-be-processed session is the first to-be-processed session at the head of the task queue.

[0196] In some embodiments, the number of session rounds included in the first to-be-processed session is determined based on the capacity of the memory.

[0197] In some embodiments, the loading unit is configured to:

[0198] If the available storage space of the memory is sufficient, loading the historical KV cache of the first session to be processed from the external storage device into the memory;

[0199] If the available storage space in the memory is insufficient, the historical KV cache of the second pending session in the task queue is written from the memory to the external storage device, and the historical KV cache of the first pending session is loaded from the external storage device to the memory. The second pending session is the session in the task queue other than the first pending session.

[0200] In some embodiments, the apparatus further includes a deletion module configured to:

[0201] If the available storage space of the external storage device is insufficient, the historical KV cache of the third pending session in the task queue is deleted from the external storage device. The third pending session is the second-to-last pending session in the task queue.

[0202] The processing module 901, the loading module 902, the loading module 1001, and the storage module 1002 can all be implemented by software or hardware. For example, the implementation of the processing module 901 will be described below using the processing module 901 as an example. Similarly, the implementation of the loading module 902, the loading module 1001, and the storage module 1002 can refer to the implementation of the processing module 901.

[0203] As an example of a software functional unit, the processing module 901 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the processing module 901 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0204] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0205] As an example of a hardware functional unit, processing module 901 may include at least one computing device, such as a server. Alternatively, processing module 901 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0206] The multiple computing devices included in processing module 901 can be distributed in the same region or in different regions. The multiple computing devices included in processing module 901 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in processing module 901 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0207] It should be noted that, in other embodiments, the processing module 901 can be used to execute any step in the multi-round conversation reasoning method, the loading module 902 can be used to execute any step in the multi-round conversation reasoning method, the loading module 1001 can be used to execute any step in the multi-round conversation reasoning method, and the storage module 1002 can be used to execute any step in the multi-round conversation reasoning method. The steps that the processing module 901, the loading module 902, the loading module 1001 and the storage module 1002 are responsible for implementing can be specified as needed. The processing module 901 and the loading module 902 respectively implement different steps in the multi-round conversation reasoning method executed by the accelerator to realize all the functions of the multi-round conversation reasoning device shown in Figure 9; or, the loading module 1001 and the storage module 1002 respectively implement different steps in the multi-round conversation reasoning method executed by the host to realize all the functions of the multi-round conversation reasoning device shown in Figure 10.

[0208] The present application also provides a computing device 1100. FIG11 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. As shown in FIG11 , the computing device 1100 includes a bus 1101, a processor 1102, a memory 1103, and a communication interface 1104. The processor 1102, the memory 1103, and the communication interface 1104 communicate with each other via the bus 1101. The computing device 1100 can be a computing device or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 1100.

[0209] Bus 1101 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG11 shows a single line, but this does not imply a single bus or type of bus. Bus 1101 may include a path for transmitting information between various components of computing device 1100 (e.g., memory 1103, processor 1102, and communication interface 1104).

[0210] The processor 1102 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0211] The memory 1103 may include a volatile memory, such as a random access memory (RAM). The memory 1103 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0212] Memory 1103 stores executable program code, which processor 1102 executes to implement the functions of processing module 901 and loading module 902, respectively, thereby implementing the steps of the multi-turn conversational reasoning method performed by the accelerator. Alternatively, it implements the functions of loading module 1001 and storage module 1002, thereby implementing the steps of the multi-turn conversational reasoning method performed by the host. In other words, memory 1103 stores instructions for executing the multi-turn conversational reasoning method. Figure 11 only exemplifies that memory 1103 stores program code that implements the functions of processing module 901 and loading module 902.

[0213] The communication interface 1104 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1100 and other devices or a communication network.

[0214] The present application also provides a computing device cluster. Figure 12 is a schematic diagram of a computing device cluster provided in an embodiment of the present application. As shown in Figure 12, the computing device cluster includes at least one computing device 1100. The memory 1103 of one or more computing devices 1100 in the computing device cluster may store the same instructions for executing the multi-turn conversational reasoning method.

[0215] In some possible implementations, the memory 1103 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for executing the multi-turn conversational reasoning method. In other words, the combination of one or more computing devices 1100 can jointly execute the instructions for executing the multi-turn conversational reasoning method.

[0216] It should be noted that the memory 1103 in different computing devices 1100 in the computing device cluster can store different instructions, each used to perform part of the functions of the multi-turn conversational reasoning apparatus. In other words, the instructions stored in the memory 1103 in different computing devices 1100 can implement the functions of one or more of the aforementioned processing module 901, loading module 902, loading module 1001, and storage module 1002.

[0217] It should be understood that the functions of the computing device 1100 shown in FIG. 12 may also be performed by multiple computing devices 1100 .

[0218] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network (WAN) or a local area network (LAN), among others. FIG. 13 illustrates one possible implementation. FIG. 13 is a schematic diagram of a possible implementation of a computing device cluster provided by an embodiment of the present application. As shown in FIG. 13 , computing device 1100A and computing device 1100B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 1103 in computing device 1100A stores instructions for executing the functions of processing module 901. FIG. 13 takes the example of the memory 1103 in computing device 1100A storing instructions for executing the functions of processing module 901. At the same time, the memory 1103 in computing device 1100B stores instructions for executing the functions of loading module 902. FIG. 13 takes the example of the memory 1103 in computing device 1100B storing instructions for executing the functions of loading module 902.

[0219] The present application also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection method shown in Figures 12 or 13. The difference is that the memory 1103 in one or more computing devices 1100 in the computing device cluster can store the same instructions for executing the multi-round conversation reasoning method.

[0220] In some possible implementations, the memory 1103 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for executing the multi-turn conversational reasoning method. In other words, the combination of one or more computing devices 1100 can jointly execute the instructions for executing the multi-turn conversational reasoning method.

[0221] It should be noted that the memory 1103 in different computing devices 1100 in the computing device cluster can store different instructions, each used to perform part of the functions of the multi-turn conversational reasoning apparatus. In other words, the instructions stored in the memory 1103 in different computing devices 1100 can implement the functions of one or more of the aforementioned processing module 901, loading module 902, loading module 1001, and storage module 1002.

[0222] An embodiment of the present application provides an accelerator, which includes a computing core and a memory. The accelerator is used to execute the steps performed by the accelerator in the multi-round conversational reasoning method provided by any possible implementation method in the aforementioned method embodiment. The memory is used to store computing data, and the computing core is used to perform computing operations on the computing data stored in the memory.

[0223] An embodiment of the present application provides a host, which includes a processor and a memory, and is used to execute the steps performed by the host in the multi-round conversation reasoning method provided by any possible implementation manner in the aforementioned method embodiment.

[0224] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a multi-turn conversational reasoning method.

[0225] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the multi-round conversational reasoning method.

[0226] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, and display, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the sessions and storage space involved in this application are obtained with full authorization.

[0227] Those skilled in the art will appreciate that the various method steps and units described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0228] Those skilled in the art will clearly understand that, for the sake of convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0229] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the unit is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0230] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0231] In addition, each unit in each embodiment of the present application may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software units.

[0232] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computing device (which can be a personal computer, a server, or a computing device, etc.) to execute all or part of the steps of the method in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0233] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there a limit on quantity and execution order. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various examples, a first session may be referred to as a second session, and similarly, a second session may be referred to as a first session. Both the first session and the second session may be node sessions, and in some cases, may be separate and different sessions.

[0234] The term "at least one" in this application means one or more, and the term "plurality" in this application means two or more. The terms "system" and "network" are often used interchangeably in this document.

[0235] It should also be understood that the term "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined that..." or "if [stated condition or event] is detected" may be interpreted to mean "upon determining that..." or "in response to determining that..." or "upon detecting [stated condition or event]" or "in response to detecting [stated condition or event]," depending on the context.

[0236] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0237] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer program instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0238] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless method. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital video disc (DVD), or a semiconductor medium (e.g., a solid-state drive)), etc.

[0239] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0240] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A computing system, characterized in that: The computing system includes a host, an accelerator of the host, and an external storage device; wherein, The external storage device is used to store the historical key-value cache KV cache of the session; The host is configured to load the historical KV cache of the pending sessions in the task queue from the external storage device to the memory of the host; The accelerator is configured to perform reasoning on a first session in the task queue using a generative large model based on the task queue, where the first session is one of multiple rounds of sessions, and the generative large model includes N layers, wherein: When performing the i-th level calculation on the first session, the input of the first session is loaded from the memory into the history KV cache at the i+1th level, where i is an integer greater than or equal to 1 and less than N.

2. The computing system according to claim 1, wherein: The historical KV cache of the input of the first session at the (i+1) layer has nothing to do with the position encoding of the input of the first session.

3. The computing system according to claim 2, wherein: The number of word-grams included in the input of the first conversation is greater than or equal to the number of word-grams that can be accommodated by the context window of the generative large model; The accelerator is used to: When performing an i-th level calculation on the first session, a first sub-input is loaded from the memory into the historical KV cache at the i+1 level, where the first sub-input is a sub-input of the input of the first session, and the number of tokens included in the first sub-input is smaller than the number of tokens that can be accommodated by the context window.

4. The computing system according to claim 3, wherein: The accelerator is also used to: Embed the position code of the first sub-input into the history KV cache of the first sub-input at the i+1th layer; Based on the input of the first session and the first sub-input after embedding position encoding in the historical KV cache at the i+1 level, the i+1 level calculation is performed on the first session. The computing system according to claim 1 , wherein: The accelerator includes a first buffer and a second buffer, the first buffer being used to store a history KV cache of the first session, and the second buffer being used to store a history KV cache of the second session input in the task queue at the first layer, the second session being the first session to be processed after the first session in the task queue; The accelerator is used to: When performing layer i calculation on the first session, load the historical KV cache of the input of the first session at layer i+1 from the memory into the first buffer; When performing the Nth layer calculation on the first session, the historical KV cache of the second session input at the first layer is loaded from the memory to the second buffer. The computing system according to claim 1 , wherein: The accelerator is also used to: When performing the (i+1)th layer calculation on the first session, the KV cache generated by performing the i-th layer calculation on the first session is written into the memory.

7. The computing system according to claim 6, wherein: The accelerator includes a third buffer, the third buffer being used to store a KV cache generated by the generative large model processing the first session; The host's accelerator is used to: If, upon completion of the Nth layer calculation for the first session, there is a KV cache in the KV cache generated by the generative large model for processing the first session that has not been written into the memory, writing the KV cache that has not been written into the memory into the third buffer; When the second session in the task queue is processed by the generative large model, the KV cache in the third buffer is written into the memory.

8. The computing system according to claim 1, wherein: The host is used to: If the historical KV cache of the first session to be processed in the task queue does not exist in the memory, the historical KV cache of the first session to be processed is loaded from the external storage device into the memory, where the first session to be processed is the first number of sessions to be processed at the head of the task queue.

9. The computing system according to claim 8, wherein: The number of session rounds included in the first session to be processed is determined based on the capacity of the memory.

10. The computing system according to claim 8 or 9, characterized in that The host is used to: If the available storage space of the memory is sufficient, loading the history KV cache of the first session to be processed from the external storage device into the memory; If the available storage space in the memory is insufficient, the historical KV cache of the second pending session in the task queue is written from the memory to the external storage device, and the historical KV cache of the first pending session is loaded from the external storage device to the memory, where the second pending session is a session in the task queue other than the first pending session.

11. The computing system according to claim 1, wherein: The host is also used to: If the available storage space of the external storage device is insufficient, the historical KV cache of the third pending session in the task queue is deleted from the external storage device, where the third pending session is the second-to-last pending session in the task queue.

12. A multi-round conversation reasoning method, characterized in that: The method is applied to a computing system including a host, an accelerator of the host, and an external storage device, wherein the external storage device is used to store a historical key-value cache (KV cache) of sessions, and the host is used to load the historical KV cache of pending sessions in a task queue from the external storage device to a memory of the host, and the method includes: The accelerator is used to perform reasoning on a first session in the task queue using a generative large model based on the task queue, where the first session is one of multiple rounds of sessions, and the generative large model includes N layers, wherein: When performing layer i calculation on the first session through the accelerator, the historical KV cache of the input of the first session at layer i+1 is loaded from the memory to the accelerator, where i is an integer greater than or equal to 1 and less than N.

13. A multi-round conversation reasoning method, characterized in that: An accelerator for a host in a computing system, the computing system further comprising the host and an external storage device, the external storage device being used to store a historical key-value cache (KV cache) of sessions, the host being used to load the historical KV cache of pending sessions in a task queue from the external storage device into a memory of the host, the method comprising: Based on the task queue, reasoning is performed on a first session in the task queue using a generative big model, where the first session is one round of multiple rounds of sessions, and the generative big model includes N layers, wherein: When performing the i-th level calculation on the first session, the input of the first session is loaded from the memory into the history KV cache at the i+1th level, where i is an integer greater than or equal to 1 and less than N.

14. The method according to claim 13, characterized in that The historical KV cache of the input of the first session at the (i+1) layer has nothing to do with the position encoding of the input of the first session.

15. The method according to claim 14, characterized in that The number of word-grams included in the input of the first conversation is greater than or equal to the number of word-grams that can be accommodated by the context window of the generative large model; The step of loading the input of the first session into the historical KV cache at the i+1 level from the memory when performing the i-th level calculation on the first session includes: When performing an i-th level calculation on the first session, a first sub-input is loaded from the memory into the historical KV cache at the i+1 level, where the first sub-input is a sub-input of the input of the first session, and the number of tokens included in the first sub-input is smaller than the number of tokens that can be accommodated by the context window.

16. The method according to claim 15, characterized in that The processing of the first session in the task queue by using the generative large model includes: Embed the position code of the first sub-input into the history KV cache of the first sub-input at the i+1th layer; Based on the input of the first session and the first sub-input after embedding position encoding in the historical KV cache at the i+1 level, the i+1 level calculation is performed on the first session.

17. The method according to claim 13, wherein The accelerator includes a first buffer and a second buffer, the first buffer being used to store a history KV cache of the first session, and the second buffer being used to store a history KV cache of the second session input in the task queue at the first layer, the second session being the first session to be processed after the first session in the task queue; The step of loading the input of the first session into the historical KV cache at the i+1 level from the memory when performing the i-th level calculation on the first session includes: When performing the i-th layer calculation on the first session, the historical KV cache of the input of the first session at the i+1th layer is loaded from the memory into the first buffer.

18. The method according to claim 17, characterized in that The method further comprises: When performing the Nth layer calculation on the first session, the historical KV cache of the second session input at the first layer is loaded from the memory to the second buffer.

19. The method according to claim 13, wherein The method further comprises: When performing the (i+1)th layer calculation on the first session, the KV cache generated by performing the i-th layer calculation on the first session is written into the memory.

20. The method according to claim 19, characterized in that The accelerator includes a third buffer, the third buffer being used to store a KV cache generated by the generative large model processing the first session; The step of writing, when performing the i+1th layer calculation on the first session, a KV cache generated by performing the i-th layer calculation on the first session into the memory includes: If, upon completion of the Nth layer calculation for the first session, there is a KV cache in the KV cache generated by the generative large model for processing the first session that has not been written into the memory, writing the KV cache that has not been written into the memory into the third buffer; When the second session in the task queue is processed by the generative large model, the KV cache in the third buffer is written into the memory.

21. A multi-round conversation reasoning method, characterized in that: A host computer in a computing system, the computing system further comprising an accelerator of the host computer and an external storage device, the external storage device being used to store a historical key-value cache (KV cache) of sessions; the accelerator being used to perform inference on a first session in the task queue using a generative large model based on a task queue, the first session being one of multiple rounds of sessions, the generative large model comprising N layers, wherein when performing an i-th layer calculation on the first session, the historical KV cache at the i+1th layer of the input of the first session is loaded from the memory of the host computer, where i is an integer greater than or equal to 1 and less than N; The method comprises: Loading the historical KV cache of the pending sessions in the task queue from the external storage device to the host's memory; A historical KV cache storing the session to be processed.

22. The method according to claim 21, characterized in that The step of loading the historical KV cache of the pending sessions in the task queue from the external storage device to the memory of the host includes: If the historical KV cache of the first session to be processed in the task queue does not exist in the memory, the historical KV cache of the first session to be processed is loaded from the external storage device into the memory, where the first session to be processed is the first number of sessions to be processed at the head of the task queue.

23. The method according to claim 22, characterized in that The number of session rounds included in the first session to be processed is determined based on the capacity of the memory.

24. The method according to claim 22 or 23, characterized in that If the historical KV cache of the first session to be processed in the task queue does not exist in the memory, the host loads the historical KV cache of the first session to be processed from the external storage device to the memory, including: If the available storage space of the memory is sufficient, loading the history KV cache of the first session to be processed from the external storage device into the memory; If the available storage space in the memory is insufficient, the historical KV cache of the second pending session in the task queue is written from the memory to the external storage device, and the historical KV cache of the first pending session is loaded from the external storage device to the memory, where the second pending session is a session in the task queue other than the first pending session.

25. The method according to any one of claims 21 to 24, characterized in that The method further comprises: If the available storage space of the external storage device is insufficient, the historical KV cache of the third pending session in the task queue is deleted from the external storage device, where the third pending session is the second-to-last pending session in the task queue.

26. A multi-round conversation reasoning device, characterized in that: An accelerator for a host in a computing system, the computing system also including the host and an external storage device, the external storage device being used to store a historical key-value cache (KV cache) of sessions, the host being used to load the historical KV cache of pending sessions in a task queue from the external storage device into a memory of the host, the apparatus comprising: a processing module, configured to perform reasoning on a first session in the task queue using a generative large model based on the task queue, where the first session is one round of multiple rounds of sessions, and the generative large model includes N layers; The loading module is configured to load the historical KV cache of the input of the first session at the i+1th level from the memory when performing the i-th level calculation on the first session, where i is an integer greater than or equal to 1 and less than N.

27. A multi-round conversation reasoning device, characterized in that: A host computer in a computing system, the computing system further comprising an accelerator of the host computer and an external storage device, the external storage device being used to store a historical key-value cache (KV cache) of sessions; the accelerator being used to perform inference on a first session in the task queue using a generative large model based on a task queue, the first session being one of multiple rounds of sessions, the generative large model comprising N layers, wherein when performing an i-th layer calculation on the first session, the historical KV cache at the i+1th layer of the input of the first session is loaded from the memory of the host computer, where i is an integer greater than or equal to 1 and less than N; The device comprises: The loading module loads the historical KV cache of the to-be-processed sessions in the task queue from the external storage device to the memory of the host.

28. An accelerator, characterized in that: The accelerator includes a computing core and a memory, and the accelerator is used to execute the multi-round conversation reasoning method of any one of claims 13 to 20, the memory is used to store computing data, and the computing core is used to perform computing operations on the computing data stored in the memory.

29. A host, characterized in that: The host includes a processor and a memory, and is used to execute any one of the multi-round conversation reasoning methods in claims 21 to 25.

30. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, each of which includes a host and an accelerator of the host. An external storage device is located in at least one computing device, the external storage device is used to store a historical KV cache of a session, and the host is used to load the historical KV cache of a session to be processed in a task queue from the external storage device to a memory of the host. The host in the computing device cluster is used to execute program code to implement the steps performed by the host in the multi-round conversation reasoning method according to any one of claims 13 to 20; The accelerator in the computing device cluster is used to execute program code to implement the steps performed by the accelerator in the multi-round conversation reasoning method as described in any one of claims 21 to 25.

31. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the multi-round conversation reasoning method as described in any one of claims 13 to 20 or claims 21 to 25.

32. A computer-readable storage medium, characterized in that The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the multi-round conversation reasoning method according to any one of claims 13 to 20 or claims 21 to 25.

Images