Data processing method, processor, electronic device, and storage medium

Abstract

A data processing method, a processor, an electronic device and a storage medium. The data processing method comprises: pre-allocating a target storage space in the device; in response to receiving a kth inference request, performing a first inference on kth input data using a persistent kernel function to obtain an initial first cache-type intermediate result, and storing the initial first cache-type intermediate result in the target storage space; using the persistent kernel function to perform autoregressive inference of the kth inference request using the first initial data unit in the kth input data and the initial first cache-type intermediate result as input to obtain a kth output result and a plurality of first cache-type intermediate results, and storing the kth output result and the plurality of first cache-type intermediate results in the target storage space. The data processing method effectively solves the performance bottleneck of repeatedly starting the kernel function in large language model inference, significantly improving the computing efficiency and memory utilization.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computers, and more specifically to a data processing method, processor, electronic device, and storage medium. Background Technology

[0002] The rapid development of technologies such as Artificial Intelligence (AI), text processing, speech processing, and image processing has led to increasingly higher demands on computing performance and data processing efficiency. Current computing devices widely utilize processors such as Central Processing Units (CPUs), Graphics Processing Units (GPUs), and General-purpose Computing on Graphics Processing Units (GPGPUs). By processing different data on different processors and performing data migration, transfer, and storage between different processors, the computing performance of computing devices has been improved. Summary of the Invention

[0003] This disclosure provides at least one embodiment of a data processing method applied to a device, wherein a persistent kernel function runs on the device. During the startup and exit process of the persistent kernel function, the persistent kernel function performs a corresponding autoregressive inference operation on each of N inference requests, where N is an integer greater than 1. The data processing method includes: pre-allocating target storage space on the device; in response to receiving the k-th inference request among the N inference requests, performing an initial inference on the k-th input data of the k-th inference request using the persistent kernel function to obtain an initial first cached intermediate result, and storing the initial... The first cached intermediate result is stored in the target storage space, where k is a positive integer and not greater than N; the initial first cached intermediate result is used as the input to the autoregressive inference of the k-th inference request, and the autoregressive inference of the k-th inference request is executed using the persistent kernel function to obtain the k-th output result and multiple first cached intermediate results, and the k-th output result and multiple first cached intermediate results are stored in the target storage space, wherein, during the autoregressive inference process of the k-th inference request, the initial first cached intermediate result and multiple first cached intermediate results are used to calculate the k-th output result.

[0004] For example, in a data processing method provided in at least one embodiment of this disclosure, the target storage space includes a first buffer and an intermediate cache. The k-th input data is written into the first buffer. Storing the initial first cache intermediate result into the target storage space includes: storing the initial first cache intermediate result into the intermediate cache. Storing the k-th output result and the plurality of first cache intermediate results into the target storage space includes: storing the k-th output result into the first buffer and storing the plurality of first cache intermediate results into the intermediate cache.

[0005] For example, in a data processing method provided in at least one embodiment of this disclosure, the target storage space includes a first buffer and an intermediate cache. Before performing the first inference on the k-th input data of the k-th inference request using the persistent kernel function, the data processing method further includes: setting a first flag bit for the first buffer; in response to receiving the k-th inference request among the N inference requests, performing the first inference on the k-th input data of the k-th inference request using the persistent kernel function, including: in response to the k-th input data being written to the first buffer and the first flag bit being in a host-side ready state, performing the first inference on the k-th input data using the persistent kernel function; in response to the completion of the autoregressive inference of the k-th inference request, the data processing method further includes: updating the first flag bit to a device-side ready state to indicate that the host can read the k-th output result.

[0006] For example, in a data processing method provided in at least one embodiment of this disclosure, the k-th input data includes at least one data unit. In response to receiving the k-th inference request among the N inference requests, the method performs an initial inference on the k-th input data of the k-th inference request using the persistent kernel function, including: in response to receiving the k-th inference request, performing a first calculation on each data unit in the at least one data unit using the persistent kernel function to obtain the initial first cached intermediate result, wherein the initial first cached intermediate result includes the cached intermediate result of each data unit.

[0007] For example, in a data processing method provided in at least one embodiment of this disclosure, the target storage space includes a first buffer and an intermediate cache. Before performing the first inference on the k-th input data of the k-th inference request using the persistent kernel function, the data processing method further includes: setting a first offset pointer in the first buffer according to the size of the k-th input data, and pointing the first offset pointer to the end position of the k-th input data; storing the initial first cache intermediate result in the target storage space, including: storing the initial first cache intermediate result in the intermediate cache, setting a second offset pointer in the intermediate cache, and pointing the second offset pointer to the end of the initial first cache intermediate result.

[0008] For example, in a data processing method provided in at least one embodiment of this disclosure, the target storage space includes a first buffer and an intermediate cache. The first buffer is provided with a first offset pointer, and the intermediate cache is provided with a second offset pointer. In response to the completion of autoregressive inference for the k-th inference request, the data processing method further includes: setting the values ​​of both the first offset pointer and the second offset pointer to 0, so that the first offset pointer points to the starting position of the first buffer and the second offset pointer points to the starting position of the intermediate cache.

[0009] For example, in a data processing method provided in at least one embodiment of this disclosure, the k-th output result includes M data units, the plurality of first cache intermediate results include M first cache intermediate results, the first first cache intermediate result among the plurality of first cache intermediate results includes the initial first cache intermediate result, the initial first cache intermediate result is used as the input of the autoregressive inference of the k-th inference request, the autoregressive inference of the k-th inference request is executed using the persistent kernel function, the k-th output result and the plurality of first cache intermediate results are obtained, and the k-th output result and the plurality of first cache intermediate results are stored in the target storage space, including: using the persistent kernel function to predict the first data unit among the M data units based on the initial first cache intermediate result, and calculating the first data unit. The first cached intermediate result of the first data unit is obtained, and the first cached intermediate result of the first data unit is stored in the target storage space; the persistent kernel function is used to perform a second calculation on the j-th data unit and the first to j-th cached intermediate results of the M data units to obtain the (j+1)-th data unit, and the (j+1)-th data unit is stored in the target storage space; in response to j+1 being less than M, the persistent kernel function is used to perform a third calculation on the (j+1)-th data unit to obtain the (j+1)-th cached intermediate result of the M cached intermediate results, and the (j+1)-th cached intermediate result is stored in the target storage space, where M is an integer greater than 1, and j is an integer not greater than M-1 and greater than 1.

[0010] For example, in a data processing method provided in at least one embodiment of this disclosure, the target storage space includes a first buffer, and a first offset pointer is provided in the first buffer. Storing the (j+1)th data unit in the target storage space includes: placing the (j+1)th data unit at the end of the jth data unit in the first buffer, and incrementing the value of the first offset pointer by one so that the first offset pointer points to the end of the (j+1)th data unit.

[0011] For example, in a data processing method provided in at least one embodiment of this disclosure, the target storage space includes an intermediate cache, and the intermediate cache is provided with a second offset pointer. Storing the (j+1)th first cache intermediate result in the target storage space includes: placing the (j+1)th first cache intermediate result at the end of the jth first cache intermediate result in the intermediate cache, and incrementing the value of the second offset pointer by one so that the second offset pointer points to the end of the (j+1)th first cache intermediate result.

[0012] For example, in the data processing method provided in at least one embodiment of this disclosure, the target storage space further includes a second buffer, which is different from the first buffer. The data processing method further includes: in response to receiving the (k+1)th inference request among N inference requests, performing initial inference on the (k+1)th input data of the (k+1)th inference request using the persistent kernel function to obtain an initial second cache intermediate result, and storing the initial second cache intermediate result in the intermediate cache, wherein the (k+1)th input data is written to the second buffer; using the initial second cache intermediate result as input to the autoregressive inference of the (k+1)th inference request, performing the autoregressive inference of the (k+1)th inference request using the persistent kernel function to obtain a (k+1)th output result and a plurality of second cache intermediate results, storing the (k+1)th output result in the second buffer, and storing the plurality of second cache intermediate results in the intermediate cache, wherein, during the autoregressive inference process of the (k+1)th inference request, the initial second cache intermediate result and the plurality of second cache intermediate results are used to calculate the (k+1)th output result.

[0013] For example, at least one embodiment of the data processing method provided in this disclosure further includes: setting a second flag bit for the second buffer; in response to receiving the (k+1)th inference request among N inference requests, performing the first inference on the (k+1)th input data of the (k+1)th inference request using the persistent kernel function, including: in response to the (k+1)th input data being written to the second buffer and the second flag bit being in a host-side ready state, performing the first inference on the (k+1)th input data using the persistent kernel function; in response to the completion of the autoregressive inference of the (k+1)th inference request, the data processing method further includes: updating the second flag bit to a device-side ready state to indicate that the host can read the (k+1)th output result.

[0014] At least one embodiment of this disclosure also provides a data processing method applied to a host side. The data processing method includes: sending the kth inference request out of N inference requests to a device side, wherein the device side runs a persistent kernel function, the persistent kernel function is used to perform a corresponding autoregressive inference operation on each of the N inference requests, after the persistent kernel function performs the autoregressive inference operation on the kth inference request, the obtained kth output result is stored in a target storage space on the device side, wherein N is an integer greater than 1, and k is a positive integer not greater than N; and reading the kth output result from the target storage space to the host side.

[0015] For example, in a data processing method provided in at least one embodiment of this disclosure, the target storage space includes a first buffer, the first buffer being configured with a first flag, and sending the kth inference request among the N inference requests to the device includes: writing the kth input data of the kth inference request into the first buffer; updating the first flag to a host-side ready state to indicate that the persistent kernel function can execute the autoregressive inference operation of the kth inference request, wherein the kth output result obtained by executing the autoregressive inference operation of the kth inference request is stored in the first buffer, and reading the kth output result from the target storage space to the host includes: in response to the first flag being in a device-side ready state, reading the kth output result from the first buffer to the host.

[0016] For example, in a data processing method provided in at least one embodiment of this disclosure, in response to the k-th output result being read to the host, the data processing method further includes: in response to the N inference requests including a k+2 inference request to be processed, sending the k+2 inference request to the device; updating the first flag bit to a host ready state to indicate that the device can execute the autoregressive inference operation of the k+2 inference request.

[0017] For example, in the data processing method provided in at least one embodiment of this disclosure, the target storage space further includes a second buffer, which is different from the first buffer. The second buffer is provided with a second flag bit. The data processing method further includes: in response to the fact that the N inference requests also include a k+1th inference request to be processed, sending the k+1th inference request to the device, wherein the k+1th input data of the k+1th inference request is written into the second buffer; updating the second flag bit to a host-side ready state to indicate that the persistent kernel function can execute the autoregressive inference operation of the k+1th inference request, wherein after the persistent kernel function executes the autoregressive inference operation of the k+1th inference request, the obtained k+1th output result is stored in the second buffer; and in response to the second flag bit being a device-side ready state, reading the k+1th output result from the second buffer to the host.

[0018] For example, in a data processing method provided in at least one embodiment of this disclosure, in response to the (k+1)th output result being read to the host, the data processing method further includes: in response to the N inference requests including a (k+3)th inference request to be processed, sending the (k+3)th inference request to the device; updating the second flag bit to a host ready state to indicate that the device can execute the autoregressive inference operation of the (k+3)th inference request.

[0019] At least one embodiment of this disclosure also provides a device-side processor, wherein a persistent kernel function runs on the device-side processor. During the startup and exit process of the persistent kernel function, the persistent kernel function performs a corresponding autoregressive inference operation on each of N inference requests, where N is an integer greater than 1. The device-side processor includes: a pre-allocation module configured to: pre-allocate target storage space in the device; and an inference module configured to: in response to receiving the k-th inference request among the N inference requests, perform initial inference on the k-th input data of the k-th inference request using the persistent kernel function to obtain an initial first cached intermediate result, and store the... An initial first cached intermediate result is stored in the target storage space, where k is a positive integer and not greater than N; the inference module is further configured to: use the initial first cached intermediate result as input to the autoregressive inference of the k-th inference request, execute the autoregressive inference of the k-th inference request using the persistent kernel function, obtain the k-th output result and multiple first cached intermediate results, and store the k-th output result and the multiple first cached intermediate results in the target storage space, wherein, during the autoregressive inference process of the k-th inference request, the initial first cached intermediate result and the multiple first cached intermediate results are used to calculate the k-th output result.

[0020] At least one embodiment of this disclosure also provides a host-side processor, including: a sending module configured to send the kth inference request out of N inference requests to a device, wherein the device runs a persistent kernel function, the persistent kernel function is used to perform a corresponding autoregressive inference operation on each of the N inference requests, and after the persistent kernel function performs the autoregressive inference operation on the kth inference request, the obtained kth output result is stored in a target storage space on the device, wherein N is an integer greater than 1, and k is a positive integer not greater than N; and a reading module configured to read the kth output result from the target storage space to the host.

[0021] At least one embodiment of this disclosure also provides an electronic device. The electronic device includes: one or more processors; a memory including one or more computer program modules; wherein the one or more computer program modules are stored in the memory and configured to be executed by the one or more processors, the one or more computer program modules being used to implement the data processing method provided in any embodiment of this disclosure.

[0022] At least one embodiment of this disclosure also provides a storage medium storing non-transitory computer-readable instructions that, when executed by a computer, implement the data processing method provided in any embodiment of this disclosure.

[0023] The data processing method, processor, electronic device, and storage medium provided in at least one embodiment of this disclosure respond to multiple autoregressive inference requests issued by the host by using a persistent kernel function running on the device side, and storing the input and output data and intermediate calculation results of each inference request by pre-allocating target storage space. The entire process of autoregressive inference calculation of multiple inference requests is migrated to the device side for execution, which effectively solves the performance bottleneck of repeatedly starting kernel functions in large language model inference and significantly improves computational efficiency and memory utilization. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0025] Figure 1 This is a schematic diagram illustrating the execution of computing tasks on both the host and device sides.

[0026] Figure 2 An exemplary flowchart illustrating a data processing method provided in at least one embodiment of this disclosure;

[0027] Figure 3 Another exemplary flowchart of a data processing method provided in at least one embodiment of this disclosure;

[0028] Figure 4 A schematic diagram illustrating an example of a data processing method provided in at least one embodiment of this disclosure;

[0029] Figure 5 A schematic diagram illustrating another example of the data processing method provided in at least one embodiment of this disclosure;

[0030] Figure 6 A schematic diagram illustrating yet another example of a data processing method provided in at least one embodiment of this disclosure;

[0031] Figure 7 A schematic block diagram of a device-side processor provided for at least one embodiment of this disclosure;

[0032] Figure 8 A schematic block diagram of a host-side processor provided for at least one embodiment of this disclosure;

[0033] Figure 9 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure;

[0034] Figure 10 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure;

[0035] Figure 11 This is a schematic diagram of a storage medium provided for at least one embodiment of the present disclosure. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0037] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “including,” “comprising,” or “containing,” and similar terms mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. The terms “connected,” “linked,” or similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “upper,” “lower,” “left,” and “right,” etc., are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0038] The present disclosure will now be described through several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and components may be omitted. When any component of the embodiments of the present disclosure appears in more than one drawing, the component is represented by the same or similar reference numerals in each drawing.

[0039] Autoregressive models are core models in the fields of natural language processing, speech generation, text generation, and dialogue systems within artificial intelligence. Their core characteristic is the generation of tokens (the basic units of model output, such as text or speech fragments) one by one, with each new token generation relying on all historical tokens. This generation method inherently makes model inference sequential. As the model size and the length of the generated sequence increase, higher demands are placed on inference efficiency, throughput, and latency control. In practical applications, autoregressive model inference employs a heterogeneous computing architecture, mainly consisting of two parts: a host program running on the host and a device program running on the device. These two parts collaborate to complete the entire inference task. The host is responsible for task scheduling, memory management, and kernel startup control, while the device is responsible for executing the specific parallel computing operations.

[0040] Figure 1 This is a schematic diagram illustrating the execution of computing tasks on both the host and device sides.

[0041] For example, such as Figure 1 As shown, when various computing tasks are performed on the device, the entire process follows a clear collaborative logic: First, the host computer acts as the control core of the entire computing task. The host computer's processor runs the host computer program, which calls preset kernel functions and provides the input data required for the computation to the device computer. Subsequently, the device computer program starts running, and its internal grid is used to execute the kernel functions. Specifically, the grid contains multiple thread blocks (including but not limited to...) Figure 1 The kernel function (thread blocks 1-1 to 1-4 and thread blocks 2-1 to 2-4) in the kernel execute in parallel to process the input data. During this process, the threads within the thread blocks can cooperate through shared memory and other means to further improve data processing efficiency. Finally, after the data processing is completed on the device side, the host processor controls the device side through the host program to copy the processed results from the device side back to the host side's memory, thus completing the closed loop of the entire computing task. For example, the device-side processor may include AI acceleration chips such as GPUs or GPGPUs, and the host-side processor may include CPUs, etc. The embodiments disclosed herein are not limited in this respect.

[0042] The current autoregressive model inference process has obvious performance bottlenecks. Specifically, since each new word depends on all historical words, after the model completes each round of word generation (i.e., one round of calculation), the host needs to reallocate memory resources to the device and restart a new independent kernel function to send the next round of calculation tasks to the device to execute the new word generation calculation, thus forming a multi-round "start-execute-exit" kernel function running mode.

[0043] Since kernel function startup incurs a fixed performance overhead, this overhead primarily arises from non-computational aspects such as task queuing, resource scheduling, and context switching between the host and device sides. These overheads consume a significant amount of inference time. As the number of lexical generation rounds increases, the cumulative kernel function startup overhead rises sharply, severely limiting the improvement of autoregressive model inference efficiency. It also leads to frequent fluctuations in memory allocation between the host and device sides, increasing memory management complexity, affecting the stability of collaboration between the two ends, and reducing the reliability of model inference.

[0044] At least one embodiment of this disclosure provides a data processing method applied to a device, wherein a persistent kernel function runs on the device, and during the process of starting and exiting the persistent kernel function, the persistent kernel function performs a corresponding autoregressive inference operation on each of N inference requests, wherein N is an integer greater than 1. The data processing method includes: pre-allocating target storage space on the device side; in response to receiving the kth inference request out of N inference requests, performing the first inference on the kth input data of the kth inference request using a persistent kernel function to obtain an initial first cached intermediate result, and storing the initial first cached intermediate result in the target storage space, where k is a positive integer and not greater than N; using the initial first cached intermediate result as the input to the autoregressive inference of the kth inference request, performing the autoregressive inference of the kth inference request using a persistent kernel function to obtain the kth output result and multiple first cached intermediate results, and storing the kth output result and multiple first cached intermediate results in the target storage space, wherein, during the autoregressive inference process of the kth inference request, the initial first cached intermediate result and multiple first cached intermediate results are used to calculate the kth output result.

[0045] At least one embodiment of this disclosure also provides a data processing method applied to a host. The data processing method includes: sending the k-th inference request out of N inference requests to a device, wherein a persistent kernel function runs on the device, the persistent kernel function performs a corresponding autoregressive inference operation on each of the N inference requests, and after the persistent kernel function performs the autoregressive inference operation on the k-th inference request, the resulting k-th output result is stored in a target storage space on the device; and reading the k-th output result from the target storage space to the host.

[0046] At least one embodiment of this disclosure also provides a device-side processor, a host-side processor, an electronic device, and a storage medium.

[0047] The data processing method, processor, electronic device, and storage medium provided in at least one embodiment of this disclosure respond to multiple autoregressive inference requests issued by the host by using a persistent kernel function running on the device side, and storing the input and output data and intermediate calculation results of each inference request by pre-allocating target storage space. The entire process of autoregressive inference calculation of multiple inference requests is migrated to the device side for execution, which effectively solves the performance bottleneck of repeatedly starting kernel functions in large language model inference and significantly improves computational efficiency and memory utilization.

[0048] At least one embodiment of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the same reference numerals will be used to refer to the same elements described in different drawings.

[0049] Figure 2 An exemplary flowchart of a data processing method provided in at least one embodiment of this disclosure.

[0050] For example, such as Figure 2 As shown, at least one embodiment of this disclosure provides a data processing method applied to a device, on which a persistent kernel function runs. For example, during the startup and exit process of the persistent kernel function, the persistent kernel function performs a corresponding autoregressive inference operation on each of N inference requests, where N is an integer greater than 1.

[0051] For example, the N inference requests can be N autoregressive inference operation requests, used to perform corresponding inference operations on the autoregressive model. The embodiments of this disclosure do not limit the specific implementation form of the inference requests. For example, a persistent kernel function is a kernel function that can reside on the device for a long time, be executed repeatedly, and not exit. In the autoregressive inference operations of the N inference requests, the persistent kernel function can remain on the device without being released until all N inference requests have been calculated. The embodiments of this disclosure also do not limit the specific implementation form of the persistent kernel function.

[0052] For example, such as Figure 2 As shown, the data processing method includes the following steps S110~S130.

[0053] Step S110: Pre-allocate target storage space on the device side.

[0054] For example, in step S110, the target storage space can be a sufficiently large memory, enough to store the input and output data and intermediate calculation results of N inference requests. For example, a sufficiently large target storage space can be pre-allocated based on the maximum text length (max sequence length) that the autoregressive model can process in one operation, such as the number of tokens. For example, when the autoregressive model is determined and the maximum text length that the autoregressive model can process in one operation is determined, the storage capacity of the target storage space is also fixed.

[0055] Step S120: In response to receiving the kth inference request out of N inference requests, perform the first inference on the kth input data of the kth inference request using the persistent kernel function to obtain the initial first cached intermediate result, and store the initial first cached intermediate result in the target storage space, where k is a positive integer and not greater than N.

[0056] For example, in step S120, the k-th inference request can be an inference operation request for an autoregressive model sent by the host. The k-th inference request includes the k-th input data of the k-th inference request, various parameter data (also called metadata) related to the request, control instructions, etc. For example, the k-th input data can be a sequence of data units consisting of at least one data unit. When performing the first inference on the k-th input data, the persistent kernel function performs prediction calculations on each data unit in the k-th input data to obtain a cached intermediate result for each data unit. This cached intermediate result can be the key vector and value vector of the data unit. For example, all the key vectors and value vectors of the obtained data unit sequence are the initial first cached intermediate result. This initial first cached intermediate result is used as the key-value cache (KVCache) input in the subsequent autoregressive inference process in step S130 and is stored in the target storage space.

[0057] It should be noted that performing prediction calculations on each data unit in each k input data will also yield the initial prediction result for each data unit, but these initial prediction results will be ignored (i.e., not used in subsequent autoregressive inference).

[0058] For example, the data unit provided in at least one embodiment of this disclosure can be a token, or it can be selected as other types of data as needed. The embodiments of this disclosure do not limit this.

[0059] Step S130: Use the initial first cache intermediate result as the input of the autoregressive inference of the kth inference request, use the persistent kernel function to execute the autoregressive inference of the kth inference request, obtain the kth output result and multiple first cache intermediate results, and store the kth output result and multiple first cache intermediate results in the target storage space.

[0060] For example, in step S130, the initial first cached intermediate result obtained in step S120 is used as input, and the autoregressive inference process (also known as the decoding stage) for the k-th inference request is executed using a persistent kernel function. For example, in the autoregressive inference process of the k-th inference request, the initial first cached intermediate result and the generated multiple first cached intermediate results are used to calculate the k-th output result.

[0061] For example, the k-th output result can be a sequence consisting of at least one predicted data unit; based on the initial first cached intermediate result of the input, the first data unit in the k-th output result can be predicted, and the cached intermediate result of the first data unit can be calculated; based on the initial first cached intermediate result and the cached intermediate result of the first data unit, the second data unit in the k-th output result can be predicted, and the cached intermediate result of the second data unit can be calculated; ...; and so on, until all data units in the k-th output result are predicted. For example, in the above autoregressive inference process, each set of cached intermediate results can be added to the key-value cache to form a new key-value cache, so that the updated key-value cache can be used for the prediction of the next data unit.

[0062] For example, in the above autoregressive inference process, at least one predicted data unit and all cached intermediate results (i.e., multiple first cached intermediate results) obtained during the prediction process are stored in the target storage space. For example, after the k-th inference request is completed, the host will copy the k-th output result from the target storage space back to the host memory.

[0063] For example, taking the k-th input data as a word sequence, in the autoregressive inference stage, the predicted words can be used to predict the next word until the k-th inference request is processed. The resulting k-th output can be the output sequence composed of all predicted words. Each predicted word is stored in the target storage space. During the prediction of each word, the corresponding first cache intermediate result can be calculated. All historical first cache intermediate results (including the initial first cache intermediate result and the cache intermediate results of all words predicted before this word) can be used to predict the next new word. Each calculated first cache intermediate result is stored in the target storage space.

[0064] It should be noted that, in at least one embodiment of this disclosure, the k-th inference request can be any one of the N inference requests. In other words, any one of the inference requests in at least one embodiment of this disclosure can be used for autoregressive inference operations in accordance with the processing method for the k-th inference request. The specific method can be selected according to actual needs, and the embodiments of this disclosure do not limit this.

[0065] The data processing method provided in at least one embodiment of this disclosure responds to multiple autoregressive inference requests issued by the host by using a persistent kernel function running on the device side, and stores the input and output data and intermediate calculation results of each inference request by pre-allocating target storage space. The entire process of autoregressive inference calculation of multiple inference requests is migrated to the device side for execution, which effectively solves the performance bottleneck of repeatedly starting kernel functions in large language model inference and significantly improves computational efficiency and memory utilization.

[0066] For example, in some examples, the target storage space includes a first buffer and an intermediate cache. For instance, the first buffer can be reused to store the input and output results of each inference request, as well as intermediate prediction results obtained during the autoregressive inference process. The intermediate cache can be used to store cached intermediate results obtained during the autoregressive inference process (e.g., key-value cache (KVCache) used in autoregressive model inference). For example, the k-th input data is written to the first buffer. For instance, the specific implementation of the first buffer and the intermediate cache can be selected according to actual needs, and the embodiments of this disclosure do not limit this.

[0067] For example, Figure 2 The step S120, "Storing the initial first cache intermediate result into the target storage space", includes: storing the initial first cache intermediate result into the intermediate cache.

[0068] For example, Figure 2 The step S130 of "storing the k-th output result and multiple first-cache intermediate results into the target storage space" includes: storing the k-th output result into the first buffer, and storing multiple first-cache intermediate results into the intermediate cache.

[0069] For example, in some examples, in Figure 2 Before step S120, the data processing method provided in at least one embodiment of this disclosure may further include the following step S140.

[0070] Step S140: Set the first flag bit for the first buffer.

[0071] For example, in step S140, the first flag bit is a marker for communication between the host and the device. For example, the first flag bit includes a host-ready state and a device-ready state; when the first flag bit is host-ready, it indicates that the host has completed the actions related to the first buffer (e.g., completing the writing of input requests, copying output results, etc.), and the device can use the first buffer; when the first flag bit is device-ready, it indicates that the device has completed the actions related to the first buffer (e.g., completing the autoregressive inference operation for the corresponding request, etc.), and the host can use the first buffer.

[0072] For example, Figure 2 Step S120 may further include: in response to the k-th input data being written to the first buffer and the first flag being in a host-side ready state, performing the first inference on the k-th input data using a persistent kernel function.

[0073] For example, in response to Figure 2 After step S130 is completed, the data processing method provided in at least one embodiment of this disclosure may further include: updating the first flag bit to the device ready state to indicate that the host can read the k-th output result.

[0074] For example, after the k-th input data is written to the first buffer, by checking that the first flag is in a host-ready state, the corresponding autoregressive inference operation can be performed on the k-th input data using a persistent kernel function on the device side. For example, after the autoregressive inference operation of the k-th inference request is completed, the first flag is updated to a device-ready state; the host can then read the k-th output result from the first buffer based on the device-ready state of the first flag.

[0075] The data processing method provided in at least one embodiment of this disclosure achieves state interaction and synchronization control between the host and device by using flag bits. Compared with traditional synchronization methods based on large data transmissions or shared memory blocks, this significantly saves memory space between the host and device. During synchronization, only a small number of flag bits are needed to transmit status information such as task completion, data validity, and buffer reusability. This eliminates the need for large amounts of additional memory for storing synchronization information or providing status feedback, effectively reducing storage resource overhead between the host and device, simplifying synchronization logic, and enhancing the lightweight nature of data interaction.

[0076] For example, in some examples, the k-th input data includes at least one data unit. Specifically, for the initial inference process of the k-th input data, Figure 2Step S120 may further include the following step S121.

[0077] Step S121: In response to receiving the k-th inference request, perform a first computation on each data unit in at least one data unit using a persistent kernel function to obtain an initial first cached intermediate result.

[0078] For example, in step S121, the k-th input data can be a sequence of data units consisting of at least one data unit, and the first calculation can be an inference calculation for each data unit in the data unit sequence. For example, taking the k-th input data as a word sequence, the persistent kernel function is used to perform projection calculation on each word in the word sequence, which can obtain the corresponding query vector, key vector, and value vector for the word. Then, attention calculation is performed using the query vector, key vector, and value vector to obtain the first prediction result for the word (as described above, this first prediction result will be ignored). In this example, the key vectors and value vectors of the multiple words obtained are the initial first cached intermediate results.

[0079] It should be noted that the key vector and value vector are stored in the intermediate cache, while the query vector does not need to be stored in the intermediate cache. This is because the initial first cache intermediate result is used in each prediction step of the subsequent autoregressive inference, while the query vector is only used once in the current operation.

[0080] For example, in some examples, a first offset pointer is set in the first buffer, and a second offset pointer is set in the intermediate buffer. For example, the first and second offset pointers are used to record which data in the first buffer and intermediate buffer is valid data, respectively. For example, data from the initial position in the first buffer (i.e., the position pointed to when the first offset pointer is 0) to the position pointed to by the first offset pointer can be set as valid data, and data from the initial position in the intermediate buffer (i.e., the position pointed to when the second offset pointer is 0) to the position pointed to by the second offset pointer can be set as valid data; other setting methods can also be selected according to actual needs, and the embodiments of this disclosure do not limit this.

[0081] For example, in Figure 2 Before step S120, the data processing method provided in at least one embodiment of this disclosure may further include the following step S150.

[0082] Step S150: Based on the size of the k-th input data, set a first offset pointer in the first buffer and point the first offset pointer to the end position of the k-th input data.

[0083] For example, in step S150, the actual size corresponding to the first offset pointer can be determined by the size of the k-th input data, and the first offset pointer can be set in the first buffer. For example, after the k-th input data is input into the first buffer, the first offset pointer is set to point to the end position of the k-th input data; thereafter, each time the next data is predicted, the value of the first offset pointer is incremented by 1 so that the first offset pointer points to the end of the new predicted data.

[0084] For example, Figure 2 The step S120, "Storing the initial first cached intermediate result into the target storage space", can further include the following step S122.

[0085] Step S122: Store the initial first cache type intermediate result in the intermediate cache, set the second offset pointer in the intermediate cache, and point the second offset pointer to the end of the initial first cache type intermediate result.

[0086] For example, in step S122, after the initial first cache intermediate result is stored in the intermediate cache, the second offset pointer is set to point to the end of the initial first cache intermediate result; thereafter, each time the next first cache intermediate result is calculated, the value of the second offset pointer is incremented by 1 so that the second offset pointer points to the end of the newly stored first cache intermediate result.

[0087] For example, in some examples, in Figure 2 After step S130, the data processing method provided in at least one embodiment of this disclosure may further include the following step S160.

[0088] Step S160: Set the values ​​of the first offset pointer and the second offset pointer to 0, so that the first offset pointer points to the starting position of the first buffer and the second offset pointer points to the starting position of the intermediate buffer.

[0089] For example, in step S160, by setting both the values ​​of the first offset pointer and the second offset pointer to 0, the data after the first offset pointer position in the first buffer can be set as invalid data, and the data after the second offset pointer position in the intermediate buffer can be set as invalid data. This prepares the first buffer and the intermediate buffer for the computation of the next inference request. For example, the relevant data of the next request can directly overwrite the historical data in the first buffer and the intermediate buffer, thereby realizing the reuse of the storage space of the first buffer and the intermediate buffer, and omitting the step of clearing the historical data (e.g., rewriting the position of the historical data to zero), thus improving data processing efficiency.

[0090] The data processing method provided in at least one embodiment of this disclosure uses an offset pointer to identify the valid data region and directly writes the feature data of the current inference output into the storage space corresponding to the original input by using an auto-incrementing offset, thereby achieving the reuse of cache space. Compared with the traditional approach that requires regenerating a new key-value cache and allocating an additional independent memory region for each inference, the data processing method provided in at least one embodiment of this disclosure does not require frequently opening new storage units, avoiding redundant memory allocation and occupation, effectively reducing device-side memory overhead, and improving storage resource utilization.

[0091] For example, in some examples, the k-th output includes M data units, where M is an integer greater than 1. For example, the data units can be tokens, or other forms can be selected as needed; the embodiments of this disclosure do not impose such limitations.

[0092] For example, multiple first-cached intermediate results include M first-cached intermediate results, and the first first-cached intermediate result among the multiple first-cached intermediate results includes the initial first-cached intermediate result.

[0093] For example, for the subsequent autoregressive inference process of the k-th input data, specifically, Figure 2 Step S130 may further include the following steps S131 to S133.

[0094] Step S131: Using a persistent kernel function, predict the first data unit among M data units based on the initial first cache intermediate result, calculate the first first cache intermediate result of the first data unit, and store the first data unit and the first cache intermediate result of the first data unit into the target storage space;

[0095] Step S132: Use the persistent kernel function to perform a second calculation on the j-th data unit and the first to the j-th first cache intermediate results in the M data units to obtain the (j+1)-th data unit, and store the (j+1)-th data unit in the target storage space, where j is an integer not greater than M-1 and greater than 1;

[0096] Step S133: In response to j+1 being less than M, the third calculation is performed on the (j+1)th data unit using the persistent kernel function to obtain the (j+1)th first cache intermediate result among the M first cache intermediate results, and the (j+1)th first cache intermediate result is stored in the target storage space.

[0097] For example, in step S131, after the initial inference is completed, the first data unit can be predicted based on the obtained initial first cache intermediate result. For example, the k-th input data and the k-th output result are both data unit sequences, and the initial first cache intermediate result is the key-value cache of the data units in the k-th input data obtained in the initial inference (i.e., the initial first cache intermediate result); based on the data unit at the last position in the k-th input data, the corresponding query vector can be obtained. By performing attention calculation on the query vector and the initial first cache intermediate result, the first data unit in the k-th output result can be predicted, and the first data unit is stored in the target storage space; by performing projection calculation on the first data unit, the key vector and value vector corresponding to the first data unit can be obtained (i.e., the first cache intermediate result of the first data unit), and the first cache intermediate result is stored in the target storage space.

[0098] For example, in step S132, the second calculation can be a prediction calculation. Specifically, based on the j-th data unit, the corresponding query vector can be obtained. Attention calculation is performed on the query vector, the first first cache intermediate result to the j-th first cache intermediate result, and the (j+1)-th data unit can be predicted and stored in the target storage space.

[0099] For example, in step S133, the second calculation can be a projection calculation. Specifically, if j+1 is less than M (i.e., the (j+1)th data unit is not the last data unit of the k-th output result), a projection calculation is performed on the (j+1)th data unit using a persistent kernel function to obtain the (j+1)th first cached intermediate result; the (j+1)th first cached intermediate result is stored in the target storage space for prediction of the next data unit. For example, if j+1 equals M (i.e., the (j+1)th data unit is the last data unit of the k-th output result), then there is no need to perform a projection calculation on the (j+1)th data unit, because the (j+1)th data unit is already the last data unit of the k-th output result, and therefore there is no need to use the (j+1)th first cached intermediate result to predict the next data unit.

[0100] For example, the step S133 of "storing the (j+1)th first cache intermediate result into the target storage space" can further include: in the intermediate cache, placing the (j+1)th first cache intermediate result at the end of the jth first cache intermediate result, and incrementing the value of the second offset pointer by one so that the second offset pointer points to the end of the (j+1)th first cache intermediate result.

[0101] For example, the step S132 of "storing the (j+1)th data unit into the target storage space" can further include: in the first buffer, placing the (j+1)th data unit at the end of the jth data unit, and incrementing the value of the first offset pointer by one so that the first offset pointer points to the end of the (j+1)th data unit.

[0102] For example, in some examples, the target storage space further includes a second buffer, which is different from the first buffer. For example, the data processing method provided in at least one embodiment of this disclosure may further include the following steps S170-S180.

[0103] Step S170: In response to receiving the (k+1)th inference request out of N inference requests, perform the first inference on the (k+1)th input data of the (k+1)th inference request using the persistent kernel function to obtain the initial second cache intermediate result, and store the initial second cache intermediate result in the intermediate cache;

[0104] Step S180: Use the initial second-cached intermediate result as input to the autoregressive inference of the (k+1)th inference request, execute the autoregressive inference of the (k+1)th inference request using the persistent kernel function, obtain the (k+1)th output result and multiple second-cached intermediate results, store the (k+1)th output result in the second buffer, and store the multiple second-cached intermediate results in the intermediate cache. For example, during the autoregressive inference process of the (k+1)th inference request, the initial second-cached intermediate result and multiple second-cached intermediate results are used to calculate the (k+1)th output result.

[0105] For example, in at least one embodiment of this disclosure, the process of performing initial inference and autoregressive inference on the (k+1)th inference request using a persistent kernel function is essentially the same as the process of performing initial inference and autoregressive inference on the kth inference request using a persistent kernel function. The only difference is that the (k+1)th input data in step S170 is written to a second buffer different from the first buffer, and the (k+1)th output result obtained in step S180 and the prediction result of the intermediate autoregressive inference process are stored in a second buffer different from the first buffer. Other similarities can be found in the description of the kth inference request above, and will not be repeated here.

[0106] For example, the data processing method provided in at least one embodiment of this disclosure may further include the following step S190.

[0107] Step S190: Set the second flag bit for the second buffer.

[0108] For example, in step S190, the second flag is a marker for communication between the host and the device. For instance, the second flag includes a host-ready state and a device-ready state; when the second flag is host-ready, it indicates that the host has completed the actions related to the second buffer (e.g., writing an input request, copying an output result, etc.), and the device can use the second buffer; when the second flag is device-ready, it indicates that the device has completed the actions related to the second buffer (e.g., performing an autoregressive inference operation for the corresponding request, etc.), and the host can use the second buffer.

[0109] For example, step S170 may further include: in response to the (k+1)th input data being written to the second buffer and the second flag being in a host-side ready state, performing the first inference on the (k+1)th input data using a persistent kernel function.

[0110] For example, after step S180, the data processing method provided in at least one embodiment of this disclosure may further include: updating the second flag bit to a device-side ready state to indicate that the host can read the (k+1)th output result.

[0111] For example, after the (k+1)th input data is written to the second buffer, by checking that the second flag is in a host-ready state, the corresponding autoregressive inference operation can be performed on the (k+1)th input data using a persistent kernel function on the device side. For example, after the autoregressive inference operation of the (k+1)th inference request is completed, the second flag is updated to a device-ready state, and the host can then read the (k+1)th output result from the second buffer based on the device-ready state of the second flag.

[0112] For example, without two buffers (e.g., only the first buffer stores the input and output data for each inference request), the computation on the device side and the reading and copying (or data migration) of the computation results on the host side can only be performed serially: after the device side completes the computation of the current request, it must wait for the host side to complete copying the result data in the first buffer before the host side sends the next inference request to the device side to execute the corresponding computation; that is, the device side is idle during the data migration of all computation results, and the overall process time is a linear sum of the device side computation time and the data migration time, and the data migration overhead is directly reflected in the overall latency.

[0113] For example, at least one embodiment of this disclosure provides a data processing method that implements a double-buffered alternating working mechanism by setting a second buffer on the device side that is different from the first buffer. This effectively hides the memory copy overhead between the host and the device. For instance, after the device completes the calculation of the k-th inference request in the first buffer, the host can move the calculation result from the first buffer. At the same time, the device can use the second buffer to execute the calculation of the (k+1)-th inference request in parallel, so that the data moving time overlaps with the calculation time of the device. By parallelizing the data moving time with the calculation time of subsequent requests, the device can avoid being idle while waiting for data moving, significantly improving the overall execution parallelism, reducing the overall waiting latency, and improving the execution efficiency of autoregressive model inference.

[0114] It should be noted that, in addition to setting two buffers on the device side as described above, more buffers can also be set to alternately process different inference requests. The specific settings can be made according to actual needs, and the embodiments disclosed herein do not impose any restrictions on this.

[0115] Figure 3 This is an exemplary flowchart illustrating a data processing method provided in at least one embodiment of this disclosure. For example, Figure 3 The data processing method is with Figure 2 The data processing method and the corresponding host-side operation flow.

[0116] For example, such as Figure 3 The data processing method includes the following steps S210~S220.

[0117] Step S210: Send the kth inference request out of the N inference requests to the device, where N is an integer greater than 1 and k is a positive integer not greater than N.

[0118] For example, in step S210, a persistent kernel function runs on the device. This persistent kernel function performs the corresponding autoregressive inference operation on each of the N inference requests. After the persistent kernel function performs the autoregressive inference operation on the k-th inference request, the resulting k-th output is stored in the target storage space on the device. For example, the process of performing the autoregressive inference operation on the k-th inference request and the storage process of the k-th output can be referred to the above description. Figure 2 The descriptions of steps S110 to S130 will not be repeated here.

[0119] Step S220: Read the k-th output result from the target storage space to the host.

[0120] For example, in step S220, the host reads the k-th output result from the target storage space to complete the data transfer of the autoregressive inference operation result for the k-th inference request.

[0121] For example, in some examples, Figure 3 Step S210 may further include the following steps S211~S212.

[0122] Step S211: Write the k-th input data of the k-th inference request into the first buffer;

[0123] Step S212: Update the first flag to the host-side ready state to indicate that the persistent kernel function can perform the autoregressive inference operation for the k-th inference request.

[0124] For example, after the host writes the k-th input data into the first buffer in step S211, the host updates the first flag to the host ready state in step S212; the device can use the persistent kernel function to perform the corresponding autoregressive inference operation on the k-th input data by checking that the first flag is in the host ready state.

[0125] For example, the k-th output result obtained from the autoregressive inference operation of the k-th inference request is stored in the first buffer. Figure 3 Step S220 may further include the following step S221.

[0126] Step S221: In response to the first flag indicating that the device is ready, read the k-th output result from the first buffer to the host.

[0127] For example, in step S221, the host can read and copy the k-th output result from the first buffer by checking that the first flag bit is in the ready state of the device, so as to complete the data transfer from the device to the host.

[0128] For example, in Figure 3 After step S220, the data processing method provided in at least one embodiment of this disclosure may further include the following steps S230 to S240.

[0129] Step S230: In response to the fact that the N inference requests also include the (k+2)th inference request to be processed, send the (k+2)th inference request to the device.

[0130] Step S240: Update the first flag to the host-side ready state to indicate that the device-side can execute the autoregressive inference operation for the (k+2)th inference request.

[0131] For example, in step S230, after the k-th output result is read to the host, if the N inference requests also include the (k+2)-th inference request to be processed, the host continues to send the (k+2)-th inference request to the device. For example, in step S240, after the host writes the (k+2)-th input data of the (k+2)-th inference request into the first buffer, it updates the first flag to the host-ready state; after the device checks that the first flag is in the host-ready state, it can execute the autoregressive inference operation of the (k+2)-th inference request.

[0132] For example, based on steps S221 and S230~S240, for the first flag bit of the first buffer, when the first flag bit is in the device ready state, the host first completes the data transfer of the kth output result of the kth inference request from the first buffer, and then sends the (k+2)th inference request to write the corresponding (k+2)th input data into the first buffer; after completing the above two steps, the host will update the first flag bit to the host ready state.

[0133] For example, in some examples, the data processing method provided in at least one embodiment of this disclosure may further include the following steps S250 to S270.

[0134] Step S250: In response to the fact that the N inference requests also include the (k+1)th inference request to be processed, the (k+1)th inference request is sent to the device, wherein the (k+1)th input data of the (k+1)th inference request is written to the second buffer;

[0135] Step S260: Update the second flag to the host-side ready state to indicate that the persistent kernel function can execute the autoregressive inference operation of the k+1th inference request, wherein the k+1th output result obtained after the persistent kernel function executes the autoregressive inference operation of the k+1th inference request is stored in the second buffer.

[0136] Step S270: In response to the second flag indicating that the device is ready, read the (k+1)th output result from the second buffer to the host.

[0137] For example, in at least one embodiment of this disclosure, the processing of the (k+1)th inference request by the host is basically the same as the processing of the kth inference request, except that the processing of the (k+1)th inference request is bound to a second buffer and its second flag bit, which is different from the first buffer; other similarities can be referred to the description of the kth inference request above, and will not be repeated here.

[0138] For example, after step S270, the data processing method provided in at least one embodiment of this disclosure may further include the following steps S280 to S290.

[0139] Step S280: In response to the fact that the N inference requests also include the (k+3)th inference request to be processed, send the (k+3)th inference request to the device.

[0140] Step S290: Update the second flag to the host-side ready state to indicate that the device-side can execute the autoregressive inference operation for the (k+3)th inference request.

[0141] For example, in step S230, after the (k+1)th output result is read to the host, if the N inference requests also include the (k+3)th inference request to be processed, the host continues to send the (k+3)th inference request to the device. For example, in step S240, after the host writes the (k+3)th input data of the (k+3)th inference request into the second buffer, it updates the second flag to the host ready state; after the device checks that the second flag is in the host ready state, it can execute the autoregressive inference operation of the (k+3)th inference request.

[0142] For example, based on steps S270~S290, for the second flag bit of the second buffer, when the second flag bit is in the device ready state, the host first completes the data transfer of the k+1 output result of the k+1 inference request from the second buffer, and then sends the k+3 inference request to write the corresponding k+3 input data into the second buffer; after completing the above two steps, the host will update the second flag bit to the host ready state.

[0143] the following Figures 4-6 A data processing method (including) provided by at least one embodiment of the present disclosure is described. Figure 2 and Figure 3 Here are a few specific examples of the various steps in the process.

[0144] Figure 4 This is a schematic diagram illustrating an example of a data processing method provided in at least one embodiment of the present disclosure.

[0145] For example, such as Figure 4 As shown, a persistent kernel function runs on the device side, and a target storage space is pre-allocated on the device side, which includes a first buffer, a second buffer, and an intermediate cache.

[0146] For example, such as Figure 4 As shown, after the device receives the kth inference request sent by the host, it uses the persistent kernel function to perform the first inference on the kth input data of the kth inference request, obtains the initial first cached intermediate result, and stores the initial first cached intermediate result in the intermediate cache.

[0147] For example, further, the device uses the initial first-cache intermediate result as input to the autoregressive inference of the k-th inference request, executes the autoregressive inference of the k-th inference request using a persistent kernel function, obtains the k-th output result and multiple first-cache intermediate results, stores the k-th output result in the first buffer, and stores the multiple first-cache intermediate results in the intermediate buffer. For example, during the autoregressive inference of the k-th inference request, the initial first-cache intermediate result and multiple first-cache intermediate results cached in the intermediate buffer are used to calculate the k-th output result. For example, after the autoregressive inference of the k-th inference request is completed, the first flag bit of the first buffer is updated to the device-ready state, at which point the host can read the k-th output result in the first buffer.

[0148] For example, such as Figure 4 As shown, during the process of reading the k-th output result in the first buffer on the host side, if the host side has already sent the (k+1)-th inference request to the device side, and the second flag bit in the second buffer is in the host side ready state, then the device side uses the persistent kernel function to perform the first inference on the (k+1)-th input data of the (k+1)-th inference request, obtains the initial second cache intermediate result, and stores the initial second cache intermediate result in the intermediate cache.

[0149] For example, further, the device uses the initial second-cache intermediate result as input to the autoregressive inference of the (k+1)th inference request, executes the autoregressive inference of the (k+1)th inference request using a persistent kernel function, obtains the (k+1)th output result and multiple second-cache intermediate results, stores the (k+1)th output result in the second buffer, and stores the multiple second-cache intermediate results in the intermediate cache. For example, during the autoregressive inference of the (k+1)th inference request, the initial second-cache intermediate result and multiple second-cache intermediate results cached in the intermediate cache are used to calculate the (k+1)th output result. For example, after the autoregressive inference of the (k+1)th inference request is completed, the second flag bit of the second buffer is updated to the device-ready state, at which point the host can read the (k+1)th output result in the second buffer.

[0150] Figure 5 This is a schematic diagram of another example of a data processing method provided in at least one embodiment of the present disclosure.

[0151] For example, Figure 5 As an example of the execution flow on the device side, firstly, based on the maximum text length (such as the number of tokens) that the autoregressive model can process at one time, a sufficiently large target storage space is pre-allocated on the device side; then, a persistent kernel function is started on the device side to perform autoregressive inference operations on N inference requests sent from the host side, where N is an integer greater than 1.

[0152] For example, such as Figure 5 As shown, during the execution of the persistent kernel function, a first buffer or a second buffer is selected for the current inference request (e.g., the k-th inference request). If one of the buffers is selected for the first inference request, then the buffer selected for each subsequent inference request is different from the previous one.

[0153] For example, taking the input data of the current inference request as a sequence of data units (e.g., a word sequence), this input data unit sequence includes at least one data unit, and the input data is written into the current buffer (a first buffer or a second buffer). For example, in the pre-filling stage, a first calculation is performed on each data unit sequence in the at least one data unit using a persistent kernel function to obtain an initial cached intermediate result, and this initial cached intermediate result is stored in an intermediate buffer. For example, an offset pointer (offset_input) is set for the current buffer according to the size of the input data, and this offset pointer is set to point to the end of the input data in the current buffer; a second offset pointer (offset_kv) is set for the intermediate buffer according to the size of the first cached intermediate result, and this second offset pointer is set to point to the end of the initial cached intermediate result in the intermediate buffer.

[0154] For example, such as Figure 5 As shown, further, the first data unit in the output result is predicted using a persistent kernel function based on the initial cached intermediate result, and the obtained first data unit is stored in the current buffer. For example, in the current buffer, the first data unit is stored at the end of the input data, and the value of the corresponding offset pointer is incremented by one (i.e., offset_input++) so that the corresponding offset pointer points to the end of the first data unit.

[0155] For example, such as Figure 5 As shown, further, it is determined whether the current inference request has been completed. If the current inference request has not been completed, the third calculation is performed on the first data unit using the persistent kernel function to obtain the first cached intermediate result, and the first cached intermediate result is stored in the intermediate cache. In the intermediate cache, the first cached intermediate result is stored at the end of the initial cached intermediate result, and the value of the second offset pointer is incremented by one (i.e., offset_kv++) so that the second offset pointer points to the end of the first cached intermediate result.

[0156] For example, such as Figure 5As shown, further, the persistent kernel function is used to perform a second calculation on the first data unit, the initial cached intermediate result, and the first cached intermediate result to obtain the second data unit; in the current buffer, the second data unit is stored at the end of the first data unit, and the value of the corresponding offset pointer is incremented by one (i.e., offset_input++) so that the corresponding offset pointer points to the end of the second data unit.

[0157] For example, such as Figure 5 As shown, further, it is determined whether the current inference request has been completed. If the current inference request has not been completed, the same logic is followed to perform further calculations according to the above process. Until the current inference request is completed, the flag of the current buffer is set to device_ready, and the value of the offset pointer corresponding to the current buffer and the value of the second offset pointer of the intermediate cache are both set to 0.

[0158] For example, such as Figure 5 As shown, further, the value of the completion flag (endflag) is checked to see if it is 1. If endflag is 1, it means that all N inference requests sent by the host have been processed, and the device program can be terminated. For example, if endflag is not 1, it means that the N inference requests sent by the host have not been processed, and the flag of another buffer is checked to see if it is host-ready. If the flag is not host-ready, it means that the host has not yet completed processing of the buffer, and it is necessary to continue waiting for the host to complete processing. If the flag is host-ready, it means that the host has completed processing of the buffer, and the device can use the buffer. The buffer can then be selected for the next pending inference request (e.g., the (k+1)th inference request), and so on, performing the corresponding autoregressive inference operation on the inference request according to the above process.

[0159] Figure 6 This is a schematic diagram of yet another example of a data processing method provided in at least one embodiment of the present disclosure.

[0160] For example, Figure 6As an example of the execution flow on the host side, firstly, the host side sends a relevant request to the device side, instructing the device side to pre-allocate a sufficiently large target storage space based on the maximum text length that the autoregressive model can process in one go. Then, the host side sends the inference request to be processed, writes the input data of the inference request into the corresponding buffer, and sets the corresponding flag bit of the buffer to host_ready after writing. For example, when the host side sends the first inference request for the first time, it can send the first inference request and the second inference request simultaneously, write the first input data of the first inference request into the first buffer, and write the second input data of the second inference request into the second buffer; or it can send only one inference request at the first time, etc. The specific settings can be configured according to actual needs, and the embodiments of this disclosure do not limit this.

[0161] For example, such as Figure 6 As shown, further, the host sends a relevant request to the device to instruct the device to start the persistent kernel function, and the device can then proceed accordingly. Figure 5 The execution flow of the persistent kernel function handles inference requests sent from the host.

[0162] For example, such as Figure 6 As shown, taking the writing of the first input data into the first buffer as an example, the host checks whether the first flag bit of the first buffer is device_ready. If the first flag bit is not device_ready, it means that the device has not yet completed processing of the first buffer, and it needs to continue waiting for the device to complete processing. If the first flag bit is device_ready, it means that the device has completed processing of the first buffer, and the host can use the first buffer. For example, further, when the first flag bit is detected as device_ready, the host reads the first output result in the first buffer and writes the third input data of the new inference request (i.e., the third inference request) into the first buffer. After writing, the first flag bit is set to host_ready to indicate that the device can perform further processing on the first buffer. At the same time, the length of the third input data is recorded for use in setting the relevant offset pointer in the device.

[0163] For example, such as Figure 6As shown, further, the host checks whether the second flag of the second buffer is device_ready. If the second flag is not device_ready, it means that the device has not yet completed processing of the second buffer, and it needs to continue waiting for the device to complete processing. If the second flag is device_ready, it means that the device has completed processing of the second buffer, and the host can use the second buffer. For example, further, when the second flag is detected to be device_ready, the host reads the second output result in the second buffer and writes the fourth input data of the new inference request (i.e., the fourth inference request) into the second buffer. After writing, the second flag is set to host_ready to indicate that the device can perform further processing on the second buffer. At the same time, the length of the fourth input data is recorded for use in setting the relevant offset pointer in the device.

[0164] For example, such as Figure 6 As shown, the host checks whether it needs to exit. If it finds that it needs to exit, it means that all inference requests have been completed. In this case, it sets endflag to 1 and instructs the device to exit the operation. If it finds that it does not need to exit, the host continues to check whether the first flag is device_ready. Then, it follows the same process to process subsequent inference requests accordingly.

[0165] It should be noted that, Figures 4-6 The hardware structures and processing flows shown are merely illustrative, and other hardware structures and processing flows may be selected according to actual needs. The embodiments disclosed herein do not impose any limitations on this.

[0166] It should be noted that, apart from the specific execution process of the autoregressive inference operation described above, other execution details of the autoregressive inference operation provided in at least one embodiment of this disclosure can be found in descriptions in related fields, and will not be repeated here.

[0167] The data processing method provided in at least one embodiment of this disclosure runs a persistent kernel function on the device side to respond to multiple autoregressive inference requests from the host side, and pre-allocates buffers and intermediate caches on the device side to store input, output, and intermediate calculation results. This places the entire multi-regressive inference process on the device side, solving the performance bottleneck of frequently launching kernel functions in large language model inference and improving computational efficiency and memory utilization. Secondly, state synchronization between the host and device sides is achieved through flag bits, using only a few bits to complete information exchange such as task status, data validity, and buffer reuse, without needing to allocate a large-capacity synchronization storage area, saving memory overhead and simplifying synchronization logic. Thirdly, valid data is identified using offset pointers, and input / output data reuse is achieved through offset increment, further saving device-side storage space and improving storage utilization. Furthermore, through a dual-buffer alternation mechanism, while the device side performs inference operations in one buffer, the host side can move data from the operation structure of the other buffer, hiding data movement latency and effectively improving the overall throughput and execution efficiency of autoregressive inference operations.

[0168] Figure 7 This is a schematic block diagram of a device-side processor provided for at least one embodiment of the present disclosure. For example, the device-side processor runs a persistent kernel function. During the startup and exit process of the persistent kernel function, the persistent kernel function performs a corresponding autoregressive inference operation on each of N inference requests, where N is an integer greater than 1.

[0169] For example, such as Figure 7 As shown, the device-side processor 200 includes a pre-allocation module 210 and an inference module 220.

[0170] For example, the pre-allocation module 210 is configured to pre-allocate target storage space on the device side. That is, the pre-allocation module 210 can be configured to perform, for example... Figure 2 The step S110 is shown.

[0171] For example, inference module 220 is configured to: in response to receiving the k-th inference request out of N inference requests, perform initial inference on the k-th input data of the k-th inference request using a persistent kernel function, obtain an initial first cached intermediate result, and store the initial first cached intermediate result in the target storage space, where k is a positive integer and not greater than N. That is, inference module 220 can be configured to perform, for example... Figure 2 The step S120 shown.

[0172] For example, inference module 220 is further configured to: use the initial first cached intermediate result as input to the autoregressive inference of the k-th inference request, execute the autoregressive inference of the k-th inference request using a persistent kernel function, obtain the k-th output result and multiple first cached intermediate results, and store the k-th output result and multiple first cached intermediate results in the target storage space. For example, during the autoregressive inference process of the k-th inference request, the initial first cached intermediate result and multiple first cached intermediate results are used to calculate the k-th output result. That is, the inference module 220 can also be configured to perform, for example... Figure 2 The step S130 shown.

[0173] For example, in some examples, the target storage space includes a first buffer and an intermediate buffer, with the k-th input data written to the first buffer. For example, the inference module 220 is also configured to: store the initial first-buffered intermediate result in the intermediate buffer; store the k-th output result in the first buffer; and store multiple first-buffered intermediate results in the intermediate buffer.

[0174] For example, in some examples, the device-side processor 200 also includes a setup module 230 (not shown) configured to set a first flag for the first buffer. For example, the inference module 220 is also configured to perform the first inference on the k-th input data using a persistent kernel function in response to the k-th input data being written to the first buffer and the first flag being in a host-ready state. For example, in response to the completion of autoregressive inference for the k-th inference request, the setup module 230 is also configured to update the first flag to a device-ready state to indicate that the host can read the k-th output result.

[0175] For example, in some examples, the k-th input data includes at least one data unit, and the inference module 220 is further configured to: in response to receiving the k-th inference request, perform a first computation on each data unit in the at least one data unit using a persistent kernel function to obtain an initial first cached intermediate result, wherein the initial first cached intermediate result includes a cached intermediate result for each data unit.

[0176] For example, in some examples, before performing the first inference on the k-th input data of the k-th inference request using the persistent kernel function, the inference module 220 is also configured to: set a first offset pointer in a first buffer according to the size of the k-th input data, and point the first offset pointer to the end of the k-th input data. For example, the inference module 220 is also configured to: store the initial first cached intermediate result in an intermediate cache, set a second offset pointer in the intermediate cache, and point the second offset pointer to the end of the initial first cached intermediate result.

[0177] For example, in some examples, in response to the completion of autoregressive inference execution for the k-th inference request, the inference module 220 is also configured to set the values ​​of both the first offset pointer and the second offset pointer to 0, so that the first offset pointer points to the beginning of the first buffer and the second offset pointer points to the beginning of the intermediate buffer.

[0178] For example, in some examples, the k-th output includes M data units; multiple first-cache intermediate results include M first-cache intermediate results, and the first first-cache intermediate result among the multiple first-cache intermediate results includes the initial first-cache intermediate result. For example, the inference module 220 is further configured to: use a persistent kernel function to predict the first data unit among M data units based on the initial first cache intermediate result, calculate the first first cache intermediate result of the first data unit, and store the first data unit and the first cache intermediate result of the first data unit in the target storage space; use a persistent kernel function to perform a second calculation on the j-th data unit and the first to j-th first cache intermediate results among M data units to obtain the (j+1)-th data unit, and store the (j+1)-th data unit in the target storage space; in response to j+1 being less than M, use a persistent kernel function to perform a third calculation on the (j+1)-th data unit to obtain the (j+1)-th first cache intermediate result among M first cache intermediate results, and store the (j+1)-th first cache intermediate result in the target storage space, where M is an integer greater than 1, and j is an integer not greater than M-1 and greater than 1.

[0179] For example, the inference module 220 is also configured to: place the (j+1)th data unit at the end of the jth data unit in the first buffer, and increment the value of the first offset pointer by one so that the first offset pointer points to the end of the (j+1)th data unit.

[0180] For example, the inference module 220 is also configured to: place the (j+1)th first cache intermediate result at the end of the jth first cache intermediate result in the intermediate cache, and increment the value of the second offset pointer by one so that the second offset pointer points to the end of the (j+1)th first cache intermediate result.

[0181] For example, in some examples, the target storage space also includes a second buffer, which is different from the first buffer. For example, inference module 220 is further configured to: in response to receiving the (k+1)th inference request out of N inference requests, perform initial inference on the (k+1)th input data of the (k+1)th inference request using a persistent kernel function to obtain an initial second-cached intermediate result, and store the initial second-cached intermediate result in an intermediate cache, wherein the (k+1)th input data is written to the second buffer; use the initial second-cached intermediate result as input to the autoregressive inference of the (k+1)th inference request, perform the autoregressive inference of the (k+1)th inference request using a persistent kernel function to obtain the (k+1)th output result and multiple second-cached intermediate results, store the (k+1)th output result in the second buffer, and store the multiple second-cached intermediate results in the intermediate cache. For example, during the autoregressive inference process of the (k+1)th inference request, the initial second-cached intermediate result and multiple second-cached intermediate results are used to calculate the (k+1)th output result.

[0182] For example, the setup module 230 is further configured to set a second flag for the second buffer. For example, the inference module 220 is further configured to perform the first inference on the (k+1)th input data using a persistent kernel function in response to the (k+1)th input data being written to the second buffer and the second flag being in a host-ready state. For example, in response to the completion of autoregressive inference for the (k+1)th inference request, the setup module 230 is further configured to update the second flag to a device-ready state to indicate that the host can read the (k+1)th output result.

[0183] Figure 8 This is a schematic block diagram of a host-side processor provided for at least one embodiment of the present disclosure.

[0184] For example, such as Figure 8 As shown, the host-side processor 300 includes a sending module 310 and a reading module 320.

[0185] For example, the sending module 310 is configured to send the k-th inference request out of N inference requests to the device. For example, the device runs a persistent kernel function that performs an autoregressive inference operation on each of the N inference requests. After the persistent kernel function performs the autoregressive inference operation on the k-th inference request, the resulting k-th output is stored in the target storage space on the device. Here, N is an integer greater than 1, and k is a positive integer not greater than N. That is, the sending module 310 can be configured to perform, for example... Figure 3 The step S210 is shown.

[0186] For example, the read module 320 is configured to read the k-th output result from the target storage space to the host. That is, the read module 320 can be configured to perform, for example... Figure 3 Step S220 is shown.

[0187] For example, in some examples, the target storage space includes a first buffer, which is configured with a first flag. For example, the sending module 310 is also configured to write the k-th input data of the k-th inference request into the first buffer. For example, the host processor 300 also includes an update module 330 (not shown) configured to update the first flag to a host-ready state, indicating that the persistent kernel function can execute the autoregressive inference operation for the k-th inference request.

[0188] For example, the kth output result obtained from the autoregressive inference operation of the kth inference request is stored in the first buffer. For example, the read module 320 is also configured to read the kth output result from the first buffer to the host in response to the first flag being in a device-ready state.

[0189] For example, in response to the k-th output result being read to the host, the sending module 310 is further configured to send the (k+2)-th inference request to the device in response to the N inference requests including the (k+2)-th inference request to be processed. For example, the updating module 330 is further configured to update the first flag to the host-side ready state, indicating that the device can execute the autoregressive inference operation of the (k+2)-th inference request.

[0190] For example, in some examples, the target storage space also includes a second buffer, which differs from the first buffer and has a second flag bit set. For example, the sending module 310 is further configured to: in response to the N inference requests including the (k+1)th inference request to be processed, send the (k+1)th inference request to the device, wherein the (k+1)th input data of the (k+1)th inference request is written to the second buffer. For example, the updating module 330 is further configured to: update the second flag bit to a host-ready state to indicate that the persistent kernel function can execute the autoregressive inference operation of the (k+1)th inference request. For example, after the persistent kernel function executes the autoregressive inference operation of the (k+1)th inference request, the obtained (k+1)th output result is stored in the second buffer. For example, the reading module 320 is further configured to: in response to the second flag bit being a device-ready state, read the (k+1)th output result from the second buffer to the host.

[0191] For example, in response to the (k+1)th output result being read to the host, the sending module 310 is further configured to send the (k+3)th inference request to the device in response to the N inference requests including the (k+3)th inference request to be processed. For example, the updating module 330 is further configured to update the second flag to a host-ready state, indicating that the device can execute the autoregressive inference operation for the (k+3)th inference request.

[0192] Due to the above description, for example Figure 2 The data processing methods shown and, for example Figure 3 The data processing method described above has already detailed the operations of the device-side processor 200 and the host-side processor 300. Therefore, for the sake of brevity, it will not be repeated here. For relevant details, please refer to the above descriptions. Figures 2-6 The description.

[0193] It should be noted that, in Figure 7 The device-side processor 200 and shown Figure 8 In the host processor 300 shown, each of the aforementioned modules can be configured as software, hardware, firmware, or any combination thereof to perform a specific function. For example, these modules may correspond to dedicated integrated circuits, pure software code, or modules combining software and hardware. As an example, see [reference needed]. Figure 7 and Figure 8 The device described may be a PC computer, tablet device, personal digital assistant, smartphone, web application or other device capable of executing program instructions, but is not limited thereto.

[0194] Furthermore, although the device-side processor 200 and the host-side processor 300 are described above as modules for executing corresponding processes respectively, those skilled in the art will understand that the processes executed by each module can also be executed without any specific module division in the device or without clear boundaries between the modules. In addition, the above references... Figure 7 The described device-side processor 200 and Figure 8 The host-side processor 300 described is not limited to the modules described above, but may also include other modules (e.g., writing modules, control modules, etc.) as needed, or the above modules may be combined.

[0195] At least one embodiment of this disclosure also provides an electronic device including one or more processors and a memory; the memory includes one or more computer program modules; the one or more computer program modules are stored in the memory and configured to be executed by one or more processors, and the one or more computer program modules include methods for implementing the data processing methods provided by the embodiments of this disclosure above. For example, the one or more processors include a host-side processor and a device-side processor provided by at least one embodiment of this disclosure.

[0196] Figure 9 This is a schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0197] For example, such as Figure 9 As shown, the electronic device 400 includes a processor 410 and a memory 420. For example, the memory 420 is used to store non-transitory computer-readable instructions (e.g., one or more computer program modules). The processor 410 is used to execute the non-transitory computer-readable instructions, which, when executed by the processor 410, can perform one or more steps according to the data processing method described above. The memory 420 and the processor 410 can be interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0198] For example, processor 410 may be a central processing unit (CPU), a digital signal processor (DSP), or other processing unit with data processing and / or program execution capabilities, such as a field-programmable gate array (FPGA); for example, the central processing unit (CPU) may be an x86 or ARM architecture. Processor 410 may be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 400 to perform desired functions.

[0199] For example, memory 420 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and processor 410 may run one or more computer program modules to implement various functions of electronic device 400. Various application programs and various data, as well as various data used and / or generated by the application programs, may also be stored in the computer-readable storage medium.

[0200] It should be noted that when the processor 410 is a device-side processor provided in at least one embodiment of the present disclosure, the memory 420 can be the memory of the device-side; when the processor 410 is a host-side processor provided in at least one embodiment of the present disclosure, the memory 420 can be the memory of the host-side.

[0201] It should be noted that, in the embodiments of this disclosure, the specific functions and technical effects of the electronic device 400 can be referred to the description above of the data processing method provided in at least one embodiment of this disclosure, and will not be repeated here.

[0202] Figure 10 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0203] For example, such as Figure 10 As shown, the electronic device 500 is, for example, suitable for implementing the data processing method provided in the embodiments of this disclosure. It should be noted that... Figure 10 The illustrated electronic device 500 is merely an example and does not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0204] For example, such as Figure 10 As shown, the electronic device 500 may include a processing unit 51 (host-side processor, device-side processor) that can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 52 or a program loaded from storage device 58 into random access memory (RAM) 53. For example, as Figure 10 As shown, RAM 53 also stores various programs and data required for the operation of the electronic device 500 used in the cache system simulation. Processing device 51, ROM 52, and RAM 53 are interconnected via bus 54. Input / output (I / O) interface 55 is also connected to bus 54. Typically, the following devices can be connected to I / O interface 55: input devices 56 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 57 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 58 including, for example, magnetic tapes, hard disks, etc.; and communication devices 59. Communication device 59 allows electronic device 500 to communicate wirelessly or wiredly with other electronic devices to exchange data.

[0205] Although Figure 10 An electronic device 500 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 500 may alternatively implement or have more or fewer devices.

[0206] For detailed descriptions and technical effects of the electronic device 500, please refer to the relevant description of the data processing method above, which will not be repeated here.

[0207] Figure 11 This is a schematic diagram of a storage medium provided for at least one embodiment of the present disclosure.

[0208] For example, such as Figure 11 As shown, storage medium 600 stores non-transitory computer-readable instructions 610. For example, when non-transitory computer-readable instructions 610 are executed by a computer, one or more steps in the data processing method described above are performed.

[0209] For example, this storage medium 600 can be applied to Figure 9 The electronic device 400 is shown. For example, the storage medium 600 can be the memory 420 in the electronic device 400. For example, relevant descriptions of the storage medium 600 can be found in [reference needed]. Figure 9 The corresponding description of the memory 420 in the illustrated electronic device 400 will not be repeated here.

[0210] The following points need to be clarified regarding this disclosure:

[0211] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0212] (2) Where there is no conflict, features of the same embodiment and different embodiments of this disclosure can be combined with each other.

[0213] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, This is applied to the device side, where a persistent kernel function runs. During the startup and exit process of the persistent kernel function, it performs a corresponding autoregressive inference operation on each of N inference requests, where N is an integer greater than 1. The data processing method includes: Pre-allocate target storage space in the device; In response to receiving the kth inference request among the N inference requests, the persistent kernel function is used to perform the first inference on the kth input data of the kth inference request to obtain an initial first cached intermediate result, and the initial first cached intermediate result is stored in the target storage space, where k is a positive integer and not greater than N; The initial first cached intermediate result is used as the input to the autoregressive inference of the k-th inference request. The autoregressive inference of the k-th inference request is executed using the persistent kernel function to obtain the k-th output result and multiple first cached intermediate results. The k-th output result and the multiple first cached intermediate results are stored in the target storage space. During the autoregressive inference process of the k-th inference request, the initial first cached intermediate result and the multiple first cached intermediate results are used to calculate the k-th output result.

2. The data processing method according to claim 1, characterized in that, The target storage space includes a first buffer and an intermediate buffer, and the k-th input data is written to the first buffer. Storing the initial first cached intermediate result into the target storage space includes: The initial first cached intermediate result is stored in the intermediate cache. Storing the k-th output result and the plurality of first-cached intermediate results into the target storage space includes: The k-th output result is stored in the first buffer, and the plurality of first-buffered intermediate results are stored in the intermediate buffer.

3. The data processing method according to claim 1, characterized in that, The target storage space includes a first buffer and an intermediate cache. Before performing the first inference on the k-th input data of the k-th inference request using the persistent kernel function, the data processing method further includes: Set the first flag bit for the first buffer. In response to receiving the k-th inference request out of the N inference requests, the first inference is performed on the k-th input data of the k-th inference request using the persistent kernel function, including: In response to the k-th input data being written to the first buffer and the first flag indicating a host-side ready state, the first inference is performed on the k-th input data using the persistent kernel function. In response to the completion of the autoregressive inference execution for the k-th inference request, the data processing method further includes: Update the first flag to the device ready state to indicate that the host can read the k-th output result.

4. The data processing method according to claim 1, characterized in that, The k-th input data includes at least one data unit. In response to receiving the k-th inference request out of the N inference requests, the first inference is performed on the k-th input data of the k-th inference request using the persistent kernel function, including: In response to receiving the k-th inference request, the persistent kernel function is used to perform a first computation on each of the at least one data unit to obtain the initial first cached intermediate result. The initial first cached intermediate result includes the cached intermediate result for each data unit.

5. The data processing method according to claim 1, characterized in that, The target storage space includes a first buffer and an intermediate cache. Before performing the first inference on the k-th input data of the k-th inference request using the persistent kernel function, the data processing method further includes: Based on the size of the k-th input data, a first offset pointer is set in the first buffer, and the first offset pointer is set to point to the end position of the k-th input data. Storing the initial first cached intermediate result into the target storage space includes: The initial first cache intermediate result is stored in the intermediate cache, a second offset pointer is set in the intermediate cache, and the second offset pointer is set to point to the end of the initial first cache intermediate result.

6. The data processing method according to claim 1, characterized in that, The target storage space includes a first buffer and an intermediate cache. The first buffer is configured with a first offset pointer, and the intermediate cache is configured with a second offset pointer. In response to the completion of the autoregressive inference execution for the k-th inference request, the data processing method further includes: The values ​​of both the first offset pointer and the second offset pointer are set to 0, so that the first offset pointer points to the starting position of the first buffer and the second offset pointer points to the starting position of the intermediate buffer.

7. The data processing method according to claim 1, characterized in that, The k-th output result includes M data units. The plurality of first cached intermediate results includes M first cached intermediate results, and the first first cached intermediate result among the plurality of first cached intermediate results includes the initial first cached intermediate result. Using the initial first-cached intermediate result as input to the autoregressive inference of the k-th inference request, the autoregressive inference of the k-th inference request is executed using the persistent kernel function to obtain the k-th output result and multiple first-cached intermediate results. The k-th output result and the multiple first-cached intermediate results are then stored in the target storage space, including: The persistent kernel function is used to predict the first data unit among the M data units based on the initial first cache intermediate result, calculate the first first cache intermediate result of the first data unit, and store the first data unit and the first cache intermediate result of the first data unit in the target storage space. The persistent kernel function is used to perform a second calculation on the j-th data unit and the first to j-th first cache intermediate results among the M data units to obtain the (j+1)-th data unit, and the (j+1)-th data unit is stored in the target storage space. In response to j+1 being less than M, the persistent kernel function is used to perform a third calculation on the (j+1)th data unit to obtain the (j+1)th first cached intermediate result among the M first cached intermediate results, and the (j+1)th first cached intermediate result is stored in the target storage space. Where M is an integer greater than 1, and j is an integer not greater than M-1 and greater than 1.

8. The data processing method according to claim 7, characterized in that, The target storage space includes a first buffer, in which a first offset pointer is configured. Storing the (j+1)th data unit into the target storage space includes: In the first buffer, the (j+1)th data unit is placed at the end of the j-th data unit, and the value of the first offset pointer is incremented by one so that the first offset pointer points to the end of the (j+1)-th data unit.

9. The data processing method according to claim 7, characterized in that, The target storage space includes an intermediate cache, and the intermediate cache contains a second offset pointer. Storing the (j+1)th first cached intermediate result into the target storage space includes: In the intermediate cache, the (j+1)th first cache intermediate result is placed at the end of the j-th first cache intermediate result, and the value of the second offset pointer is incremented by one so that the second offset pointer points to the end of the (j+1)th first cache intermediate result.

10. The data processing method according to claim 2, characterized in that, The target storage space further includes a second buffer, which is different from the first buffer. The data processing method further includes: In response to receiving the (k+1)th inference request out of N inference requests, the persistent kernel function is used to perform the first inference on the (k+1)th input data of the (k+1)th inference request to obtain an initial second cache intermediate result, and the initial second cache intermediate result is stored in the intermediate cache, wherein the (k+1)th input data is written to the second buffer; The initial second-cached intermediate result is used as the input to the autoregressive inference of the (k+1)th inference request. The autoregressive inference of the (k+1)th inference request is executed using the persistent kernel function to obtain the (k+1)th output result and multiple second-cached intermediate results. The (k+1)th output result is stored in the second buffer, and the multiple second-cached intermediate results are stored in the intermediate cache. During the autoregressive inference process of the (k+1)th inference request, the initial second-cached intermediate result and the multiple second-cached intermediate results are used to calculate the (k+1)th output result.

11. The data processing method according to claim 10, characterized in that, The data processing method further includes: Set the second flag bit for the second buffer. In response to receiving the (k+1)th inference request out of N inference requests, the first inference is performed on the (k+1)th input data of the (k+1)th inference request using the persistent kernel function, including: In response to the (k+1)th input data being written to the second buffer and the second flag indicating a host-side ready state, the first inference is performed on the (k+1)th input data using the persistent kernel function. In response to the completion of the autoregressive inference execution for the (k+1)th inference request, the data processing method further includes: Update the second flag to indicate that the device is ready, so as to indicate that the host can read the (k+1)th output result.

12. A data processing method, characterized in that, Applied to the host side, the data processing method includes: The kth inference request out of N inference requests is sent to the device. The device runs a persistent kernel function, which performs a corresponding autoregressive inference operation on each of the N inference requests. After the persistent kernel function performs the autoregressive inference operation on the kth inference request, the kth output result is stored in the target storage space of the device. N is an integer greater than 1, and k is a positive integer not greater than N. The k-th output result is read from the target storage space and sent to the host.

13. The data processing method according to claim 12, characterized in that, The target storage space includes a first buffer, which is configured with a first flag bit. Sending the k-th inference request out of the N inference requests to the device includes: Write the kth input data of the kth inference request into the first buffer; The first flag is updated to indicate that the host is ready, thus indicating that the persistent kernel function can execute the autoregressive inference operation for the k-th inference request. The k-th output result obtained from the autoregressive inference operation of the k-th inference request is stored in the first buffer. Reading the k-th output result from the target storage space to the host terminal includes: In response to the first flag indicating that the device is ready, the k-th output result is read from the first buffer and sent to the host.

14. The data processing method according to claim 13, characterized in that, In response to the k-th output result being read to the host, the data processing method further includes: In response to the fact that the N inference requests also include the (k+2)th inference request to be processed, the (k+2)th inference request is sent to the device. Update the first flag to the host-side ready state to indicate that the device-side can execute the autoregressive inference operation for the (k+2)th inference request.

15. The data processing method according to claim 13, characterized in that, The target storage space further includes a second buffer, which differs from the first buffer in that it is equipped with a second flag bit. The data processing method further includes: In response to the fact that the N inference requests also include a k+1 inference request to be processed, the k+1 inference request is sent to the device, wherein the k+1 input data of the k+1 inference request is written into the second buffer; The second flag is updated to the host-side ready state to indicate that the persistent kernel function can execute the autoregressive inference operation of the (k+1)th inference request, wherein the (k+1)th output result obtained after the persistent kernel function executes the autoregressive inference operation of the (k+1)th inference request is stored in the second buffer. In response to the second flag indicating that the device is ready, the (k+1)th output result is read from the second buffer and sent to the host.

16. The data processing method according to claim 15, characterized in that, In response to the (k+1)th output result being read to the host, the data processing method further includes: In response to the fact that the N inference requests also include the (k+3)th inference request to be processed, the (k+3)th inference request is sent to the device. The second flag is updated to the host-side ready state to indicate that the device-side can execute the autoregressive inference operation for the (k+3)th inference request.

17. A device-side processor, characterized in that, The device-side processor runs a persistent kernel function. During the startup and exit process of the persistent kernel function, it performs a corresponding autoregressive inference operation on each of the N inference requests, where N is an integer greater than 1. The device-side processor includes: The pre-allocation module is configured to pre-allocate target storage space on the device side; The inference module is configured to: in response to receiving the kth inference request among the N inference requests, perform the first inference on the kth input data of the kth inference request using the persistent kernel function, obtain an initial first cached intermediate result, and store the initial first cached intermediate result in the target storage space, where k is a positive integer and not greater than N; The inference module is further configured to: use the initial first cached intermediate result as input to the autoregressive inference of the k-th inference request, execute the autoregressive inference of the k-th inference request using the persistent kernel function, obtain the k-th output result and multiple first cached intermediate results, and store the k-th output result and the multiple first cached intermediate results in the target storage space, wherein, during the autoregressive inference process of the k-th inference request, the initial first cached intermediate result and the multiple first cached intermediate results are used to calculate the k-th output result.

18. A host-side processor, characterized in that, The host-side processor includes: The sending module is configured to send the kth inference request out of N inference requests to the device, wherein the device runs a persistent kernel function, which performs a corresponding autoregressive inference operation on each of the N inference requests. After the persistent kernel function performs the autoregressive inference operation on the kth inference request, the kth output result is stored in the target storage space of the device, where N is an integer greater than 1 and k is a positive integer not greater than N. The reading module is configured to read the k-th output result from the target storage space to the host.

19. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory, including one or more computer program modules; The one or more computer program modules are stored in the memory and configured to be executed by the one or more processors, and the one or more computer program modules are used to implement the data processing method according to any one of claims 1-16.

20. A storage medium, characterized in that, The storage medium stores non-transitory computer-readable instructions that, when executed by a computer, implement the data processing method according to any one of claims 1-16.

Images