Large language model offline inference task inference acceleration method and system under limited resources

Abstract

The application discloses a large language model offline reasoning task reasoning acceleration method and system under limited resources, a model construction module, a dynamic batch processing reorganization module, a dynamic memory management module and an unloading strategy adjustment module; based on the lexGen large language model offline reasoning system, dynamic batch processing reorganization is performed on the reasoning task, dynamic memory management design is combined, reasoning tasks are managed in an iterative level fine-grained manner, occupied memory of completed reasoning tasks is released in real time, and dynamic arrangement is performed on uncompleted reasoning tasks, so that the waste of reasoning resources in the reasoning process is reduced, and the offline reasoning system throughput based on the unloading technology is improved. The application efficiently utilizes the idle memory of hardware, improves the throughput of offline reasoning, reduces the influence of unloading on the reasoning process, and transfers part of tensors in a device with low bandwidth to the idle memory, so that the I / O overhead during transmission is reduced.

Description

Technical Field

[0001] The present invention relates to the field of large language model inference acceleration, namely resource scheduling technology, and more specifically to an optimization method and system for large language model offline inference services under limited resources. Background Art

[0002] With the rapid development of large language models, a large number of large language model applications have emerged. To achieve low-latency, high-throughput inference services, accelerating inference requests for large language models is a hot research direction. Large language model inference tasks can be divided into two types: online and offline. Online inference tasks include chatbots like ChatGPT, which require real-time user interaction. These tasks are latency-sensitive, and their inference completion time directly impacts the user experience. Offline inference tasks include large-scale document translation, document summarization, and structured data processing. These tasks do not require user interaction and therefore do not require timely delivery of inference results. Due to the large number of inference tasks, the number of tokens generated, and the long inference time, these tasks require high-throughput inference systems. Currently, the memory required for inference in large language models is mainly divided into three types: model weights, the KV cache generated during inference, and intermediate variables during inference. As the number of model parameters increases, existing processor memory cannot meet the memory requirements of large language model inference. If you want to perform inference on large language models with limited resources, you need to use memory outside of the GPU. During inference, you can offload some of the memory requirements to devices such as the CPU and disks. Each time, you load a portion of the tensors to be calculated onto the GPU for computation, and then switch to the next portion after the computation is complete. However, this approach incurs significant I / O overhead due to the multiple transfers of data between devices, resulting in longer inference latency and failing to meet the latency requirements of online tasks. For offline inference tasks that are less sensitive to latency, offloading technology can be applied, and high-throughput inference can be achieved by using larger batch sizes during inference.

[0003] In addition, the inference mode in the inference framework is divided into a general sequential inference mode and a zigzag sequential inference mode. The general sequential inference mode prioritizes completing a single batch processing inference task; and the existing offline inference system based on the unloading method adopts a zigzag inference sequence for inference. The zigzag sequential inference mode adds multiple batch processing inference tasks for simultaneous inference on the basis of originally inferring a single batch task. The zigzag inference sequence does not prioritize completing a single batch processing inference task, but rather prioritizes calculating multiple different batch processing (Batch) inference tasks under the same large language model Transformer layer weight, and switches to the next Transformer layer for calculation in the same way until all inference tasks are completed. However, FlexGen cannot remove completed inference tasks in real time during inference, and needs to wait for the longest task in the inference task to complete inference to release all, so that the remaining completed requests continue to output invalid tokens, causing continuous waste of computing power and memory, and increasing inference delay and reducing the throughput of generating valid tokens.

[0004] Therefore, reducing the I / O overhead of the zigzag inference sequence model in the inference framework and improving the throughput by using a larger Batch size are technical problems to be solved by the present application. SUMMARY

[0005] In view of the current ice flood disaster prediction problems and technical limitations, the present application provides a large language model offline inference task inference acceleration method and system under limited resources. The FlexGen inference framework based on unloading under limited resources is used for offline inference tasks, the I / O overhead of the model weight is reduced through a zigzag inference sequence, and the throughput is improved by using a larger Batch size.

[0006] In a first aspect, the present application provides a large language model offline inference task inference acceleration method under limited resources, comprising:

[0007] S1, defining an inference task model for offline inference, including the number of current inference tasks, the batch processing of the current inference tasks, and the unloading strategy during inference under limited resources, the unloading strategy further including the weight of the inference task model in the graphics processing unit, the key value memory in the attention mechanism of the large language model during inference, starting an inference service, traversing the inference tasks, setting the number of iterations, and iteratively calculating the inference task model;

[0008] S2, when receiving the inference request, performing the inference task, performing dynamic batch processing reorganization of the inference task, including marking the completed inference task from the inference process when generating the Token each time, and selecting the uncompleted inference task from the last batch to place in the batch with the completed inference task for dynamic batch processing reorganization, and moving the key value memory corresponding to the inference task to the designated memory;

[0009] S3, when there are remaining inference requests in the inference task queue, performing dynamic memory management, including releasing the idle memory when the completed tasks exceed the defined batch size of the current inference task, and deleting the completed inference task from the inference process, and moving the remaining inference key value memory and the buffer memory address allocated by the intermediate activation;

[0010] S4, readjusting the offloading strategy, including increasing the proportion of key value memory in the device with high bandwidth when the system detects that the number of released requests is sufficient, reducing the proportion of key value memory in the device with low bandwidth, and transferring the key value memory placed in the device with low bandwidth to the idle memory released in S3 to reduce the data transmission time during system inference.

[0011] In some embodiments, the traversing inference task of S1 further includes traversing the inference task of the user in a zigzag inference order, and each row represents the result of multi-layer inference of the inference task model, and each column represents the result calculated by different batches under the same model weight.

[0012] In some embodiments, S1 further includes, when the number of completed inference tasks finTaskNum exceeds Bs, removing the last ReduceBatchNum batches from the inference process, wherein the calculation method of ReduceBatchNum is:

[0013] ReduceBatchNum = finTaskNum / / Bs

[0014] Wherein, ReduceBatchNum represents the number of reduced batches, finTaskNum represents the number of completed inference tasks, and Bs represents the batch task size currently set by the system.

[0015] In some embodiments, S3 further includes, after releasing the memory occupied by the completed inference task, transferring the tensor part in the device with low bandwidth to the idle memory.

[0016] In some embodiments, in S3, the released idle memory is as follows:

[0017]

[0018] The memory occupied by the remaining inference tasks in the key-value memory (KV Cache) of the three devices is:

[0019]

[0020] wherein, represents the number of free memories of the GPU, and Cg represents the weight proportion of the inference task model allocated on the GPU, represents the number of free memories of the CPU, and Cc represents the weight proportion of the inference task model allocated on the CPU, represents the memory occupied by the remaining inference tasks in the KV Cache of the GPU, represents the memory occupied by the remaining inference tasks in the KV Cache of the CPU, represents the memory occupied by the remaining inference tasks in the Disk, Bn represents the number of batches set by the current inference task, Cd represents the weight of the inference task model allocated on the Disk, and Batchsize represents the batch size.

[0021] In a second aspect, the present application provides a large language model offline inference task inference acceleration system under limited resources, comprising:

[0022] a model construction module for defining an inference task model for offline inference, including the number of current inference tasks, the batch processing of the current inference task, and the unloading strategy during inference under limited resources, wherein the unloading strategy further includes the weight of the inference task model in the graphics processor, the key-value memory in the attention mechanism of the large language model generated during inference, starting the inference service, traversing the inference tasks, setting the number of iterations, and iteratively performing the calculation of the inference task model;

[0023] a dynamic batch reorganization module for performing inference tasks and dynamic batch reorganization of the inference tasks after receiving an inference request, including marking the completed inference tasks from the inference process and selecting the uncompleted inference tasks from the last batch to place in the batch with completed inference tasks for dynamic batch reorganization, and moving the key-value memory of the corresponding inference task to the designated memory;

[0024] a dynamic memory management module for performing dynamic memory management when there are remaining inference requests in the inference task queue, including releasing the free memory when the completed tasks exceed the defined batch size of the current inference task, and deleting the completed inference tasks from the inference process and moving the key-value memory of the remaining inference and the buffer memory address allocated by the intermediate activation;

[0025] The unloading strategy adjustment module is configured to readjust the unloading strategy, including increasing the proportion of the key value memory in the device with high bandwidth, reducing the proportion of the key value memory in the device with low bandwidth, and transferring the key value memory placed in the device with low bandwidth to the idle memory released in the S3 to reduce the data transmission time when the system detects that the number of released requests is sufficient.

[0026] In some embodiments, the model construction module, the traversing the inference tasks further comprises traversing the inference tasks of the user in a zigzag inference order, and each row represents the result of a multi-layer inference of the inference task model of each batch, and each column represents the result calculated by different batches under the same model weight.

[0027] The inference task model further comprises removing the last ReduceBatchNum batches from the inference process when the number of completed inference tasks finTaskNum exceeds Bs, and the calculation method of ReduceBatchNum is as follows:

[0028] ReduceBatchNum = finTaskNum / / Bs

[0029] Wherein, ReduceBatchNum represents the number of reduced batches, finTaskNum represents the number of completed inference tasks, and Bs represents the batch task size set by the current system

[0030] In some embodiments, the dynamic memory management module further comprises transferring the tensor part in the device with low bandwidth to the idle memory after releasing the memory occupied by the completed inference tasks.

[0031] In some embodiments, the dynamic memory management module further comprises that the idle memory released is as follows:

[0032]

[0033] The memory occupied by the remaining inference tasks in the key value memory KV Cache in the three devices is as follows:

[0034]

[0035] Wherein, represents the number of idle memories of the GPU, and Cg represents the proportion of the inference task model weight allocated on the GPU, represents the number of idle memories of the CPU, and Cc represents the proportion of the inference task model weight allocated on the CPU, Memory occupied by KV Cache of the remaining inference tasks in the GPU, Memory occupied by KV Cache of the remaining inference tasks in the CPU, Memory occupied by KV Cache of the remaining inference tasks in the Disk, Bn represents the number of batch processing set by the current inference task, Cd represents the inference task model weight allocated on the Disk, and Batchsize represents the batch size.

[0036] Compared with the prior art, the beneficial technical effects and technical progress achieved by the present application are as follows:

[0037] 1) The inference tasks are dynamically batched and reorganized, combined with dynamic memory management design, the inference tasks that have been completed are released in real time, and the inference tasks that have not been completed are dynamically arranged, so as to reduce the waste of inference resources in the inference process and improve the throughput of the offline inference system based on the unloading technology. Therefore, the present application can provide dynamic batch reorganization and flexible memory management, efficiently utilize the idle memory of the hardware, and improve the throughput of offline inference.

[0038] 2) A dimension composed of multiple batches is added on the linear level of the number of iterations to complete the zigzag inference order to traverse the user's inference tasks, the loading overhead is reduced by overlapping memory and calculation, and the inference is performed according to the batch dimension during calculation to increase the proportion of calculation and loading and reduce the impact of unloading on the inference process.

[0039] 3) The tensor part in the device with low bandwidth is transferred to the idle memory to reduce the I / O overhead during inference. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A flowchart of the large language model offline inference task inference acceleration method under the limited resources of the present application;

[0041] Figure 2 A flowchart of the zigzag inference mode adopted by the present application;

[0042] Figure 3 A dynamic batch process diagram during inference of the present application;

[0043] Figure 4 A diagram of reallocating KV Cache memory of unfinished inference tasks during inference of the present application;

[0044] Figure 5This is a module diagram of the inference acceleration system for offline inference tasks of large language models under limited resources of the present invention. DETAILED DESCRIPTION

[0045] The technical solution of the present invention is described below in conjunction with specific embodiments and drawings.

[0046] Example 1

[0047] like Figure 1 As shown, the inference acceleration method for offline inference tasks of large language models under limited resources of the present invention adopts a zigzag inference order, and the specific process includes the following steps:

[0048] Step 1. Formalize the parameters for offline reasoning of the inference task model, including: using a two-tuple to represent the number of current reasoning tasks, defined as Bs and Bn, where Bs and Bn represent the batch size (Batchsize) and the number of batches (Batch) set for the current reasoning task, respectively; using a sextet to represent the current offloading strategy for reasoning under limited resources, defined as Wg, Wc, Wd, Cg, Cc, Cd, where Wg, Wc, and Wd represent the percentages of the inference task model weights on the graphics processing unit (GPU), central processing unit (CPU), and hard disk (Disk), respectively; and Cg, Cc, and Cd represent the percentages of the key-value cache (KV Cache) in the attention mechanism of the large language model (LLM) generated during reasoning on the GPU, CPU, and disk, respectively; starting the inference service, traversing the user's reasoning tasks in a zigzag reasoning order, setting the number of iterations, executing the complete calculation process of the inference task model for each iteration, and calculating a token for each iteration. The system continues this process until the generated token represents the stop "EOS", and the iteration process ends;

[0049] Step 101: Set all current reasoning task models to use the zigzag reasoning method. Specifically, Figure 3 As shown in the figure, a dimension consisting of multiple batches including Batch1, Batch2, Batch3, and Batch5 is added to the linear level of the number of iterations, expanding the reasoning process from linear to a plane. After initialization, the reasoning process is started. The system performs reasoning layer by layer, including layer1, layer2, and layer3, and loads the next layer while calculating the current layer. The overhead required for loading is reduced by overlapping memory access and calculation. During calculation, reasoning is prioritized according to the batch dimension to increase the ratio of calculation to loading and reduce the impact of offloading on the reasoning process. The user's reasoning tasks are traversed in a zigzag reasoning order.

[0050] Step 102: Allocate the inference task model weights and key-value memory KV Cache to each device including the GPU, CPU, and Disk based on the offline inference parameters Bs, Bn, Wg, Wc, Wd, Cg, Cc, and Cd. During subsequent inference, load the tensors not on the GPU into the cache buffer and transfer them to the GPU for inference.

[0051] Step 2: Execute the inference task. During the execution of the inference task, check whether there are any completed inference tasks and repackage the task batches (Batching);

[0052] Step 201: The system checks the execution process of the inference task. When generating a token at the end of each iteration, it checks the completion status of the current inference task, marks the completed inference requests in the inference task queue, deletes them from the inference task queue, and releases the KV Cache occupied by these completed requests. After all iterations are completed, the system checks the completion status of all inference tasks. When a completed inference task is encountered during the execution of the inference task, it removes it from the corresponding batch and searches for unfinished inference tasks from the back to the front. It fills the vacant position of the completed inference task with the unfinished inference task, reconfigures the inference task batching, and exchanges the KV Caches of the two inference tasks.

[0053] like Figure 2 As shown in the zigzag inference process, each row represents the result of a batch passing through the multi-layer inference of the inference task model, while each column represents the results of different batches calculated under the same model weights. When the number of completed tasks exceeds Bs, the second function puts the completed tasks into the last batch. At this point, it is no longer necessary to retain them in the inference process and can be removed from it. This is reflected in the calculation of the last row in the inference process.

[0054] like Figure 3 As shown in the figure, dynamic batching of inference tasks is implemented to achieve fine-grained management. Each time a token is generated, the completed inference task is marked from the inference process, and the unfinished inference task is selected from the last batch and placed in the batch with completed inference tasks for dynamic reorganization of the batch, and the KV cache of the corresponding inference task is moved to the specified memory. When the completed inference tasks reach the size of Bs, it means that the last batch is made up of completed tasks or unfinished tasks that have been moved to the remaining batches through the movement of inference tasks. At this time, there is no need to reason about this batch anymore, and it can be removed from the inference process. Figure 2The flow for realizing dynamic batching is shown. As shown in Table 1, the pseudo code for the dynamic batching algorithm is shown.

[0055]

[0056]

[0057] Step 3, when the completed tasks exceed the defined batch size Bs of the current inference task, the number Bn of batches of the current inference task is reduced by one, at this time the inference task includes one batch composed of completed tasks, the memory occupation of this inference task batch is released, and it is removed from the inference flow, and the remaining inference request reorganization in the inference task queue is performed, and the remaining inference KV Cache and the buffer memory address allocated by the intermediate activation are moved;

[0058] Step 301, after the memory occupied by the completed inference task is released, there will be idle memory in the device, at this time the offloading ratio parameter can be adjusted, and the tensors in the device with lower bandwidth are transferred to the idle memory to reduce the I / O overhead during inference;

[0059] Step 4, readjust the offloading strategy, specifically, on the basis of step 301, readjust Wg, Wc, Wd, Cg, Cc, and Cd, that is, when the system detects that the number of released requests is sufficient, the KV Cache placement strategy for offloading is regenerated. The KV Cache placement strategy for offloading includes the parameters Cg, Cc, and Cd defined in step 1, and the system will adjust these parameters, increase the proportion of KV Cache in the device with higher bandwidth, and reduce the proportion of KV Cache in the device with lower bandwidth. The KV Cache placed in the device with lower bandwidth is transferred to the idle memory released in step 3, so as to reduce the data transmission time during system inference.

[0060] Further, the step 2 needs to support the request completion judgment of the model itself and the longest request length given by the user after judging whether the request is completed.

[0061] Further, when reorganizing the request, the system will form new Batch from the remaining requests. When the number of completed requests is greater than Batchsize, the system will release the KV Cache of the request and also remove the flow for executing them from the inference.

[0062] Further, the step 4, when executed, will preferentially move the KV Cache to a device with higher bandwidth to reduce the amount of data that needs to be loaded. Since the KV Cache space is allocated according to the request when initialized, only the corresponding index needs to be moved when the KV Cache is moved while recombining the request to form a new Batch.

[0063] In order to reduce the overhead of switching model weights caused by offloading, the zigzag inference method preferentially uses the same model weight to calculate different Batches, so that the overhead of loading the model weight is distributed to multiple Batches. When switching Batches, the KV Cache and the intermediate variables generated by the calculation of each Batch need to be loaded, and the memory occupied by this part also determines the maximum size of Bn and Bs that the hardware can accommodate. As shown in Table 2, the pseudo code of the internal execution process of the zigzag inference method is shown. First, according to the parameters related to offloading, the model weight, KV Cache and intermediate variables are allocated with corresponding memory and buffer on each machine to load the offloaded tensors. When performing inference, Tokens are generated in an iterative manner. Each iteration uses the model weight of one layer of the calculation model to calculate all Batches and obtains the inference result. After traversing all layers of the model, the generation of one Token is completed. In this process, the model weight of each layer needs to be loaded, the KV Cache and intermediate variables of each Batch need to be loaded, and after the calculation is completed, the KV Cache and intermediate variables of each Batch are stored in the allocated memory. When all tasks meet the conditions for stopping inference, the inference process is ended. The zigzag inference method uses multiple Batches and the same model weight to calculate, at the cost of switching smaller KV Cache and intermediate variables, to reduce the overhead of switching model weights. If the sequence length of the model inference is too long, causing the KV Cache switching time to be longer than the weight switching time, the model weight can also be preferentially switched by reducing Bn, realizing a flexible inference method. Note that the zigzag inference can only achieve higher throughput compared to the general inference process in the inference using offloading technology. In the case of sufficient inference resources, the throughput of the two inference methods is not much different. The execution process of the zigzag offline inference.

[0064]

[0065]

[0066] For the fine-grained inference task management of the zigzag inference, the specific implementation process is as follows:

[0067] In the inference process, due to the inference characteristics of large language models, each inference task can only be generated <eos>The inference stops only when the Token is finished, which causes different inference iterations for different inference tasks, i.e., different inference times for different inference tasks. When a Batch is inferred, the inference tasks in the Batch that have been inferred need to wait for the longest inference task in the Batch to be executed. The Tokens generated during the waiting process are invalid Tokens and will not be put into the output result, causing waste of inference resources.

[0068] For dynamic allocation of memory in the inference process using the offloading mode, the specific implementation process is as follows:

[0069] In the fine-grained inference task management of zigzag inference, when the number of completed inference tasks finTaskNum exceeds Bs, the last ReduceBatchNum Batch will be removed from the inference process, and the calculation method of ReduceBatchNum is as follows:

[0070] ReduceBatchNum = finTaskNum / / Bs

[0071] Wherein, ReduceBatchNum represents the number of reduced batches, finTaskNum represents the number of completed inference tasks, and Bs represents the batch task size currently set by the system.

[0072] At the same time, the memory occupied by the inference tasks in these Batch will be released. This part of the memory is distributed on the GPU, CPU and Disk according to the parameters set during the initialization of offline inference. This part of the memory includes the KV Cache of the inference task and the memory allocated by the inference task for storing intermediate variables during inference. After releasing this part of the memory, there will be idle memory on the GPU and CPU, so the memory can be reallocated, and the tensors stored in the lower bandwidth device can be moved to the higher bandwidth device, such as moving from Disk to CPU, or from CPU to GPU. The execution process of dynamic allocation of memory in the inference process using the offloading mode is shown in the pseudo code of Table 3. The idle memory after release is:

[0073]

[0074] The memory occupied by the remaining inference tasks in the key-value memory KV Cache in the three devices is:

[0075]

[0076] Wherein, represents the number of idle memory on the GPU, Cg represents the proportion of the model weight of the inference task allocated on the GPU, Cc represents the weight proportion of the inference task model allocated on the CPU, Bn represents the number of batches set by the current inference task, Cd represents the weight of the inference task model allocated on the Disk, and Batchsize represents the batch size. Bn represents the number of batches set by the current inference task, Cd represents the weight of the inference task model allocated on the Disk, and Batchsize represents the batch size. Bn represents the number of batches set by the current inference task, Cd represents the weight of the inference task model allocated on the Disk, and Batchsize represents the batch size.

[0077] In the initialization of the allocated KV Cache memory, the system allocates the KV Cache of each inference task in the batch to different devices according to the Batch, so that when the idle memory can accommodate one request of each Batch in the remaining Bn-ReduceBatchNum Batch, the memory can be reallocated. However, if only a small number of inference tasks are moved each time, the memory of the inference task will be frequently reallocated, resulting in a large amount of additional overhead. Therefore, the system designs a hyperparameter minTask to specify the minimum number of inference tasks moved each time. In order to ensure the order of the KV Cache, the system needs to fill the free memory of the GPU with the tensors in the CPU first, and then move the tensors in the Disk to the free memory of the CPU. By moving the tensors from the device with lower bandwidth to the device with higher bandwidth, the I / O overhead of each iteration is reduced, the latency of each iteration is reduced, and the overall throughput of the system is improved.

[0078]

[0079]

[0080] Embodiment Two

[0081] As shown in Figure 5 The large language model offline inference task inference acceleration system under limited resources of the present application, the system comprises a model construction module 100, a dynamic batch reorganization module 200, a dynamic memory management module 300 and an unloading strategy adjustment module 400, which is used to execute the overall flow of the large language model offline inference task inference acceleration method under limited resources in embodiment one, wherein:

[0082] The model construction module 100 is used for defining an inference task model for offline inference, including the number of current inference tasks, the batch processing of current inference tasks, and an unloading strategy during inference under limited resources, the unloading strategy further including the weight of the inference task model in a graphics processor, the key-value memory in an attention mechanism of a large language model during inference, starting an inference service, traversing inference tasks, setting the number of iterations, and iteratively performing calculation of the inference task model;

[0083] The dynamic batch reorganization module 200 is used for performing inference tasks and dynamic batch reorganization of the inference tasks after receiving an inference request, including marking the completed inference tasks from the inference process when generating a Token each time, and placing the uncompleted inference tasks from the last batch into the batch in which the completed inference tasks exist for dynamic batch reorganization, while moving the key-value memory of the corresponding inference tasks to the specified memory;

[0084] The dynamic memory management module 300 is used for performing dynamic memory management when there are remaining inference requests in the inference task queue, including releasing the idle memory when the completed tasks exceed the defined batch size of the current inference tasks, and deleting the completed inference tasks from the inference process, and moving the key-value memory of the remaining inference and the buffer memory address allocated by the intermediate activation;

[0085] The unloading strategy adjustment module 400 is used for readjusting the unloading strategy, including increasing the proportion of the key-value memory in the device with high bandwidth when the system detects that the number of released requests is sufficient, reducing the proportion of the key-value memory in the device with low bandwidth, and transferring the key-value memory placed in the device with low bandwidth to the idle memory released in the S3, so as to reduce the data transmission time during system inference.

[0086] In summary, the present application is based on the FlexGen large language model offline inference system, and designs the dynamic batch processing (Batching) and dynamic memory management of the inference tasks, manages the inference tasks at the iteration level in a fine-grained manner, releases the memory occupied by the completed inference tasks in real time, dynamically arranges the uncompleted inference tasks, reduces the waste of inference resources during inference, and improves the throughput of the offline inference system based on the unloading technology.

[0087] The above merely illustrates the specific implementation of the present application, and the above implementation steps are only used to help understand the specific method and core idea of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of several changes or equivalent replacements, combinations and refinements without departing from the principle of the present application, and these changes or equivalent replacements, combinations and refinements should be considered to fall within the protection scope of the present application, and should be covered in the protection scope of the present application.< / eos>

Claims

1. A method for accelerating inference of large language model offline inference tasks under limited resources, characterized by: include: S1, define the inference task model for offline inference, including the number of current inference tasks, the batch processing of current inference tasks, and the offloading strategy when inferring under limited resources. The offloading strategy further includes the weight of the inference task model in the graphics processor, the key-value memory in the attention mechanism of the large language model generated during inference, start the inference service, traverse the inference tasks, set the number of iterations, and iteratively calculate the inference task model; S2, upon receiving an inference request, executes the inference task and performs dynamic batch reorganization of the inference task. This includes marking the completed inference task from the inference process each time a token is generated, selecting unfinished inference tasks from the last batch and placing them into the batch with completed inference tasks for dynamic batch reorganization. At the same time, the key-value memory of the corresponding inference task is moved to the designated memory. S3, when there are remaining inference requests in the inference task queue, dynamic memory management is performed, including releasing free memory when the completed tasks exceed the defined batch size of the current inference task, deleting the completed inference tasks from the inference process, and moving the remaining inference key-value memory and the buffer memory address allocated by the intermediate activation; S4, readjusting the offloading strategy, including increasing the proportion of key-value memory in bandwidth devices and reducing the proportion of key-value memory in bandwidth devices when the system detects that the number of released requests is sufficient, and transferring the key-value memory placed in the device with lower bandwidth to the free memory released in S3, so as to reduce the data transmission time during system reasoning.

2. The method for accelerating inference of large language model offline inference tasks under limited resources according to claim 1 is characterized in that: The traversal reasoning task of S1 further includes traversing the user's reasoning tasks in a zigzag reasoning order, with each row representing the result of a batch processing through multi-layer reasoning of the reasoning task model, and each column representing the result of different batch processing calculated under the same model weight.

3. The method for accelerating inference of large language model offline inference tasks under limited resources according to claim 1, characterized in that: Said S1 further includes, when the number of completed reasoning tasks finTaskNum exceeds Bs, removing the last one from the reasoning process Batch, where The calculation method is: ; in, represents the reduced number of batches, Indicates the number of tasks that have completed reasoning, Indicates the batch processing task size currently set by the system.

4. The method for accelerating inference of large language model offline inference tasks under limited resources according to claim 1, characterized in that: The S3 further includes, after releasing the memory occupied by the completed inference task, using the free memory to transfer the tensor portion in the low-bandwidth device to the free memory.

5. The method for accelerating inference of large language model offline inference tasks under limited resources according to claim 1, characterized in that: In the S3, the free memory after release is represented as follows: ; ; The memory occupied by the KV Cache of the remaining inference tasks on the three devices is: ; ; ; in, Indicates the amount of free memory on the GPU. Indicates the weight ratio of the inference task model allocated to the GPU, Indicates the amount of free memory in the CPU. Indicates the weight ratio of the inference task model allocated to the CPU. Indicates the memory occupied by the KV Cache in the GPU for the remaining inference tasks. Indicates the memory occupied by the KV Cache in the CPU for the remaining inference tasks. The memory occupied by the remaining inference tasks in the KV Cache on Disk, Indicates the number of batches set for the current inference task. Indicates the weight of the inference task model allocated on Disk, Indicates the batch size.

6. A large language model offline reasoning task acceleration system under limited resources, characterized by: The system includes a model building module, a dynamic batch reorganization module, a dynamic memory management module, and an unloading strategy adjustment module, wherein: A model building module is used to define the inference task model for offline inference, including the number of current inference tasks, the batch processing of current inference tasks, and the offloading strategy when inferring under limited resources. The offloading strategy further includes the weight of the inference task model in the graphics processor, the key-value memory in the attention mechanism of the large language model generated during inference, starting the inference service, traversing the inference tasks, setting the number of iterations, and iteratively calculating the inference task model. The dynamic batch reorganization module is used to execute inference tasks and dynamically reorganize inference tasks after receiving inference requests. This includes marking completed inference tasks from the inference process each time a token is generated, selecting unfinished inference tasks from the last batch and placing them into the batch with completed inference tasks for dynamic batch reorganization. At the same time, the key-value memory of the corresponding inference task is moved to the specified memory. A dynamic memory management module is used to perform dynamic memory management when there are remaining inference requests in the inference task queue, including releasing free memory when the completed tasks exceed the defined batch size of the current inference task, deleting the completed inference tasks from the inference process, and moving the remaining inference key-value memory and the buffer memory address allocated by the intermediate activation; The unloading policy adjustment module is used to readjust the unloading policy, including increasing the proportion of key-value memory in devices with high bandwidth and reducing the proportion of key-value memory in devices with low bandwidth when the system detects that the number of released requests is large enough, and transferring the key-value memory placed in devices with low bandwidth to the free memory released in the dynamic memory management module, so as to reduce the data transmission time during system reasoning.

7. The inference acceleration system for offline inference tasks of large language models under limited resources according to claim 6 is characterized in that: In the model building module, traversing the reasoning tasks further includes traversing the user's reasoning tasks in a zigzag reasoning order, with each row representing the result of a batch processing through multiple layers of reasoning of the reasoning task model, and each column representing the results calculated by different batch processing under the same model weight.

8. The inference acceleration system for offline inference tasks of large language models under limited resources according to claim 6 is characterized in that: The reasoning task model further includes: when the number of completed reasoning tasks finTaskNum exceeds Bs, the last task will be removed from the reasoning process. Batch, where The calculation method is: ; in, represents the reduced number of batches, Indicates the number of tasks that have completed reasoning, Indicates the batch processing task size currently set by the system.

9. The inference acceleration system for offline inference tasks of large language models under limited resources according to claim 6, characterized in that: The dynamic memory management module further includes releasing the memory occupied by the completed inference task and utilizing the free memory to transfer the tensor portion in the device with lower bandwidth to the free memory.

10. The inference acceleration system for offline inference tasks of large language models under limited resources according to claim 6, characterized in that: The dynamic memory management module further includes that the free memory after release is represented as follows: ; ; The memory occupied by the key-value memory of the remaining inference tasks on the three devices is: ; ; ; in, Indicates the amount of free memory on the GPU. Indicates the weight ratio of the inference task model allocated to the GPU, Indicates the amount of free memory in the CPU. Indicates the weight ratio of the inference task model allocated to the CPU. Indicates the memory occupied by the key-value memory of the remaining inference tasks in the GPU. Indicates the memory occupied by the key-value memory of the remaining inference tasks in the CPU. The memory occupied by the key-value memory of the remaining inference tasks in Disk, Indicates the number of batches set for the current inference task. Indicates the weight of the inference task model allocated on Disk, Indicates the batch size.

Images