Method and system for accelerating reasoning of offline reasoning task of large language model under limited resources

Abstract

The invention discloses an off-line reasoning task reasoning acceleration method and system for a large language model under limited resources. The system comprises a model construction module, a dynamic batch processing recombination module, a dynamic memory management module and an unloading strategy adjustment module. Based on a lexGen large language model offline inference system, inference tasks are subjected to dynamic batch processing recombination, dynamic memory management design is combined, the inference tasks are managed in an iteration level in a fine-grained mode, memory occupied by the completed inference tasks is released in real time, and uncompleted inference tasks are dynamically arranged. The waste of reasoning resources in the reasoning process is reduced, and the throughput of the off-line reasoning system based on the unloading technology is improved. According to the method, the hardware idle memory is efficiently utilized, and the throughput of off-line reasoning is improved; the influence of unloading on the reasoning process is reduced; and a tensor part in equipment with lower bandwidth is transferred into the part of the memory by utilizing the idle memory, so that the I / O overhead of transmission during reasoning is reduced.

Description

Technical Field

[0001] The present invention relates to the field of large language model reasoning acceleration, namely resource scheduling technology, and more specifically to an optimization method and system for large language model offline reasoning services under limited resources. Background Art

[0002] With the rapid development of large language models, a large number of large language model applications have also emerged. In order to achieve low-latency and high-throughput reasoning services, accelerating the reasoning requests of large language models is a hot research direction in the future. Large language model reasoning tasks can be divided into two types: online reasoning tasks and offline reasoning tasks. The application of online reasoning tasks includes chatbots such as ChatGPT, which need to interact with users in real time. This type of task is sensitive to latency, and their reasoning completion time will directly affect the user experience; offline reasoning task applications include translation tasks of a large number of documents, document summary extraction, structured data processing, etc. This type of task does not require interaction with users, so it is not necessary to return the reasoning results to users in a timely manner. Due to the large number of reasoning tasks, the number of tokens generated by reasoning is large, and the reasoning time is long, this type of task requires a higher throughput reasoning system for reasoning. At present, the memory required for reasoning of large language models is mainly divided into three types: model weights, KV Cache generated during reasoning, and intermediate variables during reasoning. As the number of model parameters increases, the existing processor memory cannot meet the memory requirements of large language models during reasoning. If you want to implement reasoning for a large language model with limited resources, you need to use memory other than the GPU. During reasoning, you need to offload some of the memory requirements to the CPU, DISK, and other devices. Each time, you need to load some of the tensors that need to be calculated to the GPU for calculation, and switch to the next part of the tensors after the calculation is completed. However, this method will bring a lot of I / O overhead due to the multiple transfers of data in the device, resulting in a long latency of the reasoning task, which cannot meet the latency requirements of online tasks. For offline reasoning tasks that are not sensitive to latency, you can apply offloading technology and achieve high-throughput reasoning by using a larger batch size during reasoning.

[0003] In addition, the reasoning methods in the reasoning framework are divided into general sequential reasoning methods and zigzag sequential reasoning methods. Among them, the general sequential reasoning method gives priority to completing a single batch reasoning task; the existing offline reasoning system reasoning method based on the offloading method adopts a zigzag reasoning order for reasoning. The zigzag sequential reasoning method adds multiple batch reasoning tasks to the original reasoning of a single batch task and performs reasoning at the same time. The zigzag reasoning order does not give priority to completing a single batch reasoning task, but gives priority to calculating multiple different batch reasoning tasks under the same large language model Transformer layer weight. After the calculation is completed, it switches to the next Transformer layer to calculate in the same way until all reasoning tasks are completed. However, FlexGen cannot remove the completed reasoning tasks in real time during the reasoning process. It needs to wait for the longest task in the reasoning task to complete the reasoning and release them uniformly, so that the remaining completed requests continue to output invalid tokens, resulting in a continuous waste of computing power and memory, and at the same time causing an increase in reasoning latency and a decrease in the throughput of generating valid tokens.

[0004] Therefore, reducing the I / O overhead of the zigzag reasoning sequential model in the reasoning framework and improving the throughput by using a larger batch size are technical problems that need to be solved urgently in the present invention. Summary of the invention

[0005] In view of the current difficulties in forecasting ice flood disasters and the existing technical limitations, the present invention proposes a method and system for accelerating the offline reasoning tasks of large language models under limited resources. The offload-based reasoning framework FlexGen reduces the I / O overhead of the model weights through a zigzag reasoning order for offline reasoning tasks under limited resources, and improves the throughput by using a larger batch size.

[0006] In a first aspect, the present invention provides a method for accelerating reasoning of offline reasoning tasks of a large language model under limited resources, comprising:

[0007] S1, the definition of the inference task model for offline inference, including the number of current inference tasks, the batch processing of current inference tasks, and the unloading strategy for current inference under limited resources, wherein the unloading strategy further includes the weight of the inference task model in the graphics processor, the key-value memory in the attention mechanism in the large language model generated during inference, the inference service is started, the inference tasks are traversed, the number of iterations is set, and the calculation of the inference task model is iteratively performed;

[0008] S2, when receiving the inference request, execute the inference task and perform dynamic batch reorganization of the inference task, including marking the completed inference task from the inference process every time a token is generated, and selecting the unfinished inference task from the last batch and placing it into the batch with the completed inference task for dynamic batch reorganization, and moving the key value memory of the corresponding inference task to the specified memory;

[0009] S3, when there are remaining inference requests in the inference task queue, dynamic memory management is performed, including releasing free memory when the completed tasks exceed the defined batch size of the current inference task, deleting the completed inference task from the inference process, and moving the remaining inference key value memory and the buffer memory address allocated by the intermediate activation;

[0010] S4, readjusting the unloading strategy, including increasing the proportion of key-value memory in bandwidth devices and reducing the proportion of key-value memory in low-bandwidth devices when the system detects that the number of released requests is large enough, and transferring the key-value memory placed in the device with lower bandwidth to the free memory released in S3, so as to reduce the data transmission time during system reasoning.

[0011] In some embodiments, the traversal reasoning task of S1 further includes traversing the user's reasoning tasks in a zigzag reasoning order, with each row representing the result of a batch of multi-layer reasoning through the reasoning task model, and each column representing the results of different batches calculated under the same model weights.

[0012] In some implementations, S1 further includes, when the number of completed reasoning tasks finTaskNum exceeds Bs, removing the last ReduceBatchNum batches from the reasoning process, wherein ReduceBatchNum is calculated as follows:

[0013] ReduceBatchNum=finTaskNum / / Bs

[0014] Among them, ReduceBatchNum represents the number of reduced batches, finTaskNum represents the number of tasks that have completed reasoning, and Bs represents the batch task size set by the current system.

[0015] In some implementations, S3 further includes, after releasing the memory occupied by the completed inference task, using the free memory to transfer the tensor portion in the device with lower bandwidth to the free memory.

[0016] In some implementations, in S3, the free memory after release is represented as follows:

[0017]

[0018] The memory occupied by the key-value memory KV Cache of the remaining inference tasks in the three devices is:

[0019]

[0020] in, Indicates the amount of free memory on the GPU, Cg indicates the weight ratio of the inference task model allocated on the GPU, Indicates the amount of free memory on the CPU, Cc indicates the weight ratio of the inference task model allocated to the CPU, Indicates the memory occupied by the KV Cache in the GPU for the remaining inference tasks. Indicates the memory occupied by the KV Cache in the CPU for the remaining inference tasks. The memory occupied by the remaining inference tasks in the KV Cache on Disk. Bn indicates the number of batches set for the current inference task. Cd indicates the inference task model weight allocated on Disk. Batchsize indicates the batch size.

[0021] In a second aspect, the present invention provides a large language model offline reasoning task reasoning acceleration system under limited resources, comprising:

[0022] A model building module, which is used to define the inference task model for offline inference, including the number of current inference tasks, the batch processing of current inference tasks, and the unloading strategy for current inference under limited resources, wherein the unloading strategy further includes the weight of the inference task model in the graphics processor, the key-value memory in the attention mechanism in the large language model generated during inference, start the inference service, traverse the inference tasks, set the number of iterations, and iteratively calculate the inference task model;

[0023] The dynamic batch reorganization module is used to execute the inference task after receiving the inference request, and perform dynamic batch reorganization of the inference task, including marking the completed inference task from the inference process every time a token is generated, and selecting the unfinished inference task from the last batch and placing it into the batch with the completed inference task for dynamic batch reorganization, and moving the key value memory of the corresponding inference task to the specified memory;

[0024] A dynamic memory management module is used to perform dynamic memory management when there are remaining inference requests in the inference task queue, including releasing free memory when the completed tasks exceed the defined batch size of the current inference task, deleting the completed inference task from the inference process, and moving the remaining inference key value memory and the buffer memory address allocated by the intermediate activation;

[0025] The unloading strategy adjustment module is used to readjust the unloading strategy, including increasing the proportion of key-value memory in bandwidth devices and reducing the proportion of key-value memory in low-bandwidth devices when the system detects that the number of released requests is large enough, and transferring the key-value memory placed in the device with lower bandwidth to the free memory released in the S3, so as to reduce the data transmission time during system reasoning.

[0026] In some embodiments, in the model building module, traversing the reasoning tasks further includes traversing the user's reasoning tasks in a zigzag reasoning order, with each row representing the result of a batch processing through multiple layers of reasoning of the reasoning task model, and each column representing the results of different batch processing calculated under the same model weights.

[0027] The reasoning task model further includes that when the number of completed reasoning tasks finTaskNum exceeds Bs, the last ReduceBatchNum batches will be removed from the reasoning process, where ReduceBatchNum is calculated as follows:

[0028] ReduceBatchNum=finTaskNum / / Bs

[0029] Among them, ReduceBatchNum indicates the number of reduced batches, finTaskNum indicates the number of tasks that have completed reasoning, and Bs indicates the batch task size set by the current system.

[0030] In some implementations, the dynamic memory management module further includes, after releasing the memory occupied by the completed inference task, using the free memory to transfer the tensor portion in the device with lower bandwidth to the free memory.

[0031] In some implementations, the dynamic memory management module further includes that the free memory after release is represented as follows:

[0032]

[0033] The memory occupied by the key-value memory KV Cache of the remaining inference tasks in the three devices is:

[0034]

[0035] in, Indicates the amount of free memory on the GPU, Cg indicates the weight ratio of the inference task model allocated on the GPU, Indicates the amount of free memory on the CPU, Cc indicates the weight ratio of the inference task model allocated to the CPU, Indicates the memory occupied by the KV Cache in the GPU for the remaining inference tasks. Indicates the memory occupied by the KV Cache in the CPU for the remaining inference tasks. The memory occupied by the remaining inference tasks in the KV Cache on Disk. Bn indicates the number of batches set for the current inference task. Cd indicates the inference task model weight allocated on Disk. Batchsize indicates the batch size.

[0036] Compared with the prior art, the beneficial technical effects and technical progress achieved by the present invention are as follows:

[0037] 1) Dynamically batch and reorganize the reasoning tasks, combined with the dynamic memory management design, by fine-grained management of reasoning tasks at the iteration level, the memory occupied by the completed reasoning tasks is released in real time, and the unfinished reasoning tasks are dynamically arranged, thereby reducing the waste of reasoning resources during the reasoning process and improving the throughput of the offline reasoning system based on the offloading technology. Therefore, the present invention can provide dynamic batch reorganization (Batching) and flexible memory management, efficiently utilize the idle memory of the hardware, and improve the throughput of offline reasoning;

[0038] 2) By adding a design consisting of multiple batches on the linear level of the number of iterations to complete the user's reasoning tasks in a zigzag reasoning order, the loading overhead is reduced by overlapping memory access and calculation. During calculation, reasoning is performed according to the batch dimension first to increase the ratio of calculation to loading and reduce the impact of offloading on the reasoning process.

[0039] 3) Use free memory to transfer tensors in devices with lower bandwidth to this part of memory to reduce the I / O overhead of transmission during inference. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the inference acceleration method of the large language model offline inference task under limited resources of the present invention;

[0041] Figure 2 The zigzag reasoning model flow chart adopted by the present invention;

[0042] Figure 3 It is a schematic diagram of the dynamic batch processing process in the reasoning process of the present invention;

[0043] Figure 4 A schematic diagram of KV Cache memory for reallocating unfinished reasoning tasks during the reasoning process of the present invention;

[0044] Figure 5This is a module diagram of the inference acceleration system for offline inference tasks of large language models under limited resources of the present invention. DETAILED DESCRIPTION

[0045] The technical solution of the present invention is described below in conjunction with specific embodiments and drawings.

[0046] Embodiment 1

[0047] like Figure 1 As shown, the inference acceleration method for offline inference tasks of large language models under limited resources of the present invention adopts a zigzag inference sequence, and the specific process includes the following steps:

[0048] Step 1. Formalize the parameters of offline reasoning of the reasoning task model, including: using a two-tuple to represent the number of current reasoning tasks, defined as Bs and Bn, where Bs and Bn represent the batch size (Batchsize) and the number of batches (Batch) set for the current reasoning task respectively; using a six-tuple to represent the current unloading strategy during reasoning under limited resources, which is Wg, Wc, Wd, Cg, Cc, Cd, where Wg, Wc, and Wd represent the percentage of the weight of the reasoning task model on the graphics processing unit (GPU), the central processing unit (CPU), and the hard disk (Disk), respectively, and Cg, Cc, and Cd represent the percentage of the key-value memory (KV Cache) in the Attention mechanism of the large language model (LLM) generated during reasoning on the GPU, CPU, and Disk respectively; start the reasoning service, traverse the user's reasoning tasks in a zigzag reasoning order, set the number of iterations, execute the complete calculation process of the reasoning task model once in each iteration, and calculate a token in each iteration. The system continues this process until the generated token represents the stop "EOS" and the iterative process ends;

[0049] Step 101: Set the current reasoning task model to adopt the zigzag reasoning method. Specifically, Figure 3 As shown in the figure, a dimension consisting of multiple batches including Batch1, Batch2, Batch3, and Batch5 is added to the linear level of the number of iterations, and the reasoning process is expanded from linear to a plane. After the initialization is completed, the reasoning process will be started. The system will reason layer by layer including layer1, layer2, and layer3, and load the next layer while calculating the current layer. The overhead required for loading is reduced by overlapping memory access and calculation; when calculating, reasoning is performed according to the Batch dimension first to increase the ratio of calculation to loading and reduce the impact of unloading on the reasoning process; the user's reasoning tasks are traversed in a zigzag reasoning order;

[0050] Step 102: Allocate the inference task model weight and key-value memory KV Cache to each device including GPU, CPU, and Disk according to the offline inference parameters Bs, Bn, Wg, Wc, Wd, Cg, Cc, and Cd. In subsequent inference, load the tensors not in the GPU into the cache Buffer and transfer them to the GPU for inference.

[0051] Step 2: Execute the inference task. During the execution of the inference task, check whether there are any completed inference tasks and repackage the task batches (Batching);

[0052] Step 201: The system checks the execution process of the reasoning task. When generating a token at the end of each round of iteration, it checks the completion status of the current reasoning task, marks the completed reasoning requests in the reasoning task queue, deletes them from the reasoning task queue, and releases the KV Cache occupied by these completed requests. After all rounds of iteration, the system checks the completion status of all reasoning tasks. When a completed reasoning task is encountered during the execution of the reasoning task, it removes it from the corresponding batch, searches for unfinished reasoning tasks from the back to the front, fills them in the vacant position of the completed reasoning task, reconfigures the reasoning task batching, and exchanges the KV Cache of the two reasoning tasks.

[0053] like Figure 2 As shown in the figure, in the zigzag reasoning process, each row represents the result of a batch after multi-layer reasoning of the reasoning task model, and each column represents the result of different batches calculated under the same model weight. When the completed tasks exceed Bs, the completed tasks are put into the last batch through the second function. At this time, it is no longer necessary to keep them in the reasoning process and they can be removed from the reasoning process, which is reflected in the calculation of deleting the last row in the reasoning process;

[0054] like Figure 3 As shown in the figure, dynamic Batching of reasoning tasks is implemented to achieve fine-grained management. Each time a token is generated, the completed reasoning tasks are marked from the reasoning process, and the unfinished reasoning tasks are selected from the last batch and placed in the batch with completed reasoning tasks for dynamic reorganization of the batch. At the same time, the KV Cache of the corresponding reasoning tasks is moved to the specified memory. When the completed reasoning tasks reach the size of Bs, it means that the movement of the reasoning tasks has made the last batch all completed tasks or unfinished tasks that have been moved to the remaining batches. At this time, there is no need to reason about this batch anymore and it can be removed from the reasoning process. Figure 2The following table shows the process of implementing dynamic batching. Table 1 shows the pseudo code of the dynamic batching algorithm.

[0055]

[0056]

[0057] Step 3: When the completed tasks exceed the defined batch size Bs of the current inference task, the number of batches Bn of the current inference task is reduced by one. At this time, the inference task includes a batch that is completely composed of completed tasks. The memory occupied by this inference task batch is released and removed from the inference process. The remaining inference requests in the inference task queue are reorganized, and the remaining inference KV Cache and the buffer memory address allocated by the intermediate activation are moved.

[0058] Step 301: After releasing the memory occupied by the completed inference task, free memory will appear in the device. At this time, the offload ratio parameter can be adjusted to use the free memory to transfer the tensor part in the device with lower bandwidth to this part of memory, so as to reduce the I / O overhead of transmission during inference;

[0059] Step 4: Re-adjust the unloading strategy, specifically, re-adjust Wg, Wc, Wd, Cg, Cc, and Cd based on step 301, that is, when the system detects that the number of released requests is sufficient, regenerate the unloaded KV Cache placement strategy. The unloaded KV Cache placement strategy includes the parameters such as Cg, Cc, and Cd defined in step 1. The system will adjust these parameters to increase the proportion of KV Cache in devices with higher bandwidth and reduce the proportion of KV Cache in devices with low bandwidth. The KV Cache placed in the device with lower bandwidth is transferred to the free memory released in step 3 to reduce the data transmission time during system reasoning.

[0060] Furthermore, when determining whether the request is completed, step 2 needs to support the model's own request completion determination and the maximum request length given by the user.

[0061] Furthermore, in step 3, when reorganizing the requests, the system will form a new batch of the remaining requests. When the number of requests completed by the system is more than the batch size, the system will release the KV Cache of the requests and remove the processes that execute them from the reasoning.

[0062] Furthermore, when executing step 4, the KV Cache will be moved to a device with higher bandwidth first to reduce the amount of data that needs to be loaded. Since the KV Cache space is allocated according to the request when it is initially allocated, when moving, it is only necessary to move the KV Cache according to the corresponding index while reorganizing the request to form a new batch.

[0063] In order to reduce the cost of switching model weights caused by offloading, the zigzag reasoning method gives priority to using the same model weights and different batches for calculation, so that the cost of loading model weights is amortized to multiple batches. When switching batches, it is necessary to load the KV Cache of each batch and the intermediate variables generated by the calculation, and the memory occupied by this part also determines the maximum size of Bn and Bs that the hardware can accommodate. As shown in Table 2, the pseudo code of the internal execution process of the zigzag reasoning method is shown. First, according to the various parameters related to offloading, the corresponding memory and buffer are allocated on each machine for the model weights, KV Cache and intermediate variables to load the unloaded tensors. When performing reasoning, Tokens will be generated iteratively. Each round of iteration will use the model weights of one layer of the calculation model and all batches for calculation, and obtain the reasoning results. After traversing all layers of the model, a token is generated. In this process, it is necessary to load the model weights of each layer, load the KV Cache and intermediate variables of each batch, and store the KV Cache and intermediate variables of each batch in the allocated memory after the calculation is completed. When all tasks meet the conditions for stopping reasoning, the reasoning process will end. The zigzag reasoning method uses multiple batches and the same model weights for calculation, at the expense of switching smaller KV Caches and intermediate variables, reducing the overhead of switching model weights. If the length of the model reasoning sequence is too long, causing the KV Cache switching time to be longer than the switching weight, the model weights can be switched preferentially by reducing Bn, thus achieving a flexible reasoning method. Note that zigzag reasoning can only achieve higher throughput than the general reasoning process in the reasoning of application offloading technology. When there are sufficient reasoning resources, the throughput of the two reasoning methods is not much different. The internal execution process of zigzag offline reasoning.

[0064]

[0065]

[0066] For the fine-grained reasoning task management of zigzag reasoning, the specific implementation process is as follows:

[0067] During the reasoning process, due to the reasoning characteristics of large language models, each reasoning task only generates <eos>, that is, the reasoning will stop when the token is finished, which results in different number of reasoning iterations required for different reasoning tasks, that is, different reasoning tasks have different reasoning times. When reasoning a batch, the reasoning tasks in the batch that have been reasoned need to wait for the longest reasoning task in the batch to be completed. The tokens generated during the waiting process are invalid tokens and will not be included in the output results, resulting in a waste of reasoning resources.

[0068] For the dynamic allocation of memory during inference using the offloading method, the specific implementation process is as follows:

[0069] In the fine-grained reasoning task management for zigzag reasoning, when the number of completed reasoning tasks finTaskNum exceeds Bs, the last ReduceBatchNum batches will be removed from the reasoning process, where ReduceBatchNum is calculated as follows:

[0070] ReduceBatchNum=finTaskNum / / Bs

[0071] Among them, ReduceBatchNum represents the number of reduced batches, finTaskNum represents the number of tasks that have completed reasoning, and Bs represents the batch task size set by the current system.

[0072] At the same time, the memory occupied by the inference tasks contained in these batches will also be released. This part of the memory is distributed on the GPU, CPU and Disk according to the parameters set during offline inference initialization. This part of the memory includes the KVCache of the inference task and the memory allocated by the inference task to store intermediate variables during inference. After releasing this part of the memory, there will be free memory on the GPU and CPU, so the memory can be reallocated to move tensors stored in lower bandwidth devices to higher bandwidth devices, such as from Disk to CPU, or from CPU to GPU. The dynamic allocation execution process of memory during inference using the offloading method is shown in the pseudo code of Table 3. The free memory after release is:

[0073]

[0074] The memory occupied by the key-value memory KV Cache of the remaining inference tasks in the three devices is:

[0075]

[0076] in, Indicates the amount of free memory on the GPU, Cg indicates the weight ratio of the inference task model allocated on the GPU, Indicates the amount of free memory on the CPU, Cc indicates the weight ratio of the inference task model allocated to the CPU, Indicates the memory occupied by the KV Cache in the GPU for the remaining inference tasks. Indicates the memory occupied by the KV Cache in the CPU for the remaining inference tasks. The memory occupied by the remaining inference tasks in the KV Cache on the Disk. Bn indicates the number of batches set for the current inference task. Cd indicates the inference task model weight allocated on the Disk. Batchsize indicates the batch size.

[0077] When initializing the allocation of KV Cache memory, the system allocates the KV Cache of each inference task in the Batch to different devices. Therefore, when the free memory can accommodate one request from each of the remaining Bn-ReduceBatchNum batches, the memory can be reallocated. However, if only a small number of inference tasks are moved each time, the memory of the inference tasks will be frequently reallocated, which will have a lot of additional overhead. Therefore, the system designs the hyperparameter minTask to specify the minimum number of inference tasks to be moved each time. When reallocating memory, in order to ensure the order of KV Cache, it is necessary to first fill the free memory of the GPU with tensors in the CPU, and then move the tensors in the Disk to the free memory of the CPU. By moving tensors from devices with lower bandwidth to devices with higher bandwidth, the I / O overhead loaded during each round of iteration is reduced, the latency of each round of iteration is reduced, and the overall throughput of the system is improved.

[0078]

[0079]

[0080] Embodiment 2

[0081] like Figure 5 As shown, the inference acceleration system for offline inference tasks of large language models under limited resources of the present invention includes a model construction module 100, a dynamic batch processing reorganization module 200, a dynamic memory management module 300 and an unloading strategy adjustment module 400. The system is used to execute the overall process of the inference acceleration method for offline inference tasks of large language models under limited resources in Example 1, wherein:

[0082] A model building module 100 is used to define an inference task model for offline inference, including the number of current inference tasks, batch processing of current inference tasks, and an unloading strategy for current inference under limited resources, wherein the unloading strategy further includes a weight of the inference task model in a graphics processor, a key-value memory in an attention mechanism in a large language model generated during inference, starts an inference service, traverses the inference tasks, sets the number of iterations, and iteratively calculates the inference task model;

[0083] The dynamic batch reorganization module 200 is used to execute the inference task after receiving the inference request, and perform dynamic batch reorganization of the inference task, including marking the completed inference task from the inference process each time a token is generated, and selecting the unfinished inference task from the last batch and placing it into the batch with the completed inference task for dynamic batch reorganization, and at the same time moving the key value memory of the corresponding inference task to the specified memory;

[0084] The dynamic memory management module 300 performs dynamic memory management when there are remaining inference requests in the inference task queue, including releasing free memory when the completed tasks exceed the defined batch size of the current inference task, deleting the completed inference task from the inference process, and moving the remaining inference key value memory and the buffer memory address allocated by the intermediate activation;

[0085] The unloading policy adjustment module 400 is used to readjust the unloading policy, including increasing the proportion of key-value memory in bandwidth devices and reducing the proportion of key-value memory in low-bandwidth devices when the system detects that the number of released requests is large enough, and transferring the key-value memory placed in the device with lower bandwidth to the free memory released in the S3, so as to reduce the data transmission time during system reasoning.

[0086] In summary, the present invention is based on the FlexGen large language model offline reasoning system, and designs dynamic batching (Batching) and dynamic memory management of reasoning tasks. By managing reasoning tasks at the iteration level in a fine-grained manner, the memory occupied by completed reasoning tasks is released in real time, and unfinished reasoning tasks are dynamically arranged, thereby reducing the waste of reasoning resources during the reasoning process and improving the throughput of the offline reasoning system based on offloading technology.

[0087] The above is only a specific implementation method of the present invention. The above implementation steps are only used to help understand the specific methods and core ideas of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of several changes or equivalent substitutions, combinations and modifications made within the technical scope disclosed by the present invention and based on the ideas of the present invention without departing from the principles of the present invention. These changes or equivalent substitutions, combinations and modifications should be deemed to fall within the scope of protection of the present invention and should be covered by the scope of protection of the present invention.< / eos>

Claims

1. A method for accelerating reasoning of offline reasoning tasks of large language models under limited resources, characterized in that: include: S1, the definition of the inference task model for offline inference, including the number of current inference tasks, the batch processing of current inference tasks, and the unloading strategy for current inference under limited resources, wherein the unloading strategy further includes the weight of the inference task model in the graphics processor, the key-value memory in the attention mechanism in the large language model generated during inference, the inference service is started, the inference tasks are traversed, the number of iterations is set, and the calculation of the inference task model is iteratively performed; S2, when receiving the inference request, execute the inference task and perform dynamic batch reorganization of the inference task, including marking the completed inference task from the inference process every time a token is generated, and selecting the unfinished inference task from the last batch and placing it into the batch with the completed inference task for dynamic batch reorganization, and moving the key value memory of the corresponding inference task to the specified memory; S3, when there are remaining inference requests in the inference task queue, dynamic memory management is performed, including releasing free memory when the completed tasks exceed the defined batch size of the current inference task, deleting the completed inference task from the inference process, and moving the remaining inference key value memory and the buffer memory address allocated by the intermediate activation; S4, readjusting the unloading strategy, including increasing the proportion of key-value memory in bandwidth devices and reducing the proportion of key-value memory in low-bandwidth devices when the system detects that the number of released requests is large enough, and transferring the key-value memory placed in the device with lower bandwidth to the free memory released in S3, so as to reduce the data transmission time during system reasoning.

2. The method for accelerating the reasoning of large language model offline reasoning tasks under limited resources according to claim 1, characterized in that: The traversal reasoning task of S1 further includes traversing the user's reasoning tasks in a zigzag reasoning order, with each row representing the result of a batch of multi-layer reasoning through the reasoning task model, and each column representing the results of different batches calculated under the same model weight.

3. The method for accelerating the reasoning of large language model offline reasoning tasks under limited resources according to claim 1, characterized in that: The S1 further includes that when the number of completed reasoning tasks finTaskNum exceeds Bs, the last ReduceBatchNum batches will be removed from the reasoning process, where ReduceBatchNum is calculated as follows: ReduceBatchNum=finTaskNum / / Bs Among them, ReduceBatchNum represents the number of reduced batches, finTaskNum represents the number of tasks that have completed reasoning, and Bs represents the batch task size set by the current system.

4. The method for accelerating the reasoning of large language model offline reasoning tasks under limited resources according to claim 1, characterized in that: The S3 further includes, after releasing the memory occupied by the completed reasoning task, using the free memory to transfer the tensor part in the device with low bandwidth to the free memory.

5. The method for accelerating the reasoning of large language model offline reasoning tasks under limited resources according to claim 1, characterized in that: In the S3, the free memory after release is represented as follows: The memory occupied by the key-value memory KV Cache of the remaining inference tasks in the three devices is: in, Indicates the amount of free memory on the GPU, Cg indicates the weight ratio of the inference task model allocated on the GPU, Indicates the amount of free memory on the CPU, Cc indicates the weight ratio of the inference task model allocated to the CPU, Indicates the memory occupied by the KV Cache in the GPU for the remaining inference tasks. Indicates the memory occupied by the KV Cache in the CPU for the remaining inference tasks. The memory occupied by the remaining inference tasks in the KV Cache on Disk. Bn indicates the number of batches set for the current inference task. Cd indicates the inference task model weight allocated on Disk. Batchsize indicates the batch size.

6. A large language model offline reasoning task reasoning acceleration system under limited resources, characterized in that: The system includes a model building module, a dynamic batch reorganization module, a dynamic memory management module and an unloading strategy adjustment module, wherein: A model building module, which is used to define the inference task model for offline inference, including the number of current inference tasks, the batch processing of current inference tasks, and the unloading strategy for current inference under limited resources, wherein the unloading strategy further includes the weight of the inference task model in the graphics processor, the key-value memory in the attention mechanism in the large language model generated during inference, start the inference service, traverse the inference tasks, set the number of iterations, and iteratively calculate the inference task model; The dynamic batch reorganization module is used to execute the inference task after receiving the inference request, and perform dynamic batch reorganization of the inference task, including marking the completed inference task from the inference process every time a token is generated, and selecting the unfinished inference task from the last batch and placing it into the batch with the completed inference task for dynamic batch reorganization, and moving the key value memory of the corresponding inference task to the specified memory; A dynamic memory management module is used to perform dynamic memory management when there are remaining inference requests in the inference task queue, including releasing free memory when the completed tasks exceed the defined batch size of the current inference task, deleting the completed inference task from the inference process, and moving the remaining inference key value memory and the buffer memory address allocated by the intermediate activation; The unloading strategy adjustment module is used to readjust the unloading strategy, including increasing the proportion of key-value memory in bandwidth devices and reducing the proportion of key-value memory in low-bandwidth devices when the system detects that the number of released requests is large enough, and transferring the key-value memory placed in the device with lower bandwidth to the free memory released in the S3, so as to reduce the data transmission time during system reasoning.

7. The inference acceleration system for offline inference tasks of large language models under limited resources according to claim 6, characterized in that: In the model building module, traversing the reasoning tasks further includes traversing the user's reasoning tasks in a zigzag reasoning order, with each row representing the result of a batch processing through multiple layers of reasoning of the reasoning task model, and each column representing the results of different batch processing calculated under the same model weight.

8. The inference acceleration system for offline inference tasks of large language models under limited resources according to claim 6, characterized in that: The reasoning task model further includes that when the number of completed reasoning tasks finTaskNum exceeds Bs, the last ReduceBatchNum batches will be removed from the reasoning process, where ReduceBatchNum is calculated as follows: ReduceBatchNum=finTaskNum / / Bs Among them, ReduceBatchNum represents the number of reduced batches, finTaskNum represents the number of tasks that have completed reasoning, and Bs represents the batch task size set by the current system.

9. The inference acceleration system for offline inference tasks of large language models under limited resources according to claim 6, characterized in that: The dynamic memory management module further includes, after releasing the memory occupied by the completed inference task, using the free memory to transfer the tensor part in the device with lower bandwidth to the free memory.

10. The inference acceleration system for offline inference tasks of large language models under limited resources according to claim 6, characterized in that: The dynamic memory management module further includes that the free memory after release is represented as follows: The memory occupied by the key-value memory of the remaining inference tasks in the three devices is: in, Indicates the amount of free memory on the GPU, Cg indicates the weight ratio of the inference task model allocated on the GPU, Indicates the amount of free memory on the CPU, Cc indicates the weight ratio of the inference task model allocated to the CPU, Indicates the memory occupied by the key-value memory in the GPU for the remaining inference tasks. Indicates the memory occupied by the key-value memory of the remaining inference tasks in the CPU. The memory occupied by the key-value memory of the remaining inference tasks in Disk. Bn indicates the number of batches set for the current inference task. Cd indicates the inference task model weight allocated on Disk. Batchsize indicates the batch size.

Images