High-throughput large model inference method and device based on time separation type pipeline architecture, equipment and storage medium

Abstract

The application discloses a high-throughput large model inference method and device based on a time separation type pipeline architecture, equipment and a storage medium, relates to the technical field of large model inference, and the high-throughput large model inference method based on the time separation type pipeline architecture comprises the following steps: when the current inference stage is a pre-filling stage, pre-filling is performed according to a client request, and the key value cache capacity of each request decision point is determined; the stage switching time is determined according to the key value cache capacity of each request decision point and a preset memory capacity; the current inference stage is switched from the pre-filling stage to a decoding stage according to the stage switching time, and the client request is processed according to a preset load balancing strategy to obtain a target load balancing result; the inference of the large model is performed according to the target load balancing result and the pipeline architecture, and the output text corresponding to the client request is obtained according to the inference result. The efficiency of the high-throughput large model inference is improved.

Description

Technical Field

[0001] The present application relates to the field of large model reasoning technology, and in particular to a high-throughput large model reasoning method, apparatus, device and storage medium based on a time-separated pipeline architecture. Background Art

[0002] With the rapid development of artificial intelligence (AI), key areas such as natural language processing and image and video generation are increasingly demanding cost reduction and efficiency improvements for large-model inference. Driven in particular by applications like Sora for video generation and Copilot for code generation, efficiently handling increasingly complex inference tasks has become a major challenge for large-model inference. These challenges are particularly pronounced in large models that rely on autoregressive mechanisms. Inference not only increases computational complexity, but also places higher demands on memory usage and inter-device communication efficiency. However, low-end accelerator cards such as the NVIDIA L20 and A10, as well as domestic accelerator cards like Tianshu and Muxi, struggle to meet the demands of large-model inference due to their limited video memory and weak interconnect bandwidth. Furthermore, the low efficiency of traditional tensor parallelism and the high cost of high-end equipment have led to a rapid increase in the overall cost of large-model inference. Therefore, addressing the inefficiency of existing large-model inference methods has become a pressing issue. Summary of the Invention

[0003] The main purpose of this application is to provide a high-throughput large-model reasoning method, device, equipment and storage medium based on a time-separated pipeline architecture, aiming to solve the technical problem of low efficiency of existing large-model reasoning methods.

[0004] To achieve the above objectives, the present application proposes a high-throughput large-model inference method based on a time-separated pipeline architecture, the high-throughput large-model inference method based on the time-separated pipeline architecture comprising:

[0005] When the current inference phase is the pre-fill phase, pre-fill is performed based on client requests to determine the key-value cache capacity for each request decision point;

[0006] Determine the phase switching timing based on the key-value cache capacity and preset memory capacity of each request decision point;

[0007] Switching the current inference phase from the pre-filling phase to the decoding phase according to the phase switching timing, and processing the client request according to a preset load balancing strategy to obtain a target load balancing result;

[0008] The large model is inferred based on the target load balancing result and the pipeline architecture, and the output text corresponding to the client request is obtained based on the inference result.

[0009] In one embodiment, when the current inference phase is a pre-filling phase, the step of performing pre-filling according to a client request and determining the key-value cache capacity of each request decision point includes:

[0010] When the current inference phase is the pre-filling phase, the corresponding output length range is determined according to the preset greedy pre-filling strategy and the client request;

[0011] determining a plurality of request decision points according to the output length range;

[0012] Pre-filling is performed according to the client request, and when a request decision point is detected, the key value cache capacity of each request decision point is determined.

[0013] In one embodiment, the step of determining the phase switching timing based on the key-value cache capacity and the preset memory capacity of each request decision point includes:

[0014] Compare the key value cache capacity of each request decision point with the preset memory capacity to obtain a capacity comparison result;

[0015] When the capacity comparison result shows that the key-value cache capacity is greater than or equal to the preset memory capacity, a phase switching timing is determined according to a request decision point corresponding to the key-value cache capacity.

[0016] In one embodiment, the step of switching the current inference stage from the pre-filling stage to the decoding stage according to the stage switching timing, and processing the client request according to a preset load balancing strategy to obtain a target load balancing result includes:

[0017] Switching the current inference phase from the pre-filling phase to the decoding phase according to the phase switching timing, and dividing the client requests into batches to obtain a plurality of batch requests;

[0018] Upon completion of decoding the plurality of batch requests, obtaining a current batch size of each batch request;

[0019] Load balancing is performed based on the preset load balancing strategy and the current batch size of each batch request to obtain the target load balancing result.

[0020] In one embodiment, the step of processing according to a preset load balancing algorithm and a current batch size of each batch request to obtain a target load balancing result includes:

[0021] Determine the corresponding historical batch size based on the preset load balancing algorithm and the current batch size of each batch request;

[0022] Comparing the current batch size of each batch request with the historical batch size to obtain a batch size comparison result;

[0023] A corresponding target load balancing result is determined according to the batch size comparison result.

[0024] In one embodiment, the step of determining a corresponding target load balancing result based on the batch size comparison result includes:

[0025] When the batch size comparison result is that the current batch size is smaller than the historical batch size, determining the target load balancing result as a request addition result;

[0026] When the batch size comparison result shows that the current batch size is equal to the historical batch size, determining that the target load balancing result is a normal load result;

[0027] When the batch size comparison result is that the current batch size is larger than the historical batch size, the target load balancing result is determined to be a request removal result.

[0028] In one embodiment, after the step of performing inference on the large model based on the target load balancing result and the pipeline architecture, and obtaining the output text corresponding to the client request based on the inference result, the method further includes:

[0029] Analyze the execution time of each client request in the model inference results to determine the request baseline time;

[0030] Compare real-time request data and hyperparameter information to determine pipeline idle time;

[0031] Obtaining a target switching timing according to the request baseline time and the pipeline idle time;

[0032] The current inference stage is switched from the decoding stage to the pre-filling stage according to the target switching timing.

[0033] In addition, to achieve the above objectives, the present application also proposes a high-throughput large-model inference device based on a time-separated pipeline architecture, wherein the high-throughput large-model inference device based on the time-separated pipeline architecture comprises:

[0034] The scheduling module is used to pre-fill according to the client request when the current inference phase is the pre-fill phase, and determine the key-value cache capacity of each request decision point;

[0035] A calculation module, used to determine the phase switching timing based on the key value cache capacity and preset memory capacity of each request decision point;

[0036] a load balancing module, configured to switch the current inference phase from the pre-filling phase to the decoding phase according to the phase switching timing, and process the client request according to a preset load balancing strategy to obtain a target load balancing result;

[0037] The inference module is used to perform inference on the large model according to the target load balancing result and the pipeline architecture, and obtain the output text corresponding to the client request according to the inference result.

[0038] In addition, to achieve the above-mentioned objectives, the present application also proposes a high-throughput large-model inference device based on a time-separated pipeline architecture, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the high-throughput large-model inference method based on a time-separated pipeline architecture as described above.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the high-throughput large model inference method based on the time-separated pipeline architecture as described above are implemented.

[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the high-throughput large model inference method based on the time-separated pipeline architecture as described above.

[0041] This application determines the key value cache capacity of each request decision point by performing pre-filling according to the client request when the current inference stage is the pre-filling stage; determines the stage switching timing based on the key value cache capacity and preset memory capacity of each request decision point; switches the current inference stage from the pre-filling stage to the decoding stage according to the stage switching timing, and processes the client request according to a preset load balancing strategy to obtain a target load balancing result; performs inference of a large model based on the target load balancing result and a pipeline architecture, and obtains the output text corresponding to the client request based on the inference result. This improves the efficiency of high-throughput large model inference. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 A flowchart illustrating a first embodiment of a high-throughput large-model inference method based on a time-separated pipeline architecture of this application is provided;

[0045] Figure 2 An overall schematic diagram of the runtime system provided in Example 1 of the high-throughput large-model inference method based on a time-separated pipeline architecture of this application;

[0046] Figure 3 A schematic diagram of the intensity accumulation method provided in Example 1 of the high-throughput large-model inference method based on a time-separated pipeline architecture of this application;

[0047] Figure 4 A flowchart illustrating a second embodiment of a high-throughput large-model inference method based on a time-separated pipeline architecture of this application is provided;

[0048] Figure 5 A load balancing diagram provided for the second embodiment of the high-throughput large-model inference method based on a time-separated pipeline architecture of this application;

[0049] Figure 6 This is a schematic diagram of the module structure of a high-throughput large-model inference device based on a time-separated pipeline architecture according to an embodiment of the present application;

[0050] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the high-throughput large-model inference method based on the time-separated pipeline architecture in the embodiment of the present application.

[0051] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] The main solution of the embodiment of the present application is: when the current inference stage is the pre-filling stage, pre-filling is performed according to the client request, and the key value cache capacity of each request decision point is determined; the stage switching timing is determined according to the key value cache capacity of each request decision point and the preset memory capacity; according to the stage switching timing, the current inference stage is switched from the pre-filling stage to the decoding stage, and the client request is processed according to the preset load balancing strategy to obtain the target load balancing result; the large model is inferred according to the target load balancing result and the pipeline architecture, and the output text corresponding to the client request is obtained according to the inference result.

[0055] With the rapid development of artificial intelligence (AI), key areas such as natural language processing and image and video generation are increasingly demanding cost reduction and efficiency improvements for large-model inference. Driven in particular by applications like Sora for video generation and Copilot for code generation, efficiently handling increasingly complex inference tasks has become a major challenge for large-model inference. These challenges are particularly pronounced in large models that rely on autoregressive mechanisms. Inference not only increases computational complexity, but also places higher demands on memory usage and inter-device communication efficiency. However, low-end accelerator cards such as the NVIDIA L20 and A10, as well as domestic accelerator cards like Tianshu and Muxi, struggle to meet the demands of large-model inference due to their limited video memory and weak interconnect bandwidth. Furthermore, the low efficiency of traditional tensor parallelism and the high cost of high-end equipment have led to a rapid increase in the overall cost of large-model inference. Therefore, addressing the inefficiency of existing large-model inference methods has become a pressing issue.

[0056] This application determines the key value cache capacity of each request decision point by performing pre-filling according to the client request when the current inference stage is the pre-filling stage; determines the stage switching timing based on the key value cache capacity and preset memory capacity of each request decision point; switches the current inference stage from the pre-filling stage to the decoding stage according to the stage switching timing, and processes the client request according to a preset load balancing strategy to obtain a target load balancing result; performs inference of a large model based on the target load balancing result and a pipeline architecture, and obtains the output text corresponding to the client request based on the inference result. This improves the efficiency of high-throughput large model inference.

[0057] It should be noted that the execution entity of this embodiment can be a computing service device with data processing, network communication, and program execution capabilities, such as a tablet computer, personal computer, or mobile phone, or a high-throughput, large-model inference device based on a time-separated pipeline architecture that can implement the above functions. This embodiment and the following embodiments will be described below using a high-throughput, large-model inference device based on a time-separated pipeline architecture as the execution entity.

[0058] Based on this, the embodiment of the present application provides a high-throughput large model reasoning method based on a time-separated pipeline architecture, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the high-throughput large model inference method based on the time-separated pipeline architecture of this application.

[0059] In this embodiment, the high-throughput large model inference method based on the time-separated pipeline architecture includes steps S10 to S40:

[0060] Step S10, when the current inference phase is a pre-filling phase, pre-filling is performed according to the client request to determine the key value cache capacity of each request decision point;

[0061] It should be noted that this embodiment proposes a time dimension separation architecture for optimizing the computational efficiency during the inference process. Similar to the spatial dimension separation technology (such as the MoonCake system, and the MoonCake system is a technical architecture specifically used to optimize the inference process of large-scale machine learning models), the present invention separates and processes the two types of computing tasks, Prefill and Decode, according to their different characteristics. The inference process is divided into two stages: the Prefill stage and the Decode stage. The Prefill stage is mainly responsible for gradually filling the request into the video memory to ensure that the Decode stage can continue to process enough batch data, thereby improving the overall computing efficiency. Through temporal separation, the two different computing modes can avoid mutual interference, effectively reducing the pipeline bubbles caused by the Prefill-Decode switching, and significantly optimizing the inference performance.

[0062] It should be noted that the overall design of this embodiment is as follows Figure 2As shown, the system consists of a centralized engine and a distributed runtime. The centralized engine is responsible for scheduling request batches using a custom scheduling algorithm combined with continuous batch processing technology, and employs PagedAttention technology to achieve refined management of the KV cache (key-value cache). During the scheduling phase, the centralized engine manages and controls the distributed runtime using RPC (Remote Procedure Call) technology. The distributed runtime uses stateful workers to execute tasks assigned by the engine and maintain the communication context of the tasks. Asynchronous point-to-point communication technology is used between workers to achieve efficient data transmission, ensuring fast and seamless data exchange between each stage of the pipeline. Through these optimized designs, the present invention can effectively improve the system's resource utilization and overall performance, making it particularly suitable for efficient inference scenarios for large-scale models. After receiving a request, the centralized engine places the request in the scheduler, which uses continuous batch processing to send the request in batches to the distributed pipeline runtime for execution. Therefore, designing a reasonable scheduling algorithm to achieve efficient pipeline operation is crucial.

[0063] It can be understood that the pre-filling phase refers to the model processing the input data, initializing and filling the KV cache, and preparing for subsequent decoding. The client request refers to the large model inference request initiated by the client. The request decision point refers to certain pre-specified time points or steps in the decoding process, which are used to evaluate and manage memory usage. By checking the usage of the KV cache (Key-Value Cache) at these decision points, you can decide whether to switch from the pre-filling phase to the decoding phase. The key-value cache capacity refers to the KV cache usage at the decision point.

[0064] In the specific implementation, when the inference stage of the large model is the pre-filling stage, the large model inference request initiated by the client is pre-filled, and then the KV cache usage of these decision points is updated after each pre-filling is completed, that is, the key value cache capacity of each decision point is determined.

[0065] In a feasible implementation, step S10 may include steps A11 to A13:

[0066] Step A11, when the current inference phase is the pre-filling phase, determining the corresponding output length range according to the preset greedy pre-filling strategy and the client request;

[0067] It should be noted that the preset greedy pre-filling strategy refers to a preset greedy pre-filling algorithm based on an AI method, and the output length range refers to the predicted output length range of the client request.

[0068] In the specific implementation, this embodiment is based on the AI-based greedy pre-filling method, that is, an additional header (a separate linear layer) is directly added to the last hidden state of the LLM to predict the output length range of the request, and then the large model inference request initiated by the client is predicted through the greedy pre-filling algorithm based on the AI ​​method, and finally the output length range of the predicted client request, that is, the output length range, is obtained.

[0069] Step A12, determining a plurality of request decision points according to the output length range;

[0070] It is understood that the request decision point refers to the decision point position in the client request.

[0071] In a specific implementation, based on the predicted output length range of the client request, certain future decoding steps are designated as decision points, and the positions of the decision points in the client request are determined. For example, based on the predicted output length range of the client request, a decision point is set every 20 steps, thereby obtaining multiple request decision points.

[0072] Step A13: pre-filling is performed according to the client request, and when a request decision point is detected, the key value cache capacity of each request decision point is determined.

[0073] It can be understood that when the reasoning stage of the large model is the pre-filling stage, the large model reasoning request initiated by the client is pre-filled, and then the KV cache usage of these decision points is updated after each pre-filling is completed, that is, the key value cache capacity of each decision point is determined.

[0074] Step S20, determining the phase switching timing according to the key value cache capacity and the preset memory capacity of each request decision point;

[0075] It is understandable that the preset memory capacity refers to a critical value of the memory capacity used to determine whether to perform phase switching, and the phase switching timing refers to the switching timing from the pre-filling phase to the decoding phase.

[0076] In a specific implementation, the key-value cache capacity of each request decision point is compared with the critical value of the memory capacity used to determine whether to switch the stage, so as to screen out the request decision points corresponding to the key-value cache capacity being greater than or equal to the critical value of the memory capacity used to determine whether to switch the stage, and finally determine the switching timing from the pre-filling stage to the decoding stage, that is, the stage switching timing.

[0077] In a feasible implementation, step S20 may include steps A21 to A22:

[0078] Step A21, comparing the key value cache capacity of each request decision point with the preset memory capacity to obtain a capacity comparison result;

[0079] It is understandable that the capacity comparison result refers to the size comparison result between the key-value cache capacity and the preset memory capacity.

[0080] In a specific implementation, the key value cache capacity of each request decision point is compared with the memory capacity critical value used to determine whether to switch stages, and the capacity comparison result is that the key value cache capacity of each request decision point is greater than or equal to the memory capacity critical value used to determine whether to switch stages, or the key value cache capacity of each request decision point is less than the memory capacity critical value used to determine whether to switch stages.

[0081] Step A22: When the capacity comparison result shows that the key-value cache capacity is greater than or equal to the preset memory capacity, a phase switching timing is determined according to a request decision point corresponding to the key-value cache capacity.

[0082] In a specific implementation, when the capacity comparison result is that the key-value cache capacity is greater than or equal to the memory capacity critical value used to determine whether to switch stages, it indicates that the KV cache of the decision point exceeds the memory capacity, and then the switching timing is switched from the pre-filling stage to the decoding stage according to the decision point.

[0083] It should be noted that during the prefill phase, TD-Pipe needs to determine the optimal time to switch to the decoding phase. To reduce frequent switching between prefill and decoding, TD-Pipe typically prefers to perform more prefill operations before switching. However, since memory usage in the decoding phase continues to increase, excessive prefilling can occupy a significant portion of memory, leading to insufficient memory and frequent recomputation or data offloading during the decoding phase, significantly degrading system performance. Since the output length is unknown until the request is completed, peak memory usage cannot be predicted in advance. To ensure sufficient prefill operations in the decoding phase without excessively exceeding memory capacity, this embodiment proposes an AI-based greedy prefilling method. We directly add an additional head (a separate linear layer) to the last hidden state of the LLM (Large Language Model) to predict the requested output length range. To determine the optimal time to switch from prefill to decoding, we designed a dynamic programming algorithm to simulate the memory usage of the decoding phase after the new prefill is completed. By designating certain future decoding steps as decision points, we can update the KV cache usage of these decision points after each prefill. We then determine the switching time by checking whether the memory usage of any decision point exceeds the memory capacity. This approach can help the system better control memory usage without introducing significant computational overhead, reducing recalculation or data offloading operations caused by video memory overflow.

[0084] Step S30: switching the current inference phase from the pre-filling phase to the decoding phase according to the phase switching timing, and processing the client request according to a preset load balancing strategy to obtain a target load balancing result;

[0085] It is understandable that the preset load balancing strategy refers to a pre-set strategy for balancing the load between batches in real time during the decoding process, and the target load balancing results include request addition results, normal load results, and request removal results.

[0086] In a specific implementation, the current inference stage is switched from the pre-filling stage to the decoding stage according to the switching timing from the pre-filling stage to the decoding stage, and then the client request is decoded and processed, and the load between batches is balanced in real time during the decoding process to obtain the target load balancing result.

[0087] It should be noted that in the initial stages of the decoding phase, TD-Pipe (Time-Dimension Pipeline) divides requests into batches equal to the number of GPUs, with each batch containing the same number of requests. As decoding progresses, requests in different batches complete at different times, causing the workload of each batch to constantly change, resulting in load imbalance. To address this issue, TD-Pipe adopts a cross-batch work stealing method, balancing the load between batches in real time during the decoding process through a sliding window mechanism. After each batch completes the decoding step, the method calculates the average of the current batch size and the historical sizes of other batches, and decides whether to add or remove requests from the batch based on this average. Through this iterative process, bubbles in the pipeline can be effectively reduced, load balancing between decoding batches can be achieved, and thus system performance can be improved.

[0088] Step S40: performing inference on the large model according to the target load balancing result and the pipeline architecture, and obtaining the output text corresponding to the client request according to the inference result.

[0089] It's understood that pipeline architecture refers to pipeline parallelism, a layer-by-layer parallelism method where each GPU (Graphics Processing Unit) handles a different part of the model. During inference, the same input flows sequentially through multiple devices. Output text refers to the final results generated by the large model during inference. These results are presented in text format and correspond to client requests. Output text can take various forms, depending on the type of client request and the application scenario.

[0090] In the specific implementation, different batches of requests are load balanced according to the target load balancing result, and iterative decoding is performed, that is, the large model is inferred, and then the high-throughput large model inference result is obtained. Finally, the high-throughput large model inference result is parsed to obtain the final result generated by the large model corresponding to the client request during the inference process, that is, the output text.

[0091] In a feasible implementation, step S40 may include steps A41 to A44:

[0092] Step A41: Analyze the execution time of each client request in the model inference result to determine the request baseline time;

[0093] It is understandable that the execution time refers to the time it takes for each client request to be decoded, and the request baseline time refers to the baseline time of each request under high computing intensity.

[0094] In specific implementations, the start time of each request is recorded at the beginning of processing, and the end time is recorded after each request is completed. The execution time of each request is calculated by subtracting the start time from the end time. Execution time data for all requests is collected and analyzed using statistical methods (such as mean, median, and standard deviation). A baseline request time is determined, which is the average execution time of requests under normal conditions. Finally, a baseline time for each request under high computational intensity is obtained.

[0095] Step A42: comparing the real-time request data and the hyperparameter information to determine the pipeline idle time;

[0096] It can be understood that real-time request data refers to the client requests and related information received by the system during operation, hyperparameter information refers to the various parameters set during the system design and configuration process, which affect the performance and behavior of the system, and pipeline idle time refers to the time wasted due to pipeline bubbles.

[0097] In the specific implementation, real-time request data, including request ID and processing time, is collected and compared with hyperparameter information such as batch size, number of pre-fill steps, and decision point interval. The average processing time, maximum processing time, and minimum processing time are then calculated. Finally, the time waste caused by pipeline bubbles, i.e., pipeline idle time, is calculated.

[0098] Step A43, obtaining a target switching timing according to the requested baseline time and the pipeline idle time;

[0099] It can be understood that the target switching timing refers to the switching timing from the decoding phase to the pre-filling phase.

[0100] In a specific implementation, the baseline time of each request under high computing intensity is compared with the time wasted due to pipeline bubbles. When the baseline time of each request under high computing intensity is greater than the time wasted due to pipeline bubbles, it indicates that the performance degradation exceeds the impact of pipeline idle time. The system should switch from decoding to pre-filling. The switching timing from the decoding stage to the pre-filling stage is determined, that is, the target switching timing.

[0101] Step A44: Switch the current inference stage from the decoding stage to the pre-filling stage according to the target switching timing.

[0102] In a specific implementation, the current inference stage is switched from the decoding stage to the pre-filling stage according to the switching timing from the decoding stage to the pre-filling stage. By switching the decoding stage to the pre-filling stage at the optimal switching timing, the problem of computing intensity decreasing too quickly and the decoding efficiency also decreasing is avoided.

[0103] It should be noted that in the pipeline architecture of TD-Pipe, the computational intensity of the decoding stage will decrease as the batch is completed. If the computational intensity decreases too quickly, the decoding efficiency will also decrease. However, if you frequently switch between the decoding and pre-filling stages, it may increase the bubbles in the pipeline and affect the overall performance. To solve this problem, the intensity accumulation method decides whether to switch stages by comparing the performance loss caused by the decrease in computational intensity with the pipeline bubbles generated in the switching stage. This method analyzes the execution time, calculates the baseline time of each request under high computational intensity, and estimates the time wasted due to pipeline bubbles by comparing real-time data and hyperparameter information, so as to find the best time to switch. Figure 3 As shown, the intensity accumulation method compares the performance degradation caused by the decrease in computation intensity with the pipeline idle time caused by stage switching. If the performance degradation exceeds the impact of pipeline idle time, the system should switch from decoding to prefilling.

[0104] This embodiment performs pre-filling based on client requests when the current inference stage is the pre-filling stage, determines the key-value cache capacity of each request decision point, determines the stage switching timing based on the key-value cache capacity and preset memory capacity of each request decision point, switches the current inference stage from the pre-filling stage to the decoding stage based on the stage switching timing, processes the client request based on a preset load balancing strategy, and obtains a target load balancing result; performs inference on a large model based on the target load balancing result and a pipeline architecture, and obtains the output text corresponding to the client request based on the inference result. This improves the efficiency of high-throughput large-model inference.

[0105] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 4 In the high-throughput large-model inference method based on the time-separated pipeline architecture, step S30 further includes steps S31 to S33:

[0106] Step S31, switching the current inference stage from the pre-filling stage to the decoding stage according to the stage switching timing, and dividing the client request into batches to obtain multiple batch requests;

[0107] It can be understood that batch request refers to combining multiple independent client requests and processing them as a whole.

[0108] In a specific implementation, the current inference stage is switched from the pre-filling stage to the decoding stage according to the switching timing from the pre-filling stage, and the large model inference requests initiated by the client are batched, that is, multiple independent client requests are combined together and processed as a whole to obtain multiple batch requests. In this embodiment, the requests are divided into batches equal to the number of GPUs, and each batch contains the same number of requests.

[0109] Step S32, upon completion of decoding the plurality of batch requests, obtaining a current batch size of each batch request;

[0110] It is understood that the current batch size refers to the number of requests included in one batch.

[0111] In the specific implementation, this embodiment uses a sliding window mechanism to balance the load between batches in real time during the decoding process. After each batch completes the decoding step, the method calculates the number of requests included in the batch, that is, the current batch size.

[0112] Step S33 , performing load balancing processing according to a preset load balancing strategy and the current batch size of each batch request to obtain a target load balancing result.

[0113] It can be understood that the current inference stage is switched from the pre-filling stage to the decoding stage according to the switching timing from the pre-filling stage to the decoding stage, and the number of requests in each batch of requests is load balanced according to the load balancing algorithm between decoding batches to obtain the target load balancing result.

[0114] In a feasible implementation, step S33 may include steps A331 to A333:

[0115] Step A331, determining the corresponding historical batch size according to a preset load balancing algorithm and the current batch size of each batch request;

[0116] It is understood that the historical batch size refers to the average number of requests in batches before the current batch.

[0117] In a specific implementation, the total number of requests in each batch before the current batch is obtained based on the load balancing algorithm between decoding batches, and then the average value of the total number of requests in each batch before the current batch is calculated to obtain the historical batch size.

[0118] Step A332: Compare the current batch size of each batch request with the historical batch size to obtain a batch size comparison result;

[0119] It can be understood that the batch size comparison result refers to the comparison result of the current batch size with the historical batch size.

[0120] In a specific implementation, the number of requests in the current batch is compared with the average number of requests in historical batches, and the batch size comparison result is obtained as follows: the number of requests in the current batch is greater than the average number of requests in historical batches, or the number of requests in the current batch is equal to the average number of requests in historical batches, or the number of requests in the current batch is less than the average number of requests in historical batches.

[0121] Step A333: Determine the corresponding target load balancing result according to the batch size comparison result.

[0122] In a specific implementation, the current batch size is compared with the average of the historical sizes of other batches, and a decision is made based on the average whether to add or remove requests from the batch, thereby determining the target load balancing result.

[0123] In a feasible implementation, step A333 may include steps B3331 to B3333:

[0124] Step B3331: When the batch size comparison result is that the current batch size is smaller than the historical batch size, determining the target load balancing result as a request addition result;

[0125] In a specific implementation, when the batch size comparison result shows that the number of requests in the current batch is less than the average number of requests in the historical batches, it indicates that the current request batch needs to add requests to reach the average value of the historical batches, and then the target load balancing result is determined to be the request addition result.

[0126] Step B3332: When the batch size comparison result shows that the current batch size is equal to the historical batch size, determining that the target load balancing result is a normal load result;

[0127] In a specific implementation, when the batch size comparison result is that the number of requests in the current batch is equal to the average number of requests in the historical batches, it indicates that there is no need to add or remove requests in the current request batch, and thus the target load balancing result is determined to be a normal load result.

[0128] Step B3333: When the batch size comparison result is that the current batch size is larger than the historical batch size, determine that the target load balancing result is a request removal result.

[0129] In a specific implementation, when the batch size comparison result shows that the number of requests in the current batch is greater than the average number of requests in the historical batches, it indicates that the current request batch needs to remove requests to reach the average value of the historical batches, and then the target load balancing result is determined to be the request removal result.

[0130] It should be noted that in this embodiment, due to data dependencies, load imbalance between batches during the decoding phase can cause bubbles in the pipeline, impacting system performance. To ensure load balance, TD-Pipe's batch scheduler distributes the workload evenly throughout the decoding phase. The execution time of a single decoding step depends primarily on the batch size and the length of the KV cache. Although the KV cache length can vary significantly, linear operations in the Transformer model typically dominate the execution time for most sequence lengths. Therefore, TD-Pipe uses batch size as the sole metric for load balancing between batches. During the initial decoding phase, TD-Pipe divides requests into batches equal to the number of GPUs, with each batch containing the same number of requests. As decoding progresses, requests in different batches complete at different times, resulting in varying workloads across batches and, consequently, load imbalance. To address this issue, TD-Pipe employs cross-batch work stealing, using a sliding window mechanism to balance the load between batches in real time during the decoding process. After each batch completes its decoding step, this method calculates the average of the current batch size and the historical sizes of other batches and uses this average to decide whether to add or remove requests from that batch. Through this iterative process, bubbles in the pipeline can be effectively reduced, load balancing between decoding batches can be achieved, and thus system performance can be improved.

[0131] It is understandable that if Figure 5As shown in the figure, it illustrates how this embodiment gradually achieves load balancing and brings performance improvement. After each batch completes the decoding step, the average of the current batch size and the historical sizes of other batches is calculated. Then, the current batch size is compared with the average value, and a decision is made whether requests need to be added or removed from the batch. If requests are removed, these requests are temporarily put on hold, waiting to be added to subsequent batches. If requests need to be added, a check is performed to see if there are any put-on requests in the previous batch. Through this iterative process, the load between batches gradually tends to be balanced. Through the load balancing algorithm, the unbalanced workload is quickly balanced, and the reduced pipeline bubbles are also converted into performance improvements.

[0132] This embodiment switches the current inference phase from the pre-population phase to the decoding phase based on the phase switching timing, batches the client requests, and obtains multiple batch requests. Upon decoding the multiple batch requests, the current batch size of each batch request is obtained. Load balancing is performed based on a preset load balancing strategy and the current batch size of each batch request to obtain a target load balancing result. Through cross-batch work stealing and a sliding window mechanism, the workload of each batch in the decoding phase is dynamically adjusted, reducing bubbles in the pipeline and improving the overall efficiency of large-model inference.

[0133] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the high-throughput large-model inference method based on the time-separated pipeline architecture of the present application. More forms of simple transformations based on this technical concept are all within the scope of protection of the present application.

[0134] This application also provides a high-throughput large-model inference device based on a time-separated pipeline architecture. Please refer to Figure 6 The high-throughput large-model inference device based on the time-separated pipeline architecture includes:

[0135] The scheduling module 10 is used to perform pre-filling according to the client request when the current inference phase is the pre-filling phase, and determine the key value cache capacity of each request decision point;

[0136] A calculation module 20, configured to determine a phase switching timing based on the key value cache capacity and the preset memory capacity of each request decision point;

[0137] A load balancing module 30 is configured to switch the current inference phase from the pre-filling phase to the decoding phase according to the phase switching timing, and process the client request according to a preset load balancing strategy to obtain a target load balancing result;

[0138] The reasoning module 40 is used to perform reasoning on the large model according to the target load balancing result and the pipeline architecture, and obtain the output text corresponding to the client request according to the reasoning result.

[0139] Optionally, the scheduling module 10 is further configured to:

[0140] When the current inference phase is the pre-filling phase, the corresponding output length range is determined according to the preset greedy pre-filling strategy and the client request;

[0141] determining a plurality of request decision points according to the output length range;

[0142] Pre-filling is performed according to the client request, and when a request decision point is detected, the key value cache capacity of each request decision point is determined.

[0143] Optionally, the calculation module 20 is further configured to:

[0144] Compare the key value cache capacity of each request decision point with the preset memory capacity to obtain a capacity comparison result;

[0145] When the capacity comparison result shows that the key-value cache capacity is greater than or equal to the preset memory capacity, a phase switching timing is determined according to a request decision point corresponding to the key-value cache capacity.

[0146] Optionally, the load balancing module 30 is further configured to:

[0147] Switching the current inference phase from the pre-filling phase to the decoding phase according to the phase switching timing, and dividing the client requests into batches to obtain a plurality of batch requests;

[0148] Upon completion of decoding the plurality of batch requests, obtaining a current batch size of each batch request;

[0149] Load balancing is performed based on the preset load balancing strategy and the current batch size of each batch request to obtain the target load balancing result.

[0150] Optionally, the load balancing module 30 is further configured to:

[0151] Determine the corresponding historical batch size based on the preset load balancing algorithm and the current batch size of each batch request;

[0152] Comparing the current batch size of each batch request with the historical batch size to obtain a batch size comparison result;

[0153] A corresponding target load balancing result is determined according to the batch size comparison result.

[0154] Optionally, the load balancing module 30 is further configured to:

[0155] When the batch size comparison result is that the current batch size is smaller than the historical batch size, determining the target load balancing result as a request addition result;

[0156] When the batch size comparison result shows that the current batch size is equal to the historical batch size, determining that the target load balancing result is a normal load result;

[0157] When the batch size comparison result is that the current batch size is larger than the historical batch size, the target load balancing result is determined to be a request removal result.

[0158] Optionally, the reasoning module 40 is further configured to:

[0159] Analyze the execution time of each client request in the model inference results to determine the request baseline time;

[0160] Compare real-time request data and hyperparameter information to determine pipeline idle time;

[0161] Obtaining a target switching timing according to the request baseline time and the pipeline idle time;

[0162] The current inference stage is switched from the decoding stage to the pre-filling stage according to the target switching timing.

[0163] The high-throughput large-model inference device based on a time-separated pipeline architecture provided by this application adopts the high-throughput large-model inference method based on a time-separated pipeline architecture in the above-mentioned embodiment, which can solve the technical problem of low efficiency of existing large-model inference methods. Compared with the existing technology, the beneficial effects of the high-throughput large-model inference device based on a time-separated pipeline architecture provided by this application are the same as the beneficial effects of the high-throughput large-model inference method based on a time-separated pipeline architecture provided by the above-mentioned embodiment, and the other technical features of the high-throughput large-model inference device based on a time-separated pipeline architecture are the same as the features disclosed in the above-mentioned embodiment method, and are not repeated here.

[0164] The present application provides a high-throughput large-model inference device based on a time-separated pipeline architecture. The high-throughput large-model inference device based on the time-separated pipeline architecture includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the high-throughput large-model inference method based on the time-separated pipeline architecture in the above-mentioned embodiment one.

[0165] Reference below Figure 7 , which shows a schematic diagram of the structure of a high-throughput large-model inference device based on a time-separated pipeline architecture suitable for implementing the embodiments of the present application. The high-throughput large-model inference device based on a time-separated pipeline architecture in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The high-throughput large-model inference device based on the time-separated pipeline architecture shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0166] like Figure 7 As shown, a high-throughput, large-model inference device based on a time-separated pipeline architecture may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for the operation of the high-throughput, large-model inference device based on a time-separated pipeline architecture are also stored in RAM 1004. Processing device 1001, ROM 1002, and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, hard disk, etc.; and communication devices 1009. The communication devices 1009 can allow the high-throughput large-model inference device based on the time-separated pipeline architecture to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a high-throughput large-model inference device based on the time-separated pipeline architecture with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented or have instead.

[0167] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0168] The high-throughput large-model inference device based on a time-separated pipeline architecture provided by this application adopts the high-throughput large-model inference method based on a time-separated pipeline architecture in the above-mentioned embodiment, which can solve the technical problem of low efficiency of existing large-model inference methods. Compared with the existing technology, the beneficial effects of the high-throughput large-model inference device based on a time-separated pipeline architecture provided by this application are the same as the beneficial effects of the high-throughput large-model inference method based on a time-separated pipeline architecture provided by the above-mentioned embodiment, and the other technical features of the high-throughput large-model inference device based on a time-separated pipeline architecture are the same as the features disclosed in the method of the previous embodiment, and will not be repeated here.

[0169] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0170] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0171] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, and the computer-readable program instructions are used to execute the high-throughput large-model inference method based on the time-separated pipeline architecture in the above-mentioned embodiment.

[0172] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0173] The above-mentioned computer-readable storage medium can be included in a high-throughput large-model inference device based on a time-separated pipeline architecture; or it can exist independently without being assembled into a high-throughput large-model inference device based on a time-separated pipeline architecture.

[0174] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a high-throughput large-model inference device based on a time-separated pipeline architecture, the high-throughput large-model inference device based on the time-separated pipeline architecture: when the current inference stage is the pre-filling stage, pre-filling is performed according to the client request, and the key-value cache capacity of each request decision point is determined; the stage switching timing is determined according to the key-value cache capacity of each request decision point and the preset memory capacity; according to the stage switching timing, the current inference stage is switched from the pre-filling stage to the decoding stage, and the client request is processed according to a preset load balancing strategy to obtain a target load balancing result; large model inference is performed according to the target load balancing result and the pipeline architecture, and the output text corresponding to the client request is obtained according to the inference result.

[0175] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0176] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0177] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0178] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned high-throughput, large-model inference method based on a time-separated pipeline architecture. This computer-readable storage medium can address the technical issue of low efficiency associated with existing large-model inference methods. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the high-throughput, large-model inference method based on a time-separated pipeline architecture provided in the aforementioned embodiments, and are not further elaborated here.

[0179] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the high-throughput large model inference method based on the time-separated pipeline architecture as described above.

[0180] The computer program product provided in this application can address the technical issue of low efficiency in existing large-model inference methods. Compared to the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the high-throughput large-model inference method based on a time-separated pipeline architecture provided in the above-mentioned embodiment, and are not further elaborated here.

[0181] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A high-throughput large-model inference method based on a time-separated pipeline architecture, characterized in that: The high-throughput large model inference method based on the time-separated pipeline architecture includes: When the current inference phase is the pre-fill phase, pre-fill is performed based on client requests to determine the key-value cache capacity for each request decision point; Determine the phase switching timing based on the key-value cache capacity and preset memory capacity of each request decision point; Switching the current inference phase from the pre-filling phase to the decoding phase according to the phase switching timing, and processing the client request according to a preset load balancing strategy to obtain a target load balancing result; The large model is inferred based on the target load balancing result and the pipeline architecture, and the output text corresponding to the client request is obtained based on the inference result.

2. The method according to claim 1, wherein When the current inference phase is the pre-filling phase, the step of performing pre-filling according to the client request and determining the key value cache capacity of each request decision point includes: When the current inference phase is the pre-filling phase, the corresponding output length range is determined according to the preset greedy pre-filling strategy and the client request; determining a plurality of request decision points according to the output length range; Pre-filling is performed according to the client request, and when a request decision point is detected, the key value cache capacity of each request decision point is determined.

3. The method according to claim 1, wherein The step of determining the phase switching timing according to the key value cache capacity and the preset memory capacity of each request decision point includes: Compare the key value cache capacity of each request decision point with the preset memory capacity to obtain a capacity comparison result; When the capacity comparison result shows that the key-value cache capacity is greater than or equal to the preset memory capacity, a phase switching timing is determined according to a request decision point corresponding to the key-value cache capacity.

4. The method according to claim 1, wherein The step of switching the current inference stage from the pre-filling stage to the decoding stage according to the stage switching timing, and processing the client request according to a preset load balancing strategy to obtain a target load balancing result includes: Switching the current inference phase from the pre-filling phase to the decoding phase according to the phase switching timing, and dividing the client requests into batches to obtain a plurality of batch requests; Upon completion of decoding the plurality of batch requests, obtaining a current batch size of each batch request; Load balancing is performed based on the preset load balancing strategy and the current batch size of each batch request to obtain the target load balancing result.

5. The method according to claim 4, wherein The step of performing load balancing processing according to the preset load balancing strategy and the current batch size of each batch request to obtain a target load balancing result includes: Determine the corresponding historical batch size based on the preset load balancing strategy and the current batch size of each batch request; Comparing the current batch size of each batch request with the historical batch size to obtain a batch size comparison result; A corresponding target load balancing result is determined according to the batch size comparison result.

6. The method according to claim 5, wherein The step of determining the corresponding target load balancing result according to the batch size comparison result includes: When the batch size comparison result is that the current batch size is smaller than the historical batch size, determining the target load balancing result as a request addition result; When the batch size comparison result shows that the current batch size is equal to the historical batch size, determining that the target load balancing result is a normal load result; When the batch size comparison result is that the current batch size is larger than the historical batch size, the target load balancing result is determined to be a request removal result.

7. The method according to any one of claims 1 to 6, wherein After the step of performing reasoning on the large model according to the target load balancing result and the pipeline architecture, and obtaining the output text corresponding to the client request according to the reasoning result, the method further includes: Analyze the execution time of each client request in the model inference results to determine the request baseline time; Compare real-time request data and hyperparameter information to determine pipeline idle time; Obtaining a target switching timing according to the request baseline time and the pipeline idle time; The current inference stage is switched from the decoding stage to the pre-filling stage according to the target switching timing.

8. A high-throughput large-model inference device based on a time-separated pipeline architecture, characterized in that: The device comprises: The scheduling module is used to pre-fill according to the client request when the current inference phase is the pre-fill phase, and determine the key-value cache capacity of each request decision point; A calculation module, used to determine the phase switching timing based on the key value cache capacity and preset memory capacity of each request decision point; a load balancing module, configured to switch the current inference phase from the pre-filling phase to the decoding phase according to the phase switching timing, and process the client request according to a preset load balancing strategy to obtain a target load balancing result; The inference module is used to perform inference on the large model according to the target load balancing result and the pipeline architecture, and obtain the output text corresponding to the client request according to the inference result.

9. A high-throughput large-model inference device based on a time-separated pipeline architecture, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the high-throughput large model inference method based on a time-separated pipeline architecture as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the high-throughput large model inference method based on a time-separated pipeline architecture as described in any one of claims 1 to 7 are implemented.

Images