High-throughput large model reasoning method, device and equipment based on time separation type pipeline architecture and storage medium

Abstract

The invention discloses a high-throughput large-model reasoning method, device and equipment based on a time separation type pipeline architecture and a storage medium, and relates to the technical field of large-model reasoning. The high-throughput large-model reasoning method based on the time separation type pipeline architecture comprises the steps that when a current reasoning stage is a pre-filling stage, the pre-filling stage is a pre-filling stage; performing pre-filling according to a client request, and determining the key value cache capacity of each request decision point; determining a stage switching opportunity according to the key value cache capacity of each request decision point and a preset memory capacity; switching the current reasoning stage from the pre-filling stage to a decoding stage according to the stage switching opportunity, and processing the client request according to a preset load balancing strategy to obtain a target load balancing result; and according to the target load balance result and a pipeline architecture, performing large model reasoning, and according to a reasoning result, obtaining an output text corresponding to the client request. And the reasoning efficiency of the high-throughput large model is improved.

Description

Technical Field

[0001] The present application relates to the field of large model reasoning technology, and in particular to a high-throughput large model reasoning method, device, equipment and storage medium based on a time-separated pipeline architecture. Background Art

[0002] With the rapid development of artificial intelligence technology, key areas such as natural language processing, image and video generation have an increasing demand for cost reduction and efficiency improvement for large model reasoning. In particular, driven by applications such as Sora in video generation and Copilot in code generation, how to efficiently handle increasingly complex reasoning tasks has become an important challenge facing large model reasoning. These challenges become particularly prominent in large models that rely on autoregressive mechanisms. In the reasoning process, in addition to the increase in computing power, higher requirements are also placed on memory usage and inter-device communication efficiency. However, low-end accelerator cards such as NVIDIA L20 and A10, as well as domestic accelerator cards such as Tianshu and Muxi, are difficult to meet the needs of large model reasoning due to their small video memory space and weak interconnection bandwidth. At the same time, the traditional tensor parallel efficiency is low and the use cost of high-end equipment is high, resulting in a rapid increase in the overall cost of large model reasoning. Therefore, how to solve the low efficiency of existing large model reasoning methods has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of this application is to provide a high-throughput large model reasoning method, device, equipment and storage medium based on a time-separated pipeline architecture, aiming to solve the technical problem of low efficiency of existing large model reasoning methods.

[0004] To achieve the above objectives, the present application proposes a high-throughput large model reasoning method based on a time-separated pipeline architecture, and the high-throughput large model reasoning method based on the time-separated pipeline architecture includes: When the current inference phase is the pre-fill phase, pre-fill is performed according to the client request to determine the key value cache capacity of each request decision point; Determine the phase switching timing based on the key value cache capacity and preset memory capacity of each request decision point; Switching the current inference phase from the pre-filling phase to the decoding phase according to the phase switching timing, and processing the client request according to a preset load balancing strategy to obtain a target load balancing result; The large model is inferred according to the target load balancing result and the pipeline architecture, and the output text corresponding to the client request is obtained according to the inference result.

[0005] In one embodiment, when the current inference stage is a pre-filling stage, the step of performing pre-filling according to the client request and determining the key value cache capacity of each request decision point includes: When the current inference phase is the pre-filling phase, the corresponding output length range is determined according to the preset greedy pre-filling strategy and the client request; determining a plurality of request decision points according to the output length range; Pre-filling is performed according to the client request, and when a request decision point is detected, a key value cache capacity of each request decision point is determined.

[0006] In one embodiment, the step of determining the phase switching timing according to the key value cache capacity and the preset memory capacity of each request decision point includes: Compare the key value cache capacity of each request decision point with the preset memory capacity to obtain a capacity comparison result; When the capacity comparison result is that the key-value cache capacity is greater than or equal to the preset memory capacity, the phase switching timing is determined according to the request decision point corresponding to the key-value cache capacity.

[0007] In one embodiment, the step of switching the current inference stage from the pre-filling stage to the decoding stage according to the stage switching timing, and processing the client request according to a preset load balancing strategy to obtain a target load balancing result includes: Switching the current inference stage from the pre-filling stage to the decoding stage according to the stage switching timing, and dividing the client requests into batches to obtain a plurality of batch requests; Upon completion of decoding of the plurality of batch requests, obtaining a current batch size of each batch request; Load balancing is performed according to the preset load balancing strategy and the current batch size of each batch request to obtain the target load balancing result.

[0008] In one embodiment, the step of performing processing according to a preset load balancing algorithm and a current batch size of each batch request to obtain a target load balancing result includes: Determine the corresponding historical batch size according to the preset load balancing algorithm and the current batch size of each batch request; Compare the current batch size of each batch request with the historical batch size to obtain a batch size comparison result; A corresponding target load balancing result is determined according to the batch size comparison result.

[0009] In one embodiment, the step of determining the corresponding target load balancing result according to the batch size comparison result includes: When the batch size comparison result is that the current batch size is smaller than the historical batch size, determining the target load balancing result as a request addition result; When the batch size comparison result is that the current batch size is equal to the historical batch size, determining that the target load balancing result is a normal load result; When the batch size comparison result is that the current batch size is greater than the historical batch size, the target load balancing result is determined to be a request removal result.

[0010] In one embodiment, after the step of performing reasoning of the large model according to the target load balancing result and the pipeline architecture, and obtaining the output text corresponding to the client request according to the reasoning result, the step further includes: Analyze the execution time of each client request in the model inference results to determine the request baseline time; Compare real-time request data and hyperparameter information to determine pipeline idle time; Obtaining a target switching timing according to the requested baseline time and the pipeline idle time; The current inference stage is switched from the decoding stage to the pre-filling stage according to the target switching opportunity.

[0011] In addition, to achieve the above purpose, the present application also proposes a high-throughput large model reasoning device based on a time-separated pipeline architecture, and the high-throughput large model reasoning device based on a time-separated pipeline architecture includes: The scheduling module is used to perform pre-filling according to the client request when the current inference stage is the pre-filling stage, and determine the key value cache capacity of each request decision point; A calculation module, used to determine the phase switching timing according to the key value cache capacity and preset memory capacity of each request decision point; A load balancing module, configured to switch the current inference phase from the pre-filling phase to the decoding phase according to the phase switching timing, and process the client request according to a preset load balancing strategy to obtain a target load balancing result; The inference module is used to perform inference on the large model according to the target load balancing result and the pipeline architecture, and obtain the output text corresponding to the client request according to the inference result.

[0012] In addition, to achieve the above-mentioned objectives, the present application also proposes a high-throughput large model inference device based on a time-separated pipeline architecture, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the high-throughput large model inference method based on a time-separated pipeline architecture as described above.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the high-throughput large model inference method based on the time-separated pipeline architecture as described above are implemented.

[0014] In addition, to achieve the above-mentioned objectives, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the high-throughput large model reasoning method based on the time-separated pipeline architecture as described above.

[0015] The present application determines the key value cache capacity of each request decision point by performing pre-filling according to the client request when the current reasoning stage is the pre-filling stage; determines the stage switching timing according to the key value cache capacity and the preset memory capacity of each request decision point; switches the current reasoning stage from the pre-filling stage to the decoding stage according to the stage switching timing, and processes the client request according to the preset load balancing strategy to obtain the target load balancing result; performs reasoning of the large model according to the target load balancing result and the pipeline architecture, and obtains the output text corresponding to the client request according to the reasoning result. The efficiency of high-throughput large model reasoning is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 A flowchart diagram of the first embodiment of the high-throughput large model reasoning method based on the time-separated pipeline architecture of the present application; Figure 2 An overall schematic diagram of a runtime system provided for Embodiment 1 of the high-throughput large-model reasoning method based on a time-separated pipeline architecture of the present application; Figure 3 A schematic diagram of the strength accumulation method provided in Example 1 of the high-throughput large model reasoning method based on a time-separated pipeline architecture of the present application; Figure 4 A flow chart of the second embodiment of the high-throughput large model reasoning method based on the time-separated pipeline architecture of the present application; Figure 5 A load balancing schematic diagram provided for Embodiment 2 of the high-throughput large model reasoning method based on a time-separated pipeline architecture of the present application; Figure 6 This is a schematic diagram of the module structure of a high-throughput large model reasoning device based on a time-separated pipeline architecture according to an embodiment of the present application; Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the high-throughput large model reasoning method based on the time-separated pipeline architecture in the embodiment of the present application.

[0019] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0021] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of the embodiment of the present application is: when the current reasoning stage is the pre-filling stage, pre-filling is performed according to the client request, and the key value cache capacity of each request decision point is determined; the stage switching timing is determined according to the key value cache capacity of each request decision point and the preset memory capacity; according to the stage switching timing, the current reasoning stage is switched from the pre-filling stage to the decoding stage, and the client request is processed according to the preset load balancing strategy to obtain the target load balancing result; the large model is inferred according to the target load balancing result and the pipeline architecture, and the output text corresponding to the client request is obtained according to the reasoning result.

[0023] With the rapid development of artificial intelligence technology, key areas such as natural language processing, image and video generation have an increasing demand for cost reduction and efficiency improvement for large model reasoning. In particular, driven by applications such as Sora in video generation and Copilot in code generation, how to efficiently handle increasingly complex reasoning tasks has become an important challenge facing large model reasoning. These challenges become particularly prominent in large models that rely on autoregressive mechanisms. In the reasoning process, in addition to the increase in computing power, higher requirements are also placed on memory usage and inter-device communication efficiency. However, low-end accelerator cards such as NVIDIA L20 and A10, as well as domestic accelerator cards such as Tianshu and Muxi, are difficult to meet the needs of large model reasoning due to their small video memory space and weak interconnection bandwidth. At the same time, the traditional tensor parallel efficiency is low and the use cost of high-end equipment is high, resulting in a rapid increase in the overall cost of large model reasoning. Therefore, how to solve the low efficiency of existing large model reasoning methods has become an urgent problem to be solved.

[0024] The present application determines the key value cache capacity of each request decision point by performing pre-filling according to the client request when the current reasoning stage is the pre-filling stage; determines the stage switching timing according to the key value cache capacity and the preset memory capacity of each request decision point; switches the current reasoning stage from the pre-filling stage to the decoding stage according to the stage switching timing, and processes the client request according to the preset load balancing strategy to obtain the target load balancing result; performs reasoning of the large model according to the target load balancing result and the pipeline architecture, and obtains the output text corresponding to the client request according to the reasoning result. The efficiency of high-throughput large model reasoning is improved.

[0025] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a high-throughput large model reasoning device based on a time-separated pipeline architecture that can realize the above functions. The following takes the high-throughput large model reasoning device based on the time-separated pipeline architecture as an example of the execution subject to illustrate this embodiment and the following embodiments.

[0026] Based on this, the embodiment of the present application provides a high-throughput large model reasoning method based on a time-separated pipeline architecture, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the high-throughput large model reasoning method based on the time-separated pipeline architecture of the present application.

[0027] In this embodiment, the high-throughput large model reasoning method based on the time-separated pipeline architecture includes steps S10 to S40: Step S10, when the current reasoning stage is a pre-filling stage, pre-filling is performed according to the client request to determine the key value cache capacity of each request decision point; It should be noted that this embodiment proposes a time dimension separation architecture for optimizing the computing efficiency in the reasoning process. Similar to the spatial dimension separation technology (such as the MoonCake system, and the MoonCake system is a technical architecture specifically used to optimize the reasoning process of large-scale machine learning models), the present invention separates the two types of computing tasks, Prefill and Decode, according to their different characteristics. The reasoning process is divided into two stages: the Prefill stage and the Decode stage. The Prefill stage is mainly responsible for gradually filling the request into the video memory to ensure that the Decode stage can continue to process enough batches of data, thereby improving the overall computing efficiency. Through time separation, the two different computing modes can avoid mutual interference, effectively reducing the pipeline bubbles caused by the Prefill-Decode switching, and significantly optimizing the reasoning performance.

[0028] It should be noted that the overall design of this embodiment is as follows Figure 2 As shown. The system is divided into a centralized engine and a distributed runtime. The centralized engine is responsible for scheduling request batches through a custom scheduling algorithm combined with continuous batch processing technology, and uses PagedAttention (paged attention mechanism) technology to achieve refined management of KV cache (Key-Value Cache). In the scheduling stage, the centralized engine manages and regulates the distributed runtime through RPC (Remote Procedure Call) technology. The distributed runtime uses a stateful Worker to execute tasks assigned by the engine and maintain the communication context of the task. Asynchronous point-to-point communication technology is used between workers to achieve efficient data transmission and ensure fast and seamless data exchange between each stage of the pipeline. Through these optimized designs, the present invention can effectively improve the resource utilization and overall performance of the system, and is particularly suitable for efficient reasoning scenarios of large-scale models. After receiving the request, the centralized engine puts the request into the scheduler and uses a continuous batch processing method to send the request to the distributed pipeline runtime in batches for execution. Therefore, it is very important to design a reasonable scheduling algorithm to achieve efficient operation of the pipeline.

[0029] It can be understood that the pre-filling stage refers to the model processing the input data, initializing and filling the KV cache, and preparing for subsequent decoding. The client request refers to the large model inference request initiated by the client. The request decision point refers to certain pre-specified time points or steps in the decoding process, which are used to evaluate and manage memory usage. By checking the usage of the KV cache (Key-Value Cache) at these decision points, you can decide whether to switch from the pre-filling stage to the decoding stage. The key-value cache capacity refers to the KV cache usage at the decision point.

[0030] In a specific implementation, when the reasoning stage of the large model is the pre-filling stage, the large model reasoning request initiated by the client is pre-filled, and then the KV cache usage of these decision points is updated after each pre-filling is completed, that is, the key value cache capacity of each decision point is determined.

[0031] In a feasible implementation, step S10 may include steps A11 to A13: Step A11, when the current inference stage is the pre-filling stage, determining the corresponding output length range according to the preset greedy pre-filling strategy and the client request; It should be noted that the preset greedy pre-filling strategy refers to a preset greedy pre-filling algorithm based on an AI method, and the output length range refers to the predicted output length range of the client request.

[0032] In the specific implementation, this embodiment is based on the AI ​​greedy pre-filling method, that is, an additional header (a separate linear layer) is directly added to the last hidden state of the LLM to predict the output length range of the request, and then the large model inference request initiated by the client is predicted through the greedy pre-filling algorithm based on the AI ​​method, and finally the output length range of the predicted client request, that is, the output length range, is obtained.

[0033] Step A12, determining a plurality of request decision points according to the output length range; It is understandable that the request decision point refers to the decision point location in the client request.

[0034] In a specific implementation, based on the predicted output length range of the client request, certain future decoding steps are designated as decision points, and the locations of the decision points in the client request are determined. For example, based on the predicted output length range of the client request, a decision point is set every 20 steps, thereby obtaining multiple request decision points.

[0035] Step A13, pre-filling is performed according to the client request, and when a request decision point is detected, the key value cache capacity of each request decision point is determined.

[0036] It can be understood that when the reasoning stage of the large model is the pre-filling stage, the large model reasoning request initiated by the client is pre-filled, and then the KV cache usage of these decision points is updated after each pre-filling is completed, that is, the key value cache capacity of each decision point is determined.

[0037] Step S20, determining the phase switching timing according to the key value cache capacity and the preset memory capacity of each request decision point; It can be understood that the preset memory capacity refers to a critical value of the memory capacity used to determine whether to perform a phase switch, and the phase switch timing refers to the switching timing from the pre-filling phase to the decoding phase.

[0038] In a specific implementation, the key value cache capacity of each request decision point is compared with the critical value of the memory capacity used to determine whether to switch the stage, so as to screen out the request decision points corresponding to the key value cache capacity greater than or equal to the critical value of the memory capacity used to determine whether to switch the stage, and finally determine the switching timing from the pre-filling stage to the decoding stage, that is, the stage switching timing.

[0039] In a feasible implementation, step S20 may include steps A21-A22: Step A21, comparing the key value cache capacity of each request decision point with the preset memory capacity to obtain a capacity comparison result; It is understandable that the capacity comparison result refers to the size comparison result between the key-value cache capacity and the preset memory capacity.

[0040] In a specific implementation, the key value cache capacity of each request decision point is compared with the critical value of the memory capacity used to determine whether to switch the stage, and the capacity comparison result is that the key value cache capacity of each request decision point is greater than or equal to the critical value of the memory capacity used to determine whether to switch the stage, or the key value cache capacity of each request decision point is less than the critical value of the memory capacity used to determine whether to switch the stage.

[0041] Step A22, when the capacity comparison result is that the key-value cache capacity is greater than or equal to the preset memory capacity, determining the phase switching timing according to the request decision point corresponding to the key-value cache capacity.

[0042] In a specific implementation, when the capacity comparison result is that the key-value cache capacity is greater than or equal to the memory capacity critical value used to determine whether to switch stages, it indicates that the KV cache of the decision point exceeds the memory capacity, and then the timing of switching from the pre-filling stage to the decoding stage is determined according to the decision point.

[0043] It should be noted that in the pre-filling stage, TD-Pipe needs to determine the best time to switch to the decoding stage. In order to reduce frequent switching between pre-filling and decoding, TD-Pipe usually tends to perform more pre-filling operations before switching. However, since the memory usage in the decoding stage will continue to increase, too much pre-filling may occupy most of the memory, resulting in insufficient video memory in the decoding stage and frequent recalculation or data unloading, which significantly reduces system performance. Since the output length is unknown before the request is completed, the peak of memory usage cannot be predicted in advance. In order to start enough pre-filling operations in the decoding stage without excessively exceeding the memory capacity, this embodiment proposes an AI-based greedy pre-filling method. We directly add an additional head (a separate linear layer) to the last hidden state of the LLM (Large Language Model) to predict the output length range of the request. In order to determine the best time to switch from pre-filling to decoding, we designed a dynamic programming algorithm to simulate the memory usage of the decoding stage after the new pre-filling is completed: by specifying certain future decoding steps as decision points, we can update the KV cache usage of these decision points after each pre-filling is completed, and determine the switching time by checking whether the memory usage of a decision point exceeds the memory capacity. This method can help the system better control memory usage and reduce recalculation or data offloading operations caused by video memory overflow without introducing significant computational overhead.

[0044] Step S30, switching the current inference stage from the pre-filling stage to the decoding stage according to the stage switching timing, and processing the client request according to a preset load balancing strategy to obtain a target load balancing result; It can be understood that the preset load balancing strategy refers to a preset strategy for balancing the load between batches in real time during the decoding process, and the target load balancing result includes a request addition result, a normal load result, and a request removal result.

[0045] In a specific implementation, the current inference stage is switched from the pre-filling stage to the decoding stage according to the switching timing from the pre-filling stage to the decoding stage, and then the client request is decoded and processed, and the load between batches is balanced in real time during the decoding process to obtain the target load balancing result.

[0046] It should be noted that in the initial stage of the decoding phase, TD-Pipe (Time-Dimension Pipeline) divides the requests into batches equal to the number of GPUs, and each batch contains the same number of requests. As the decoding proceeds, the requests in different batches have different completion times, resulting in constantly changing workloads in each batch, causing load imbalance. To solve this problem, TD-Pipe adopts a cross-batch work stealing method, which balances the load between batches in real time during the decoding process through a sliding window mechanism. After each batch completes the decoding step, the method calculates the average of the current batch size and the historical sizes of other batches, and decides whether to add or remove requests to the batch based on the average. Through this iterative process, bubbles in the pipeline can be effectively reduced, load balancing between decoding batches can be achieved, and system performance can be improved.

[0047] Step S40, performing inference on the large model according to the target load balancing result and the pipeline architecture, and obtaining the output text corresponding to the client request according to the inference result.

[0048] It is understandable that the pipeline architecture refers to the pipeline parallel architecture. Pipeline parallelism is an inter-layer parallel method. Each GPU (Graphics Processing Unit) is responsible for different parts of the model. The same input flows through multiple devices in sequence during the inference process. The output text refers to the final results generated by the large model during the inference process. These results are presented in text form and correspond to the client's request. The output text can be in various forms, depending on the type of client request and the application scenario.

[0049] In the specific implementation, different batches of requests are load balanced according to the target load balancing result, and iterative decoding is performed, that is, the large model is inferred, and then the high-throughput large model inference result is obtained. Finally, the high-throughput large model inference result is parsed to obtain the final result generated by the large model corresponding to the client request during the inference process, that is, the output text.

[0050] In a feasible implementation manner, step S40 may include steps A41 to A44: Step A41, analyzing the execution time of each client request in the model inference result to determine the request baseline time; It can be understood that the execution time refers to the time it takes for each client request to be decoded, and the request baseline time refers to the baseline time of each request under high computing intensity.

[0051] In the specific implementation, when each request starts to be processed, the start time is recorded. After each request is processed, the end time is recorded. The execution time of each request is calculated, that is, the end time minus the start time. Collect the execution time data of all requests. Use statistical methods (such as mean, median, standard deviation, etc.) to analyze the execution time data. Determine the baseline time of the request, that is, the average execution time of the request under normal circumstances. Finally, the baseline time of each request under high computing intensity is obtained.

[0052] Step A42, comparing the real-time request data and the hyperparameter information to determine the pipeline idle time; It can be understood that real-time request data refers to the client requests and related information received by the system during operation, hyperparameter information refers to the various parameters set during system design and configuration, which affect the performance and behavior of the system, and pipeline idle time refers to the time wasted due to pipeline bubbles.

[0053] In the specific implementation, real-time request data is collected, including request ID, processing time, etc., and hyperparameter information such as batch size, number of pre-filling steps, decision point interval, etc. is compared. Then the average processing time, maximum processing time and minimum processing time are calculated. Finally, the time waste caused by pipeline bubbles, that is, the pipeline idle time, is calculated.

[0054] Step A43, obtaining a target switching timing according to the requested baseline time and the pipeline idle time; It can be understood that the target switching timing refers to the switching timing from the decoding phase to the pre-filling phase.

[0055] In a specific implementation, the baseline time of each request under high computing intensity and the time wasted due to pipeline bubbles are compared. When the baseline time of each request under high computing intensity is greater than the time wasted due to pipeline bubbles, it indicates that the performance degradation exceeds the impact of pipeline idle time, and the system should switch from decoding to pre-filling. The switching timing from the decoding stage to the pre-filling stage, that is, the target switching timing, is determined.

[0056] Step A44: Switch the current inference stage from the decoding stage to the pre-filling stage according to the target switching timing.

[0057] In a specific implementation, the current inference stage is switched from the decoding stage to the pre-filling stage according to the switching timing from the decoding stage to the pre-filling stage. The decoding stage is switched to the pre-filling stage at the optimal switching timing, thereby avoiding the problem of too rapid decrease in computing intensity and a subsequent decrease in decoding efficiency.

[0058] It should be noted that in the pipeline architecture of TD-Pipe, the computational intensity of the decoding stage will decrease as the batch is completed. If the computational intensity decreases too quickly, the decoding efficiency will also decrease. However, if you frequently switch between the decoding and pre-filling stages, it may increase the bubbles in the pipeline and affect the overall performance. To solve this problem, the intensity accumulation method decides whether to switch stages by comparing the performance loss caused by the decrease in computational intensity with the pipeline bubbles generated in the switching stage. This method analyzes the execution time, calculates the baseline time of each request under high computational intensity, and estimates the time waste caused by pipeline bubbles by comparing real-time data and hyperparameter information, so as to find the best time to switch. Figure 3 As shown, the intensity accumulation method compares the performance degradation caused by the decrease in computation intensity with the pipeline idle time caused by stage switching. If the performance degradation exceeds the impact of pipeline idle time, the system should switch from decoding to prefilling.

[0059] This embodiment determines the key value cache capacity of each request decision point by performing pre-filling according to the client request when the current reasoning stage is the pre-filling stage; determines the stage switching timing according to the key value cache capacity and the preset memory capacity of each request decision point; switches the current reasoning stage from the pre-filling stage to the decoding stage according to the stage switching timing, and processes the client request according to the preset load balancing strategy to obtain the target load balancing result; performs reasoning of the large model according to the target load balancing result and the pipeline architecture, and obtains the output text corresponding to the client request according to the reasoning result. The efficiency of high-throughput large model reasoning is improved.

[0060] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 In the high-throughput large model reasoning method based on the time-separated pipeline architecture, step S30 further includes steps S31 to S33: Step S31, switching the current inference stage from the pre-filling stage to the decoding stage according to the stage switching timing, and dividing the client request into batches to obtain a plurality of batch requests; It can be understood that batch request refers to combining multiple independent client requests together and processing them as a whole.

[0061] In a specific implementation, the current inference stage is switched from the pre-filling stage to the decoding stage according to the switching timing from the pre-filling stage to the decoding stage, and the large model inference requests initiated by the client are divided into batches, that is, multiple independent client requests are combined together and processed as a whole to obtain multiple batch requests. In this embodiment, the requests are divided into batches equal to the number of GPUs, and each batch contains the same number of requests.

[0062] Step S32, when the decoding of the plurality of batch requests is completed, obtaining the current batch size of each batch request; It is understood that the current batch size refers to the number of requests included in one batch.

[0063] In the specific implementation, the present embodiment uses a sliding window mechanism to balance the load between batches in real time during the decoding process. When each batch completes the decoding step, the method calculates the number of requests included in a batch, that is, the current batch size.

[0064] Step S33, performing load balancing processing according to a preset load balancing strategy and the current batch size of each batch request to obtain a target load balancing result.

[0065] It can be understood that the current inference stage is switched from the pre-filling stage to the decoding stage according to the switching timing from the pre-filling stage to the decoding stage, and the number of requests in each batch of requests is load balanced according to the load balancing algorithm between decoding batches to obtain the target load balancing result.

[0066] In a feasible implementation, step S33 may include steps A331 to A333: Step A331, determining the corresponding historical batch size according to the preset load balancing algorithm and the current batch size of each batch request; It can be understood that the historical batch size refers to the average number of requests in batches before the current batch.

[0067] In a specific implementation, the total number of requests in each batch before the current batch is obtained based on the decoding batch load balancing algorithm, and then the average value of the total number of requests in each batch before the current batch is calculated to obtain the historical batch size.

[0068] Step A332, comparing the current batch size of each batch request with the historical batch size to obtain a batch size comparison result; It can be understood that the batch size comparison result refers to the comparison result of the current batch size with the historical batch size.

[0069] In a specific implementation, the number of requests in the current batch is compared with the average number of requests in historical batches, and the batch size comparison result is that the number of requests in the current batch is greater than the average number of requests in historical batches, or the number of requests in the current batch is equal to the average number of requests in historical batches, or the number of requests in the current batch is less than the average number of requests in historical batches.

[0070] Step A333: Determine the corresponding target load balancing result according to the batch size comparison result.

[0071] In a specific implementation, the current batch size is compared with the average of the historical sizes of other batches, and a decision is made based on the average whether to add or remove requests from the batch, thereby determining the target load balancing result.

[0072] In a feasible implementation, step A333 may include steps B3331 to B3333: Step B3331, when the batch size comparison result is that the current batch size is smaller than the historical batch size, determining the target load balancing result as a request addition result; In a specific implementation, when the batch size comparison result is that the number of requests in the current batch is less than the average number of requests in the historical batches, it indicates that the current request batch needs to add requests to reach the average value of the historical batches, and then the target load balancing result is determined to be the request addition result.

[0073] Step B3332, when the batch size comparison result is that the current batch size is equal to the historical batch size, determining that the target load balancing result is a normal load result; In a specific implementation, when the batch size comparison result is that the number of requests in the current batch is equal to the average number of requests in the historical batches, it indicates that there is no need to add or remove requests in the current request batch, and thus the target load balancing result is determined to be a normal load result.

[0074] Step B3333, when the batch size comparison result is that the current batch size is larger than the historical batch size, determine that the target load balancing result is a request removal result.

[0075] In a specific implementation, when the batch size comparison result is that the number of requests in the current batch is greater than the average number of requests in the historical batches, it indicates that the current request batch needs to remove requests to reach the average value of the historical batches, and then the target load balancing result is determined to be the request removal result.

[0076] It should be noted that in the decoding stage of this embodiment, due to data dependency, load imbalance between batches may cause bubbles in the pipeline, thereby affecting system performance. To ensure load balancing, TD-Pipe's batch scheduler needs to evenly distribute the workload throughout the decoding stage. The execution time of a single decoding step mainly depends on the batch size and the length of the KV cache. Although the KV cache length may vary greatly, since linear operations in the Transformer model usually dominate the execution time of most sequence lengths, TD-Pipe uses batch size as the only criterion for measuring load balance between batches. In the initial stage of the decoding stage, TD-Pipe divides the requests into batches equal to the number of GPUs, and each batch contains the same number of requests. As the decoding proceeds, the completion time of requests in different batches is different, resulting in constant changes in the workload of each batch, causing load imbalance. To solve this problem, TD-Pipe adopts a cross-batch work stealing method to balance the load between batches in real time during the decoding process through a sliding window mechanism. After each batch completes the decoding step, the method calculates the average of the current batch size and the historical sizes of other batches, and decides whether to add or remove requests to the batch based on the average. Through this iterative process, bubbles in the pipeline can be effectively reduced, load balancing between decoding batches can be achieved, and thus system performance can be improved.

[0077] It is understandable that if Figure 5 As shown, it is explained how this embodiment gradually achieves load balancing and brings performance improvement. After each batch completes the decoding step, the average of the current batch size and the historical sizes of other batches is calculated. Then, the current batch size is compared with the average value, and a decision is made whether requests need to be added or removed from the batch. If the requests are removed, these requests will be temporarily put on hold, waiting to be added to subsequent batches. If requests need to be added, it will be checked whether there are any put on hold requests in the previous batch. Through this iterative process, the load between batches gradually tends to be balanced. Through the load balancing algorithm, the unbalanced workload is quickly balanced, and the reduced pipeline bubbles are also converted into performance improvements.

[0078] This embodiment switches the current inference stage from the pre-filling stage to the decoding stage according to the stage switching timing, and divides the client request into batches to obtain multiple batch requests; when the decoding of the multiple batch requests is completed, the current batch size of each batch request is obtained; load balancing is performed according to the preset load balancing strategy and the current batch size of each batch request to obtain the target load balancing result. Through cross-batch work stealing and sliding window mechanisms, the workload of each batch in the decoding stage is dynamically adjusted, bubbles in the pipeline are reduced, and the overall efficiency of large model reasoning is improved.

[0079] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the high-throughput large model inference method based on the time-separated pipeline architecture of the present application. More forms of simple transformations based on this technical concept are all within the scope of protection of the present application.

[0080] This application also provides a high-throughput large model reasoning device based on a time-separated pipeline architecture, please refer to Figure 6 , the high-throughput large model reasoning device based on the time-separated pipeline architecture includes: The scheduling module 10 is used to perform pre-filling according to the client request and determine the key value cache capacity of each request decision point when the current reasoning stage is the pre-filling stage; A calculation module 20, used to determine the phase switching timing according to the key value cache capacity and the preset memory capacity of each request decision point; A load balancing module 30, configured to switch the current inference phase from the pre-filling phase to the decoding phase according to the phase switching timing, and process the client request according to a preset load balancing strategy to obtain a target load balancing result; The reasoning module 40 is used to perform reasoning on the large model according to the target load balancing result and the pipeline architecture, and obtain the output text corresponding to the client request according to the reasoning result.

[0081] Optionally, the scheduling module 10 is further used to: When the current inference phase is the pre-filling phase, the corresponding output length range is determined according to the preset greedy pre-filling strategy and the client request; determining a plurality of request decision points according to the output length range; Pre-filling is performed according to the client request, and when a request decision point is detected, a key value cache capacity of each request decision point is determined.

[0082] Optionally, the calculation module 20 is further used for: Compare the key value cache capacity of each request decision point with the preset memory capacity to obtain a capacity comparison result; When the capacity comparison result is that the key-value cache capacity is greater than or equal to the preset memory capacity, the phase switching timing is determined according to the request decision point corresponding to the key-value cache capacity.

[0083] Optionally, the load balancing module 30 is further used for: Switching the current inference stage from the pre-filling stage to the decoding stage according to the stage switching timing, and dividing the client requests into batches to obtain a plurality of batch requests; Upon completion of decoding of the plurality of batch requests, obtaining a current batch size of each batch request; Load balancing is performed according to the preset load balancing strategy and the current batch size of each batch request to obtain the target load balancing result.

[0084] Optionally, the load balancing module 30 is further used for: Determine the corresponding historical batch size according to the preset load balancing algorithm and the current batch size of each batch request; Compare the current batch size of each batch request with the historical batch size to obtain a batch size comparison result; A corresponding target load balancing result is determined according to the batch size comparison result.

[0085] Optionally, the load balancing module 30 is further used for: When the batch size comparison result is that the current batch size is smaller than the historical batch size, determining the target load balancing result as a request addition result; When the batch size comparison result is that the current batch size is equal to the historical batch size, determining that the target load balancing result is a normal load result; When the batch size comparison result is that the current batch size is greater than the historical batch size, the target load balancing result is determined to be a request removal result.

[0086] Optionally, the reasoning module 40 is further configured to: Analyze the execution time of each client request in the model inference results to determine the request baseline time; Compare real-time request data and hyperparameter information to determine pipeline idle time; Obtaining a target switching timing according to the requested baseline time and the pipeline idle time; The current inference stage is switched from the decoding stage to the pre-filling stage according to the target switching opportunity.

[0087] The high-throughput large-model inference device based on a time-separated pipeline architecture provided by the present application adopts the high-throughput large-model inference method based on a time-separated pipeline architecture in the above-mentioned embodiment, which can solve the technical problem of low efficiency of the existing large-model inference method. Compared with the prior art, the beneficial effects of the high-throughput large-model inference device based on a time-separated pipeline architecture provided by the present application are the same as the beneficial effects of the high-throughput large-model inference method based on a time-separated pipeline architecture provided by the above-mentioned embodiment, and the other technical features in the high-throughput large-model inference device based on a time-separated pipeline architecture are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0088] The present application provides a high-throughput large-model inference device based on a time-separated pipeline architecture. The high-throughput large-model inference device based on the time-separated pipeline architecture includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the high-throughput large-model inference method based on the time-separated pipeline architecture in the above-mentioned embodiment one.

[0089] Reference below Figure 7 , which shows a schematic diagram of the structure of a high-throughput large-model inference device based on a time-separated pipeline architecture suitable for implementing the embodiment of the present application. The high-throughput large-model inference device based on a time-separated pipeline architecture in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions: tablet computers), PMPs (Portable Media Players: portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The high-throughput large-model inference device based on the time-separated pipeline architecture shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0090] like Figure 7As shown, a high-throughput large-model inference device based on a time-separated pipeline architecture may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: ReadOnly Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: RandomAccess Memory) 1004. Various programs and data required for the operation of a high-throughput large-model inference device based on a time-separated pipeline architecture are also stored in RAM1004. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the high-throughput large-model inference device based on the time-separated pipeline architecture to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a high-throughput large-model inference device based on the time-separated pipeline architecture with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0091] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0092] The high-throughput large-model inference device based on the time-separated pipeline architecture provided by the present application adopts the high-throughput large-model inference method based on the time-separated pipeline architecture in the above-mentioned embodiment, which can solve the technical problem of low efficiency of the existing large-model inference method. Compared with the prior art, the beneficial effects of the high-throughput large-model inference device based on the time-separated pipeline architecture provided by the present application are the same as the beneficial effects of the high-throughput large-model inference method based on the time-separated pipeline architecture provided by the above-mentioned embodiment, and the other technical features of the high-throughput large-model inference device based on the time-separated pipeline architecture are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0093] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0094] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0095] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the high-throughput large model reasoning method based on the time-separated pipeline architecture in the above-mentioned embodiment.

[0096] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0097] The above-mentioned computer-readable storage medium may be included in a high-throughput large-model reasoning device based on a time-separated pipeline architecture; or it may exist independently without being assembled into a high-throughput large-model reasoning device based on a time-separated pipeline architecture.

[0098] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a high-throughput large model inference device based on a time-separated pipeline architecture, the high-throughput large model inference device based on the time-separated pipeline architecture: when the current inference stage is the pre-filling stage, pre-filling is performed according to the client request, and the key value cache capacity of each request decision point is determined; the stage switching timing is determined according to the key value cache capacity of each request decision point and the preset memory capacity; according to the stage switching timing, the current inference stage is switched from the pre-filling stage to the decoding stage, and the client request is processed according to the preset load balancing strategy to obtain the target load balancing result; the large model is inferred according to the target load balancing result and the pipeline architecture, and the output text corresponding to the client request is obtained according to the inference result.

[0099] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0100] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0101] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0102] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned high-throughput large-model reasoning method based on the time-separated pipeline architecture, and can solve the technical problem of low efficiency of the existing large-model reasoning method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the high-throughput large-model reasoning method based on the time-separated pipeline architecture provided in the above-mentioned embodiment, and will not be repeated here.

[0103] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the high-throughput large model reasoning method based on the time-separated pipeline architecture as described above.

[0104] The computer program product provided by the present application can solve the technical problem of low efficiency of existing large model reasoning methods. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the high-throughput large model reasoning method based on the time-separated pipeline architecture provided by the above embodiment, and will not be repeated here.

[0105] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A high-throughput large model reasoning method based on a time-separated pipeline architecture, characterized in that: The high-throughput large model reasoning method based on the time-separated pipeline architecture includes: When the current inference phase is the pre-fill phase, pre-fill is performed according to the client request to determine the key value cache capacity of each request decision point; Determine the phase switching timing based on the key value cache capacity and preset memory capacity of each request decision point; Switching the current inference phase from the pre-filling phase to the decoding phase according to the phase switching timing, and processing the client request according to a preset load balancing strategy to obtain a target load balancing result; The large model is inferred according to the target load balancing result and the pipeline architecture, and the output text corresponding to the client request is obtained according to the inference result.

2. The method according to claim 1, characterized in that When the current reasoning stage is the pre-filling stage, the step of pre-filling according to the client request and determining the key value cache capacity of each request decision point includes: When the current inference phase is the pre-filling phase, the corresponding output length range is determined according to the preset greedy pre-filling strategy and the client request; determining a plurality of request decision points according to the output length range; Pre-filling is performed according to the client request, and when a request decision point is detected, a key value cache capacity of each request decision point is determined.

3. The method according to claim 1, characterized in that The step of determining the phase switching timing according to the key value cache capacity and the preset memory capacity of each request decision point includes: Compare the key value cache capacity of each request decision point with the preset memory capacity to obtain a capacity comparison result; When the capacity comparison result is that the key-value cache capacity is greater than or equal to the preset memory capacity, the phase switching timing is determined according to the request decision point corresponding to the key-value cache capacity.

4. The method according to claim 1, characterized in that The step of switching the current inference stage from the pre-filling stage to the decoding stage according to the stage switching timing, and processing the client request according to a preset load balancing strategy to obtain a target load balancing result includes: Switching the current inference stage from the pre-filling stage to the decoding stage according to the stage switching timing, and dividing the client requests into batches to obtain a plurality of batch requests; Upon completion of decoding of the plurality of batch requests, obtaining a current batch size of each batch request; Load balancing is performed according to the preset load balancing strategy and the current batch size of each batch request to obtain the target load balancing result.

5. The method according to claim 4, characterized in that The step of performing processing according to the preset load balancing algorithm and the current batch size of each batch request to obtain a target load balancing result includes: Determine the corresponding historical batch size according to the preset load balancing algorithm and the current batch size of each batch request; Compare the current batch size of each batch request with the historical batch size to obtain a batch size comparison result; A corresponding target load balancing result is determined according to the batch size comparison result.

6. The method according to claim 5, characterized in that The step of determining the corresponding target load balancing result according to the batch size comparison result comprises: When the batch size comparison result is that the current batch size is smaller than the historical batch size, determining the target load balancing result as a request addition result; When the batch size comparison result is that the current batch size is equal to the historical batch size, determining that the target load balancing result is a normal load result; When the batch size comparison result is that the current batch size is greater than the historical batch size, the target load balancing result is determined to be a request removal result.

7. The method according to any one of claims 1 to 6, characterized in that After the step of performing reasoning of the large model according to the target load balancing result and the pipeline architecture, and obtaining the output text corresponding to the client request according to the reasoning result, the method further includes: Analyze the execution time of each client request in the model inference results to determine the request baseline time; Compare real-time request data and hyperparameter information to determine pipeline idle time; Obtaining a target switching timing according to the requested baseline time and the pipeline idle time; The current inference stage is switched from the decoding stage to the pre-filling stage according to the target switching opportunity.

8. A high-throughput large-model inference device based on a time-separated pipeline architecture, characterized in that: The device comprises: The scheduling module is used to perform pre-filling according to the client request when the current inference stage is the pre-filling stage, and determine the key value cache capacity of each request decision point; A calculation module, used to determine the phase switching timing according to the key value cache capacity and preset memory capacity of each request decision point; A load balancing module, configured to switch the current inference phase from the pre-filling phase to the decoding phase according to the phase switching timing, and process the client request according to a preset load balancing strategy to obtain a target load balancing result; The inference module is used to perform inference on the large model according to the target load balancing result and the pipeline architecture, and obtain the output text corresponding to the client request according to the inference result.

9. A high-throughput large-model inference device based on a time-separated pipeline architecture, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of a high-throughput large model inference method based on a time-separated pipeline architecture as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the high-throughput large model inference method based on a time-separated pipeline architecture as described in any one of claims 1 to 7 are implemented.

Images