Model inference scheduling method, readable storage medium and electronic device
By optimizing the allocation of inference requests based on the predicted number of output tokens and device performance data, the problem of device load imbalance and cache management in the pre-filling and decoding separation architecture is solved, thereby improving inference stability and resource utilization and reducing system overhead.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI CHAOWEI WUJI ELECTRONIC TECHNOLOGY CO LTD
- Filing Date
- 2026-01-22
- Publication Date
- 2026-05-29
AI Technical Summary
In existing technologies, the inference architecture that separates pre-filling and decoding lacks the ability to estimate the overall computational load of requests during scheduling, resulting in load imbalance and low resource utilization during the decoding stage. Furthermore, the key-value cache management and transmission costs are not optimized, leading to significant overhead.
By predicting the number of output tokens based on historical inference request data, and combining the performance data of pre-filled and decoding devices with key-value cache status information, the estimated cost is calculated. The device with the lowest overall time overhead is selected for inference request allocation, and the cache data layout and access path are optimized.
It effectively avoids device congestion during the decoding stage, improves inference stability in long text generation scenarios and resource utilization in heterogeneous hybrid deployment environments, and reduces cross-device transmission and repeated generation overhead.
Smart Images
Figure CN122111650A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a model inference scheduling method, a readable storage medium, and an electronic device. Background Technology
[0002] As the scale of large language model parameters continues to expand, the inference process exhibits distinct phased characteristics in terms of computational patterns and resource requirements. To improve system throughput and resource utilization, the industry has gradually adopted an inference architecture that separates the prefill phase (often referred to as the P-side) and the decoding phase (often referred to as the D-side). The prefill phase primarily performs full attention calculations for input prompts and generates the key-value cache (KV Cache). This phase is computationally intensive and is typically deployed on devices with high computational throughput. The decoding phase, on the other hand, performs incremental calculations on a token-by-token basis, making it latency-sensitive and typically deployed on devices more suitable for low-latency, frequent iterative computations. In this prefill and decoding separation architecture, a complete inference request involves P-side computation, key-value cache generation and transmission, and D-side decoding execution. In actual products, the P-side and D-side typically constitute independent device pools and collaborate through remote procedure calls or high-speed interconnects.
[0003] Currently, existing inference scheduling schemes based on pre-filling and decoding separation architectures mostly have relatively simple scheduling logic. Common practices include: simply allocating inference requests arriving at the P-side through round-robin or static resource allocation; or making scheduling decisions solely based on the instantaneous queue length or idle status of the P-side device; or, after the P-side completes computation, fixing the decoding request to a pre-defined D-side device. While these approaches achieve some degree of decoupling between the two phases in deployment, they do not fully explore and utilize the scheduling flexibility offered by the pre-filling and decoding separation architecture.
[0004] Analysis of the actual operation of inference products employing a pre-filling and decoding separation architecture reveals the following main shortcomings of existing technologies: First, the P-side scheduling lacks consideration of the overall request workload, leading to significant load pressure and increased latency during the decoding stage when facing long text generation requests. Second, the heterogeneous computing resources in the P-side and D-side device pools are not effectively utilized, resulting in uneven overall system resource utilization and some devices becoming performance bottlenecks. Third, the management and scheduling decisions for key-value caching are disconnected, leading to duplicate cache generation or high cross-device transmission overhead, reducing the overall efficiency of the inference system. Summary of the Invention
[0005] To address the problems in existing technologies, such as unbalanced decoding pressure due to lack of overall computational load estimation during scheduling, low resource utilization due to ineffective matching of heterogeneous computing power and stage task characteristics, and significant overhead and redundancy introduced by the failure to optimize the location and transmission costs of key-value caches in the early stages of scheduling, this invention provides a model inference scheduling method, a readable storage medium, and an electronic device. The technical solution of this application effectively avoids device congestion in the decoding stage caused by improper pre-filling stage scheduling, improves system stability in long text generation scenarios, enhances the overall utilization of computing devices in heterogeneous hybrid deployment environments, and reduces cross-device transmission and duplicate generation overhead caused by improper key-value cache location management.
[0006] To achieve the above objectives, a first aspect of the present invention provides a model inference scheduling method applied to a model inference scheduling system, the model inference scheduling system comprising a plurality of pre-filled side devices in a pre-filled side device pool and a plurality of decoding side devices in a decoding side device pool, the method comprising: Receive inference requests to be processed, and parse the inference requests to obtain the number of input tokens; Acquire historical reasoning request data, and predict the number of output tokens corresponding to the reasoning request to be processed based on the historical reasoning request data; The pre-fill performance data of each pre-fill side device, the decoding performance data of each decoding side device, and the status information of the first key-value cache stored on each pre-fill side device are obtained, wherein the first key-value cache is obtained based on the historical inference request data; Based on the number of input tokens, the number of output tokens, the pre-filling performance data, the decoding performance data, and the state information, the estimated cost of each pre-filling side device is calculated. The estimated cost at least represents the total time cost of the pre-filling stage and the decoding stage when the inference request to be processed is allocated to each pre-filling side device for execution. The pre-filled device with the lowest overall time overhead is selected as the target pre-filled device to perform the pre-filling calculation for the inference request to be processed. After the target pre-filling device completes the pre-filling calculation, based on the storage location information of the generated second key-value cache, the decoding performance data of each decoding device, and the real-time load status of each decoding device, a target decoding device is selected from the decoding device pool to perform the decoding calculation of the inference request to be processed.
[0007] In one possible implementation of the first aspect, calculating the estimated cost of each pre-filled device based on the number of input tokens, the number of output tokens, the pre-filling performance data, the decoding performance data, and the state information includes: The pre-filling stage cost is calculated based on the number of input lexical units, the pre-filling performance data of the pre-filling side device, and the status information of the first key-value cache. The decoding stage cost is calculated based on the number of output tokens and the decoding performance data of the decoding-side device. The key-value cache transmission cost is calculated based on the number of input tokens and the data transmission cost of the first key-value cache from the pre-filling side device to the decoding side device. The pre-filling stage cost, the decoding stage cost, and the key-value cache transmission cost are weighted and summed to obtain the estimated cost of each pre-filling side device.
[0008] In one possible implementation of the first aspect, the pre-filling performance data includes the unit input lexical computation delay, and the pre-filling stage cost is calculated using the following formula:
[0009] in, The cost of the pre-filling stage; Calculate the delay for the unit input word of the i-th pre-filled side device; The number of input tokens; The current waiting cost for the i-th pre-filled side device; This refers to the cache hit rate for the number of input tokens.
[0010] In one possible implementation of the first aspect, the state information of the first key-value cache includes reusability information of the first key-value cache, and the method further includes: If, based on the reusability information, it is determined that the prefix of the input lexical sequence of the reasoning request to be processed matches the prefix of the input lexical sequence of the historical reasoning request corresponding to the first key-value cache, then the cache hit rate... Greater than zero.
[0011] In one possible implementation of the first aspect, the decoding performance data includes the unit output lexical computation delay, and the decoding stage cost is calculated using the following formula:
[0012] in, The cost of the decoding stage; Calculate the delay for the unit output word of the j-th decoding device; The current load penalty term for the j-th decoding-side device is determined based on the real-time load status of the decoding-side device. The number of output tokens.
[0013] In one possible implementation of the first aspect, the key-value cache transfer cost is calculated using the following formula:
[0014] in, The key-value cache transmission cost; The number of input tokens; The data transfer cost of the first key-value cache from the i-th pre-filled side device to the j-th decoding side device is zero, wherein the data transfer cost is zero if the storage location of the second key-value cache is located in a storage location that can be directly accessed by the decoding side device.
[0015] In one possible implementation of the first aspect, the weighted summation of the pre-filling stage cost, the decoding stage cost, and the key-value cache transmission cost to obtain the estimated cost for each pre-filling side device includes: For each of the plurality of pre-filled side devices, the sum of the pre-filling stage cost, the decoding stage cost, and the key-value cache transmission cost corresponding to the combination of the pre-filled side device and each decoding side device is calculated to obtain the total cost of multiple combinations; The minimum value among the multiple combined total costs is selected as the estimated cost when allocating the inference request to be processed to the pre-filled side device for execution.
[0016] In one possible implementation of the first aspect, predicting the number of output tokens corresponding to the reasoning request to be processed based on the historical reasoning request data includes: Extract the metadata or content features of the reasoning request to be processed; Based on the metadata or content features, determine the historical reasoning requests that match the reasoning request to be processed; Based on the actual number of output tokens in the matched historical reasoning requests, predict the number of output tokens corresponding to the reasoning request to be processed.
[0017] In a second aspect, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the model inference scheduling method as described in the first aspect and any possible implementation thereof.
[0018] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the model inference scheduling method as described in the first aspect and any possible implementation thereof.
[0019] Compared with the prior art, the beneficial effects of this application are as follows: This application's technical solution predicts the number of output tokens corresponding to the inference request to be processed based on historical inference request data. This allows scheduling decisions to pre-assess the computational scale of the decoding stage, thus proactively considering the potential load of the decoding stage when selecting the pre-filling side (P-side) device, helping to avoid device congestion caused by excessive load in the decoding stage. The estimated cost is calculated by simultaneously acquiring and utilizing the performance data of the pre-filling side device, the performance data of the decoding side (D-side) device, and the state information of the stored first key-value cache (KV Cache). This cost comprehensively represents the overall time overhead from pre-filling to decoding. The pre-filling side device with the lowest overall time overhead is selected as the target pre-filling side device to perform the pre-filling computation of the inference request to be processed. This minimizes the overall end-to-end time overhead and improves inference stability in scenarios such as long text generation. By selecting the target decoding side device based on decoding performance data and real-time load status, refined and dynamic utilization of the computing power of heterogeneous decoding side devices is achieved, helping to improve the overall computing power utilization rate in hybrid deployment environments. Furthermore, by selecting a pre-filled device with the assistance of the first key-value cache status information, and making decisions based on the storage location information of the newly generated second key-value cache in subsequent decoding scheduling, the layout and access path of cached data can be effectively optimized, thereby reducing the system overhead of key-value cache cross-device transmission and repeated generation. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 According to some embodiments of this application, a long text application scenario is illustrated; Figure 2 According to some embodiments of this application, a structural block diagram of a model inference scheduling system is shown; Figure 3 According to some embodiments of this application, a flowchart of a model inference scheduling method is shown; Figure 4 According to some embodiments of this application, a structural block diagram of an electronic device is shown. Detailed Implementation
[0022] The illustrative embodiments of this application include, but are not limited to, a model inference scheduling method, a readable storage medium, and an electronic device.
[0023] To facilitate understanding of the technical solution of this application, some terms involved in this application are explained below.
[0024] 1. Prefill stage The Prefill phase is the first stage in the inference process of a large language model. Its task is to receive and process complete user input prompts, generate initial output results through a complete model forward computation, and prepare a key data structure called the KV Cache for subsequent generation processes. This stage is computationally intensive, capable of processing all input tokens in parallel, but its computational complexity increases rapidly with the length of the input. 2. Decode stage The Decode phase is the inference process that follows the Prefill phase. Based on the output of the Prefill phase and the KV cache, this phase generates subsequent output sequences word by word in an autoregressive manner. Each computation in this phase processes only one new word, and its performance is primarily limited by memory bandwidth.
[0025] 3. Type prompt (hint) The input prompt is a text instruction or question provided by the user to the large language model. It serves as the starting point and basis for the model's inference. The model generates relevant answers or performs specific tasks by understanding the meaning of the prompt. The clarity and specificity of the prompt directly affect the quality and relevance of the model's output.
[0026] 4. Key-Value Cache KV Cache is a key technique in attention mechanisms used to accelerate the inference process. It is generated during the Prefill phase and caches the key-value matrices generated by the model during computation. During the Decode phase, the model does not need to recalculate these intermediate results for already processed terms; instead, it directly reuses the contents of the KV Cache, thereby reducing redundant computation and improving generation efficiency.
[0027] 5. Token Tokens are the basic units used by large language models to process text. The model segments the input text into a sequence of tokens for processing, and the output is also a sequence of tokens, which is then converted into human-readable text.
[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0029] As mentioned above, existing technologies suffer from technical problems such as unbalanced decoding pressure due to lack of overall computational load estimation during scheduling, low resource utilization due to ineffective matching of heterogeneous computing power and stage task characteristics, and significant overhead and redundancy introduced due to the failure to optimize the location and transmission cost of key-value cache in the early stages of scheduling.
[0030] In view of this, this application provides a model inference scheduling method. Specifically, this application predicts the number of output tokens corresponding to the inference request to be processed based on historical inference request data, enabling scheduling decisions to pre-assess the computational scale of the decoding stage. This allows for proactive consideration of the potential load of the decoding stage when selecting the pre-filling side (P-side) device, helping to avoid device congestion caused by excessive load in the decoding stage. By simultaneously acquiring and utilizing the performance data of the pre-filling side device, the performance data of the decoding side (D-side) device, and the state information of the stored first key-value cache (KVCache), the estimated cost is calculated. This cost comprehensively represents the overall time overhead from pre-filling to decoding. The pre-filling side device with the lowest overall time overhead is selected as the target pre-filling side device to perform the pre-filling computation of the inference request to be processed. This minimizes the overall end-to-end time overhead and improves inference stability in scenarios such as long text generation. By selecting the target decoding side device based on decoding performance data and real-time load status, refined and dynamic utilization of the computing power of heterogeneous decoding side devices is achieved, which helps to improve the overall computing power utilization in a hybrid deployment environment. Furthermore, by selecting a pre-filled device with the assistance of the first key-value cache status information, and making decisions based on the storage location information of the newly generated second key-value cache in subsequent decoding scheduling, the layout and access path of cached data can be effectively optimized, thereby reducing the system overhead of key-value cache cross-device transmission and repeated generation.
[0031] The technical solution of this application effectively avoids device congestion in the decoding stage caused by improper scheduling in the pre-filling stage, improves the stability of the system in long text generation scenarios, improves the overall utilization of computing devices in heterogeneous hybrid deployment environments, and reduces the overhead of cross-device transmission and repeated generation caused by improper management of key-value cache location.
[0032] The following is a combination of... Figure 1 This paper briefly introduces one application scenario of the technical solution of this application.
[0033] Figure 1According to some embodiments of this application, a long text application scenario is illustrated. A user inputs a user request through a local terminal device 20, for example, the user inputs "Help me write a 1000-word article about spring" through a computer. After receiving the user request, the local terminal device 20 sends the user request to the model inference scheduling system 10. The model inference scheduling system 10 processes the request as follows by applying the model inference scheduling method provided in this application: First, it parses the request to obtain the number of input tokens; then, based on historical inference request data, it predicts the number of output tokens required to generate a 1000-word article about spring; simultaneously, it obtains the real-time performance data of each pre-filling side device, the decoding capability data of each decoding side device, and the status information of the existing historical key-value cache on each P-side device. Combining all the above information, an estimated cost is calculated for each candidate P-side device. This cost quantifies the total end-to-end time overhead from pre-filling to decoding that may be caused if the device is selected to perform the pre-filling calculation for the current request. Ultimately, the P-side device with the lowest estimated cost, i.e., the optimal overall time overhead, is selected as the target device to perform the pre-filling calculation for this request, generating the first word of the article and the complete key-value cache. Subsequently, based on the specific storage location of this new key-value cache, combined with the real-time decoding performance and load status of each D-side device, the most suitable target decoding-side device is selected to efficiently iterate word by word to complete the generation of the entire article, and the generated entire article (i.e., the long text) is returned to the local terminal device 20.
[0034] The following is combined with Figure 2 This paper briefly introduces a model inference scheduling system 10 provided in this application.
[0035] It should be noted that the model inference scheduling system 10 provided in this application adopts an architecture that separates pre-filling and decoding.
[0036] Figure 2 According to some embodiments of this application, a model inference scheduling system 10 is shown, with reference to... Figure 2 The model inference scheduling system 10 includes multiple pre-filled side devices in a pre-filled side device pool 13, multiple decoding side devices in a decoding side device pool 14, a scheduling control module 11, and a KV Cache management module 12. The decoding side devices are high-throughput computing devices suitable for performing large-scale matrix calculations. The decoding side devices are low-latency devices suitable for performing word-by-word decoding calculations.
[0037] The scheduling control module 11 receives inference requests to be processed and assigns them to selected pre-filling side devices for pre-filling computation. The KV Cache management module 12 manages and registers the status information of the key-value cache generated by the pre-filling computation. This status information may include the generation location, size, and reusability status of the key-value cache. After the pre-filling computation is completed, the scheduling control module 11 further assigns subsequent decoding computation tasks to selected target decoding side devices for execution based on information such as the storage location of the newly generated key-value cache, and finally outputs the results.
[0038] Specifically, the scheduling control module 11 receives the inference request to be processed and parses it to obtain the number of input tokens. Simultaneously, it acquires historical inference request data to predict the corresponding number of output tokens. Subsequently, the scheduling control module 11 acquires the pre-filling performance data of each pre-filling side device, the decoding performance data of each decoding side device, and the first key-value cache status information provided by the KV Cache management module 12. Based on the number of input tokens, the number of output tokens, the pre-filling performance data, the decoding performance data, and the status information, the scheduling control module 11 calculates the estimated cost of each pre-filling side device. This estimated cost at least represents the total time overhead of the pre-filling and decoding stages when the inference request to be processed is allocated to each pre-filling side device. The scheduling control module 11 selects the pre-filling side device with the lowest total time overhead as the target pre-filling side device and assigns a pre-filling task to it. The selected target pre-filling side device performs the pre-filling calculation and generates a new key-value cache. The KVCache management module 12 then registers information such as the storage location of this newly generated key-value cache. Subsequently, the scheduling control module 11 selects the target decoding-side device from the decoding-side device pool 14 and assigns decoding tasks based on the storage location information of the newly generated key-value cache, the decoding performance data of each decoding-side device, and the real-time load status. Finally, the target decoding-side device accesses the key-value cache and completes the decoding calculation, outputting the inference result. This means that the decision of the scheduling control module 11 is not only based on the idle status of the pre-filled side devices, but also proactively considers the impact of different device combinations on the overall time overhead, including computation time, cache transmission time, and queuing time caused by the load of the decoding devices.
[0039] The following will combine Figure 3 The flowchart shown below provides a detailed description of a model inference scheduling method provided in this application. Figure 3 According to some embodiments of this application, a flowchart of a model inference scheduling method is shown, with reference to... Figure 3 The model inference scheduling method provided in this application includes the following steps: Step S11: Receive the inference request to be processed and parse the inference request to obtain the number of input tokens.
[0040] For example, suppose that for a certain reasoning request, the number of input lexical units obtained after parsing is 2048.
[0041] Step S12: Obtain historical inference request data, and predict the number of output tokens corresponding to the inference request to be processed based on the historical inference request data.
[0042] In some embodiments, predicting the number of output tokens corresponding to a reasoning request to be processed based on historical reasoning request data includes: extracting metadata or content features of the reasoning request to be processed; determining historical reasoning requests that match the reasoning request to be processed based on the metadata or content features; and predicting the number of output tokens corresponding to the reasoning request to be processed based on the actual number of output tokens of the matched historical reasoning requests.
[0043] For example, suppose that for a certain reasoning request, the number of input tokens obtained by parsing is 2048, and the number of output tokens corresponding to the reasoning request to be processed based on historical reasoning request data is 1024.
[0044] In some embodiments, the process of acquiring and constructing historical inference request data specifically includes: during operation, the model inference scheduling system 10 continuously records each completed inference request and its corresponding actual processing data, and constructs a historical request database. For example, for each completed inference request, its request metadata, content features, number of input tokens, and the final number of actual output tokens are recorded. This data is stored in a structured manner to form a historical request data set for prediction. When it is necessary to predict the number of output tokens for a new inference request to be processed, the scheduling control module 11 extracts the metadata or content features of the new inference request to be processed, and performs similarity matching in the historical request database accordingly to find several similar historical inference requests. Then, based on the actual number of output tokens of these similar historical requests, the number of output tokens corresponding to the current inference request to be processed is predicted. Furthermore, the historical request database can be updated periodically to reflect the latest request patterns and data distribution. The request metadata includes, for example, request type, application, and user identifier.
[0045] Step S13: Obtain the pre-fill performance data of each pre-fill side device, the decoding performance data of each decoding side device, and the status information of the first key-value cache stored on each pre-fill side device, wherein the first key-value cache is obtained based on historical inference request data.
[0046] In some embodiments, the pre-filling performance data of each pre-filling side device, the decoding performance data of each decoding side device, and the status information of the first key-value cache stored on each pre-filling side device are obtained, specifically through the following methods: For each pre-filling side device, its pre-filling performance data is obtained through a combination of offline benchmark testing and online monitoring. For example, for the pre-filling side device GPU-A, its unit input token computation latency is calibrated to be 0.8μs through offline benchmark testing, indicating high parallelism; its current task queue length is obtained in real time through online monitoring to calculate the current waiting cost. For each decoding side device, its decoding performance data is also obtained through benchmark testing to obtain the unit output token computation latency, for example, the unit output token computation latency of the decoding side device GPU-C is 1.5μs; at the same time, the current resource utilization rate is collected in real time as the basis for the real-time load status, for example, the current load of GPU-C is 60%. Regarding the status information of the first key-value cache, the KVCache management module 12 maintains a global cache status table. This table records and updates in real time the storage location, occupied space size, and reusable prefix information used to determine whether a new request can reuse the cache for the first key-value cache stored in the memory of each pre-filled device. The scheduling control module 11 obtains the latest first key-value cache status information by querying this global cache status table, for example, if it finds that there is currently no reusable cache on device GPU-A that matches the prefix of the request to be processed.
[0047] Step S14: Based on the number of input tokens, the number of output tokens, pre-filling performance data, decoding performance data, and state information, calculate the estimated cost of each pre-filling side device. The estimated cost at least represents the total time cost of the pre-filling stage and the decoding stage when the inference request to be processed is allocated to each pre-filling side device for execution.
[0048] In some embodiments, the estimated cost of each pre-filling side device is calculated based on the number of input tokens, the number of output tokens, pre-filling performance data, decoding performance data, and state information. This includes: calculating the pre-filling stage cost based on the number of input tokens, the pre-filling performance data of the pre-filling side device, and the state information of the first key-value cache; calculating the decoding stage cost based on the number of output tokens and the decoding performance data of the decoding side device; calculating the key-value cache transmission cost based on the number of input tokens and the data transmission cost of the first key-value cache from the pre-filling side device to the decoding side device; and weighted summing the pre-filling stage cost, the decoding stage cost, and the key-value cache transmission cost to obtain the estimated cost of each pre-filling side device.
[0049] In some embodiments, pre-filling performance data includes unit input lexical computation latency, and the pre-filling stage cost is calculated using the following formula:
[0050] in, This is the cost of the pre-filling stage; Calculate the delay for the unit input word of the i-th pre-filled side device; Input the number of tokens; The current waiting cost for the i-th pre-filled side device; This is the cache hit rate based on the number of input tokens.
[0051] In some embodiments, the state information of the first key-value cache includes reusability information of the first key-value cache, and the method further includes: if it is determined from the reusability information that the prefix of the input lexical sequence of the inference request to be processed matches the prefix of the input lexical sequence of the historical inference request corresponding to the first key-value cache, then the cache hit rate is... Greater than zero.
[0052] In some embodiments, decoding performance data includes unit output token computation delay, and the decoding stage cost is calculated using the following formula:
[0053] in, This is the cost of the decoding phase; Calculate the delay for the unit output word of the j-th decoding device; This is the current load penalty term for the j-th decoding-side device, which is determined based on the real-time load status of the decoding-side device. This is the number of tokens to output.
[0054] In some embodiments, the key-value cache transfer cost is calculated using the following formula:
[0055] in, The cost of key-value caching for data transfer; Input the number of tokens; The data transfer cost of the first key-value cache from the i-th pre-filled side device to the j-th decoding side device is zero, wherein the data transfer cost is zero if the storage location of the second key-value cache is located in a storage location that can be directly accessed by the decoding side device.
[0056] In some embodiments, the pre-filling stage cost, decoding stage cost, and key-value cache transmission cost are weighted and summed to obtain the estimated cost of each pre-filling side device. This includes: for each of the multiple pre-filling side devices, calculating the sum of the pre-filling stage cost, decoding stage cost, and key-value cache transmission cost corresponding to the combination of the pre-filling side device and each decoding side device to obtain multiple combined total costs; and selecting the minimum value from the multiple combined total costs as the estimated cost when allocating the inference request to be processed to the pre-filling side device for execution.
[0057] In some embodiments, the total cost of multiple combinations is calculated using the following formula. : = + +
[0058] Step S15: Select the pre-filled device with the lowest overall time cost as the target pre-filled device to perform pre-filling calculations for the inference request to be processed.
[0059] For example, suppose the pre-filled device pool 13 includes devices GPU-A and GPU-B, and the performance of these two devices is shown in Table 1 below: Table 1
[0060] Assuming that the decoding-side device pool 14 includes devices GPU-C and GPU-D, the performance of these two devices is shown in Table 2 below: Table 2
[0061] The scheduling control module 11 obtains the following information in real time: Neither GPU-A nor GPU-B has a reusable prefix cache; both GPU-C and GPU-D can access GPU-A via high-speed interconnect, but accessing GPU-B requires cross-Non-Uniform Memory Access (NUMA) transmission; high-speed interconnect corresponds to a high-bandwidth network, while NUMA corresponds to a low-bandwidth network. The unit cost of key-value cache cross-device transmission is: 0.2μs from GPU-A to GPU-C; and 0.6μs from GPU-B to GPU-D. By evaluating the estimated cost of executing pending inference requests on GPU-A and GPU-B, it is found that: if GPU-A is chosen to perform pre-filling computation on pending inference requests, the pre-filling computation time on GPU-A is shorter, and the corresponding key-value cache storage location is closer to the high-performance decoding device, resulting in lower subsequent decoding costs. If GPU-B is chosen to perform pre-filling computation on the inference requests to be processed, the pre-filling computation on GPU-B takes a long time, and the storage location of the corresponding key-value cache is far from the high-performance decoding device, resulting in high transmission costs for the key-value cache and potentially becoming a bottleneck in the subsequent decoding stage. Therefore, GPU-A is ultimately selected as the target pre-filling device to perform the pre-filling computation on the inference requests to be processed.
[0062] Step S16: After the target pre-filling device completes the pre-filling calculation, based on the storage location information of the generated second key-value cache, the decoding performance data of each decoding device, and the real-time load status of each decoding device, the target decoding device is selected from the decoding device pool 14 to perform the decoding calculation of the inference request to be processed.
[0063] For example, after device GPU-A completes the pre-filling calculation, the generated second key-value cache is stored in device GPU-A. Combining the decoding performance data and real-time load status of decoding-side devices GPU-C and GPU-D, device GPU-C is selected as the target decoding-side device to achieve end-to-end collaborative inference.
[0064] This application's technical solution predicts the number of output tokens corresponding to the inference request to be processed based on historical inference request data. This allows scheduling decisions to pre-assess the computational scale of the decoding stage, thus proactively considering the potential load of the decoding stage when selecting the pre-filling side (P-side) device, helping to avoid device congestion caused by excessive load in the decoding stage. The estimated cost is calculated by simultaneously acquiring and utilizing the performance data of the pre-filling side device, the performance data of the decoding side (D-side) device, and the state information of the stored first key-value cache (KV Cache). This cost comprehensively represents the overall time overhead from pre-filling to decoding. The pre-filling side device with the lowest overall time overhead is selected as the target pre-filling side device to perform the pre-filling computation of the inference request to be processed. This minimizes the overall end-to-end time overhead and improves inference stability in scenarios such as long text generation. By selecting the target decoding side device based on decoding performance data and real-time load status, refined and dynamic utilization of the computing power of heterogeneous decoding side devices is achieved, helping to improve the overall computing power utilization rate in hybrid deployment environments. Furthermore, by selecting a pre-filled device with the assistance of the first key-value cache status information, and making decisions based on the storage location information of the newly generated second key-value cache in subsequent decoding scheduling, the layout and access path of cached data can be effectively optimized, thereby reducing the system overhead of key-value cache cross-device transmission and repeated generation.
[0065] It is understood that the execution order of steps S11 to S16 above is only an illustration. In other embodiments, other execution orders may be used, and some steps may be split or combined. This is not limited here.
[0066] Embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the model inference scheduling method in any of the above embodiments.
[0067] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the model inference scheduling method as described in the first aspect and any possible implementation thereof.
[0068] Furthermore, embodiments of this application also provide an electronic device 400, such as... Figure 4 As shown, the electronic device 400 includes a memory 401 and a processor 402. The memory 401 is used to store computer programs executable by the processor 402; the processor 402 is used to execute the computer programs in the memory 401 to implement the model inference scheduling method in any of the above embodiments.
[0069] Figure 4The electronic device 400 shown also includes a communication interface 403. The processor 402, memory 401, and communication interface 403 are connected via a communication bus and communicate with each other.
[0070] Processor 402 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the above scheme program.
[0071] The aforementioned electronic device 400 also includes an encoder (not shown). The encoder can be a specific hardware module or a functional module in the processor 402. The encoder can encode the video stream using the video encoding method provided in the embodiments of this application.
[0072] Communication interface 403 is used to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Networks (WLAN), etc.
[0073] Memory 401 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory may exist independently and be connected to the processor via a bus. Memory may also be integrated with the processor.
[0074] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.
[0075] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.
[0076] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0077] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.
Claims
1. A model inference scheduling method, characterized in that, The method, applied to a model inference scheduling system, which includes multiple pre-filled side devices in a pre-filled side device pool and multiple decoding side devices in a decoding side device pool, comprises: Receive inference requests to be processed, and parse the inference requests to obtain the number of input tokens; Acquire historical reasoning request data, and predict the number of output tokens corresponding to the reasoning request to be processed based on the historical reasoning request data; The pre-fill performance data of each pre-fill side device, the decoding performance data of each decoding side device, and the status information of the first key-value cache stored on each pre-fill side device are obtained, wherein the first key-value cache is obtained based on the historical inference request data; Based on the number of input tokens, the number of output tokens, the pre-filling performance data, the decoding performance data, and the state information, the estimated cost of each pre-filling side device is calculated. The estimated cost at least represents the total time cost of the pre-filling stage and the decoding stage when the inference request to be processed is allocated to each pre-filling side device for execution. The pre-filled device with the lowest overall time overhead is selected as the target pre-filled device to perform the pre-filling calculation for the inference request to be processed. After the target pre-filling device completes the pre-filling calculation, based on the storage location information of the generated second key-value cache, the decoding performance data of each decoding device, and the real-time load status of each decoding device, a target decoding device is selected from the decoding device pool to perform the decoding calculation of the inference request to be processed.
2. The method according to claim 1, characterized in that, The step of calculating the estimated cost of each pre-filled device based on the number of input tokens, the number of output tokens, the pre-filling performance data, the decoding performance data, and the state information includes: The pre-filling stage cost is calculated based on the number of input lexical units, the pre-filling performance data of the pre-filling side device, and the status information of the first key-value cache. The decoding stage cost is calculated based on the number of output tokens and the decoding performance data of the decoding-side device. The key-value cache transmission cost is calculated based on the number of input tokens and the data transmission cost of the first key-value cache from the pre-filling side device to the decoding side device. The pre-filling stage cost, the decoding stage cost, and the key-value cache transmission cost are weighted and summed to obtain the estimated cost of each pre-filling side device.
3. The method according to claim 2, characterized in that, The pre-filling performance data includes the computation delay per unit input lexical unit, and the cost of the pre-filling stage is calculated using the following formula: in, The cost of the pre-filling stage; Calculate the delay for the unit input word of the i-th pre-filled side device; The number of input tokens; The current waiting cost for the i-th pre-filled side device; This refers to the cache hit rate for the number of input tokens.
4. The method according to claim 3, characterized in that, The state information of the first key-value cache includes the reusability information of the first key-value cache, and the method further includes: If, based on the reusability information, it is determined that the prefix of the input lexical sequence of the reasoning request to be processed matches the prefix of the input lexical sequence of the historical reasoning request corresponding to the first key-value cache, then the cache hit rate... Greater than zero.
5. The method according to claim 2, characterized in that, The decoding performance data includes the unit output token computation delay, and the cost of the decoding stage is calculated using the following formula: in, The cost of the decoding stage; Calculate the delay for the unit output word of the j-th decoding device; The current load penalty term for the j-th decoding-side device is determined based on the real-time load status of the decoding-side device. The number of output tokens.
6. The method according to claim 2, characterized in that, The key-value cache transfer cost is calculated using the following formula: in, The key-value cache transmission cost; The number of input tokens; The data transfer cost of the first key-value cache from the i-th pre-filled side device to the j-th decoding side device is zero, wherein the data transfer cost is zero if the storage location of the second key-value cache is located in a storage location that can be directly accessed by the decoding side device.
7. The method according to claim 2, characterized in that, The weighted summation of the pre-filling stage cost, the decoding stage cost, and the key-value cache transmission cost to obtain the estimated cost for each pre-filling side device includes: For each of the plurality of pre-filled side devices, the sum of the pre-filling stage cost, the decoding stage cost, and the key-value cache transmission cost corresponding to the combination of the pre-filled side device and each decoding side device is calculated to obtain the total cost of multiple combinations; The minimum value among the multiple combined total costs is selected as the estimated cost when allocating the inference request to be processed to the pre-filled side device for execution.
8. The method according to claim 1, characterized in that, The prediction of the number of output tokens corresponding to the reasoning request to be processed based on the historical reasoning request data includes: Extract the metadata or content features of the reasoning request to be processed; Based on the metadata or content features, determine the historical reasoning requests that match the reasoning request to be processed; Based on the actual number of output tokens in the matched historical reasoning requests, predict the number of output tokens corresponding to the reasoning request to be processed.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the model inference scheduling method as described in any one of claims 1 to 8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the model inference scheduling method as described in any one of claims 1 to 8.