Offline inference task processing methods, devices, electronic equipment and storage media

By detecting the remaining online task capacity and task type, and dynamically adjusting the allocation and interruption strategies for offline inference tasks, the problem of low GPU cluster resource utilization is solved, achieving efficient utilization of online inference task resources and low-cost management of offline tasks.

CN123114753APending Publication Date: 2026-11-17SUIYUAN INTELLIGENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611305937.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-26
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, GPU clusters experience a large amount of idle resources during off-peak periods of online inference tasks and low resource utilization for offline inference tasks, resulting in resource waste and low utilization.

Method used

By detecting the remaining online task capacity, offline inference tasks are allocated to matching target clusters based on task type and cluster performance constraints. When performance indicators are abnormal, the task with the lowest cost is interrupted based on computation completion, migration overhead, and remaining completion time to release resources and meet the resource requirements of online inference tasks.

Benefits of technology

It improves the resource utilization of online inference task clusters, avoids the independent cluster configuration of offline inference tasks, improves the overall utilization of graphics processor resources, and reduces the computational waste of offline inference tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114753A_ABST
    Figure CN123114753A_ABST
Patent Text Reader

Abstract

This invention discloses an offline inference task processing method, relating to the field of large language model inference and deployment. The method includes: obtaining the task type of each offline inference task; allocating each offline inference task to a matching target cluster based on cluster performance constraints according to the task type; wherein the target cluster includes a pre-filled cluster and a decoding cluster; if an anomaly is detected in the performance indicators of the target cluster, the target offline inference task with the lowest interruption cost is selected based on computation completion, migration overhead, and remaining completion time, so as to release the cluster resources occupied by the target offline inference task. The technical solution of this invention improves the resource utilization of the online inference task cluster while avoiding the need to configure a separate cluster for offline inference tasks; simultaneously, through low-cost offline inference task interruption, it meets the resource demands of surging online traffic while reducing the computational waste of offline inference tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of large language model reasoning and deployment, and in particular to an offline reasoning task processing method, apparatus, electronic device and storage medium. Background Technology

[0002] The reasoning process of a large language model includes two stages with distinct computational features: the prefill stage and the decoding stage. To avoid conflicts in hardware resource requirements between the different stages, the prefill-decode disaggregation architecture has become a commonly used system architecture for deploying large language models.

[0003] In existing technologies, production environments typically deploy online inference tasks (e.g., interactive chat, real-time question answering) and offline inference tasks (e.g., batch document summarization, data synthesis) on different independent clusters. The online task cluster reserves resources based on peak traffic, while the offline task cluster occupies graphics processing unit (GPU) resources separately. At the same time, in the PD separation architecture, the cluster used to perform pre-filling operations (i.e., the P cluster) is also deployed separately from the cluster used to perform decoding operations (i.e., the D cluster).

[0004] However, due to the significant fluctuations in online traffic during actual operation, the P and D clusters for online inference tasks will have a large amount of idle resources during off-peak periods, while the independent clusters for offline inference tasks will have a low overall resource utilization. Summary of the Invention

[0005] This invention provides an offline inference task processing method, apparatus, electronic device, storage medium, and computer program product to solve the problem of low resource utilization of GPU clusters.

[0006] According to another aspect of the present invention, an offline inference task processing method is provided, comprising: In response to detecting that the online task reserve of the pre-filled cluster is greater than a first preset threshold, or the online task reserve of the decoding cluster is greater than a second preset threshold, the task type of each offline inference task is obtained; Based on the task type of each offline inference task, and according to cluster performance constraints, each offline inference task is assigned to a matching target cluster; wherein, the target cluster includes a pre-filled cluster and a decoding cluster; If an anomaly is detected in the performance metrics of the target cluster, the target offline inference task with the lowest interruption cost is selected based on the computation completion rate, migration overhead, and remaining completion time, so as to release the cluster resources occupied by the target offline inference task.

[0007] According to another aspect of the present invention, an offline reasoning task processing apparatus is provided, comprising: The task type acquisition module is used to acquire the task type of each offline inference task in response to the detection that the online task reserve of the pre-filled cluster is greater than a first preset threshold, or the online task reserve of the decoding cluster is greater than a second preset threshold. The target cluster allocation module is used to allocate each offline inference task to a matching target cluster based on the task type of each offline inference task and cluster performance constraints; wherein, the target cluster includes a pre-filled cluster and a decoding cluster; The cluster resource management module is used to, if an abnormality is detected in the performance indicators of the target cluster, select the target offline inference task with the lowest interruption cost based on the computation completion rate, migration overhead, and remaining completion time, so as to release the cluster resources occupied by the target offline inference task.

[0008] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the offline inference task processing method described in any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the offline inference task processing method described in any embodiment of the present invention.

[0010] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the offline inference task processing method described in any embodiment of the present invention.

[0011] The technical solution of this invention, in response to detecting that the online task reserve of the pre-filled cluster is greater than a first preset threshold, or that the online task reserve of the decoding cluster is greater than a second preset threshold, obtains the task type of each offline inference task; according to the task type of each offline inference task, and based on cluster performance constraints, assigns each offline inference task to a matching target cluster; if an abnormality is detected in the performance indicators of the target cluster, the target offline inference task with the lowest interruption cost is obtained based on the computation completion rate, migration overhead, and remaining completion time, so as to release the cluster resources occupied by the target offline inference task. By allocating offline inference tasks to the idle time slots of the online inference task cluster, the resource utilization of the online inference task cluster is improved, and the configuration of a separate cluster for offline inference tasks is avoided, thus improving the overall utilization of graphics processor resources; at the same time, through low-cost offline inference task interruption, the resource demand during surges in online traffic is met, while reducing the computational waste of offline inference tasks.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of an offline reasoning task processing method provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of another offline reasoning task processing method provided in Embodiment 2 of the present invention; Figure 3 This is a flowchart of another offline reasoning task processing method provided in Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the structure of an offline reasoning task processing device according to Embodiment 4 of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device that implements the offline inference task processing method of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] Example 1 Figure 1 This is a flowchart of an offline inference task processing method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where an offline inference task is selectively interrupted when a GPU cluster detects a surge in online traffic. This method can be executed by an offline inference task processing device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes: S101. In response to detecting that the online task reserve of the pre-filled cluster is greater than a first preset threshold, or the online task reserve of the decoding cluster is greater than a second preset threshold, obtain the task type of each offline inference task.

[0018] In the large language model of the PD separation architecture, the graphics processor cluster (P cluster) used to perform prefill operations is deployed separately from the graphics processor cluster (D cluster) used to perform decoding operations. The P cluster is used to handle the prefill stage of an access request. The process is to process all the text input by the user at once, which is computationally intensive. Therefore, its load characteristics are that it requires a lot of computing power. The D cluster is used to handle the decoding stage of an access request. The process is to generate the answer character by character. It is memory bandwidth intensive. Therefore, its load characteristics are that it requires a lot of memory bandwidth.

[0019] In this embodiment of the invention, both the P cluster and the D cluster are used to execute online inference tasks. The online task capacity refers to the ability of the P cluster and the D cluster to handle offline inference tasks while meeting online traffic requirements. In this embodiment of the invention, the online task capacity of the P cluster is related to the waiting queue length, the utilization rate of the Streaming Multiprocessor (SM), the average input text length, the key-value cache generation speed, and the time to first token (TTFT). The online task capacity of the D cluster is related to the utilization rate of video memory bandwidth, the number of active requests, the total key-value cache usage, the key-value cache fragmentation rate, the time per token output (TPOT), and the remaining available video memory.

[0020] SM utilization equals the proportion of SM resources currently actively performing computations to the total SM resources; the waiting queue length refers to the number of requests that have arrived at the P cluster but have not yet been scheduled into the GPU to begin pre-filling computations; the average input text length refers to the average number of tokens input to each pre-filling request in the continuous batch processing framework; the key-value cache generation speed refers to the number of key-value cache tokens generated by the P cluster through pre-filling computations per unit time; the first token latency time refers to the time elapsed from when a request is sent from the external system to when the model returns the first output token. Based on the above related parameters, the online task capacity of the P cluster can be calculated and obtained through pre-configured computation rules or by querying the mapping table.

[0021] Memory bandwidth utilization refers to the percentage of the actual data throughput of the GPU's memory bus relative to the theoretical maximum throughput of the hardware; the number of active requests refers to the total number of inference sessions currently running on the D cluster that have not yet completed token generation; the total key-value cache usage refers to the total number of bytes occupied by the key-value cache in GPU memory for all active inference sessions; the key-value cache fragmentation rate refers to the ratio of unused but unallocated fragmented space to the total space allocated to the key-value cache in memory; the time taken to output each token refers to the time interval between two consecutive output tokens; and the remaining available memory refers to the size of unallocated bytes in GPU memory. Similarly, based on the above related parameters, the remaining online task capacity of the D cluster can be calculated and obtained through pre-configured calculation rules or by querying the mapping table.

[0022] Both the first and second preset thresholds can be configured to 0, so that when the P cluster and D cluster have the capacity to carry offline inference tasks, the offline inference tasks are allocated to the P cluster and / or D cluster to make full use of cluster resources. The first and second preset thresholds can also be configured to positive values, so that the P cluster and D cluster can carry offline inference tasks while maintaining a certain resource margin and avoiding resource overload caused by sudden traffic surges. Accordingly, when it is detected that the online task margin of the pre-filled cluster is greater than the first preset threshold, or the online task margin of the decoding cluster is greater than the second preset threshold, the task type of each offline inference task is obtained.

[0023] Offline inference tasks can be categorized into P-cluster affinity tasks, D-cluster affinity tasks, and hybrid tasks. P-cluster affinity tasks are characterized by a large number of input tokens, a small number of output tokens, high computational consumption, and low memory bandwidth requirements; examples include long text pre-filling tasks, batch vector embedding tasks, and batch input encoding tasks. D-cluster affinity tasks are characterized by continuous token generation, long-term occupation of key-value cache, and high memory bandwidth consumption; examples include long text summarizing tasks, offline translation tasks, and batch long text generation tasks. Hybrid tasks require the complete execution of a large model inference request; the P-cluster completes pre-filling and key-value cache generation, and then transfers the cache to the D-cluster for further decoding. Offline inference tasks can be classified using the basic metadata submitted with the task.

[0024] Optionally, in this embodiment of the invention, obtaining the task type of each offline inference task includes: obtaining the computing power requirement and bandwidth requirement of the offline inference task through a task prediction model; if it is determined that the computing power requirement of the offline inference task is greater than a first preset threshold and the bandwidth requirement of the offline inference task is less than a second preset threshold, determining that the task type of the offline inference task is a pre-filling task; if it is determined that the computing power requirement of the offline inference task is less than a third preset threshold and the bandwidth requirement of the offline inference task is greater than a fourth preset threshold, determining that the task type of the offline inference task is a decoding task; if it is determined that the computing power requirement of the offline inference task is greater than a fifth preset threshold and the bandwidth requirement of the offline inference task is greater than a sixth preset threshold, determining that the task type of the offline inference task is a hybrid task; wherein, the fifth preset threshold is less than or equal to the first preset threshold and greater than or equal to the third preset threshold; the sixth preset threshold is less than or equal to the fourth preset threshold and greater than or equal to the second preset threshold.

[0025] Specifically, the task prediction model can be a rule-fitting prediction model, such as a lightweight regression fitting model or a lookup table fitting model, or a lightweight machine learning model based on gradient boosting trees, such as a regression prediction model. The task prediction model can predict the computational power and bandwidth requirements corresponding to the offline inference task based on the input token length, maximum output token length, model size, and batch characteristics. Since hybrid tasks have higher computational power requirements, the required computational power threshold for hybrid tasks cannot be lower than that for D-cluster affinity tasks; that is, the fifth preset threshold must be greater than or equal to the third preset threshold. However, its computational power threshold is usually lower than that for P-cluster affinity tasks; that is, the fifth preset threshold must be less than or equal to the first preset threshold.

[0026] Because hybrid tasks also have high bandwidth requirements, their bandwidth threshold cannot be lower than that of P-cluster affinity tasks; that is, the sixth preset threshold must be greater than or equal to the second preset threshold. However, their bandwidth threshold is usually lower than that of D-cluster affinity tasks; that is, the sixth preset threshold must be less than or equal to the second preset threshold. This achieves refined automatic classification of offline inference tasks, eliminating reliance on manual labeling. It also matches the hardware load characteristics of the two types of clusters in the PD separation architecture, improving cluster resource matching. Furthermore, it enables effective identification of hybrid tasks, providing a decision-making basis for cross-cluster splitting and scheduling.

[0027] S102. Based on the task type of each offline inference task, assign each offline inference task to a matching target cluster according to cluster performance constraints; wherein, the target cluster includes a pre-filled cluster and a decoding cluster.

[0028] For P-cluster affinity tasks, they need to be assigned to the P cluster for processing, meaning their target cluster is the P cluster. For D-cluster affinity tasks, they need to be assigned to the D cluster for processing, meaning their target cluster is the D cluster. For hybrid tasks, they are split into P cluster execution parts (i.e., P cluster tasks) and D cluster execution parts (i.e., D cluster tasks), and then the split P cluster tasks are sent to the P clusters respectively, and the split D cluster tasks are sent to the D clusters.

[0029] For P clusters, TTFT is used as a performance constraint. When continuously assigning P cluster affinity tasks and P cluster tasks to P clusters for processing, the TTFT value of P clusters is monitored synchronously to prevent the TTFT value of P clusters from exceeding the preset TTFT threshold. For D clusters, TPOT is used as a performance constraint. When continuously assigning D cluster affinity tasks and D cluster tasks to D clusters for processing, the TPOT value of D clusters is monitored synchronously to prevent the TPOT value of D clusters from exceeding the preset TPOT threshold.

[0030] S103. If an abnormality is detected in the performance indicators of the target cluster, the target offline inference task with the lowest interruption cost is obtained based on the computation completion rate, migration overhead and remaining completion time, so as to release the cluster resources occupied by the target offline inference task.

[0031] When online user requests suddenly surge, the performance metrics of the target cluster will rise sharply. For example, the TTFT of cluster P will approach the preset TTFT threshold, and the TPOT of cluster D will approach the preset TPOT threshold. At this time, it is necessary to reclaim the GPU resources occupied by offline inference tasks and allocate them to online inference tasks. At this time, it is necessary to calculate the interruption cost of all running offline inference tasks and select the target offline inference task with the lowest interruption cost. The purpose is to prioritize offline inference tasks that have just started, have less cache, and have low redo costs, so as to minimize the waste of computing power of offline inference tasks.

[0032] Computation completion rate refers to the ratio of completed computations to the total computations. The closer a task is to completion, the more computations in the early stages will be wasted if interrupted, resulting in a greater loss of computing power and video memory bandwidth. Computation completion rate is positively correlated with interruption cost. Migration overhead refers to the time required to transfer and save the key-value cache. It is the ratio of the total amount of key-value cache data to the migration bandwidth. Migration bandwidth refers to the effective transmission bandwidth available when key-value cache data is transferred between storage media. Transfer and saving refers to the process of transferring from video memory to RAM and then to a fixed hard drive. Migration overhead is positively correlated with interruption cost. Remaining completion time is how much longer the offline inference task needs to complete the computation. The longer the remaining completion time, the longer it will take to recover after an interruption, resulting in greater overall computational overhead and higher interruption cost.

[0033] Based on this, the computational completion rate, migration overhead, and remaining completion time of each offline inference task are multiplied by their respective matched weights. The product results are then summed to obtain the interruption cost score. The offline inference task with the lowest interruption cost score is selected as the target offline inference task. This interrupts the execution of the target offline inference task, releasing the cluster resources it occupies and allocating those resources to online inference tasks to meet the online request needs of users.

[0034] The technical solution of this invention, in response to detecting that the online task reserve of the pre-filled cluster is greater than a first preset threshold, or that the online task reserve of the decoding cluster is greater than a second preset threshold, obtains the task type of each offline inference task; according to the task type of each offline inference task, and based on cluster performance constraints, assigns each offline inference task to a matching target cluster; if an abnormality is detected in the performance indicators of the target cluster, the target offline inference task with the lowest interruption cost is obtained based on the computation completion rate, migration overhead, and remaining completion time, so as to release the cluster resources occupied by the target offline inference task. By allocating offline inference tasks to the idle time slots of the online inference task cluster, the resource utilization of the online inference task cluster is improved, and the configuration of a separate cluster for offline inference tasks is avoided, thus improving the overall utilization of graphics processor resources; at the same time, through low-cost offline inference task interruption, the resource demand during surges in online traffic is met, while reducing the computational waste of offline inference tasks.

[0035] Example 2 Figure 2 This is a flowchart of an offline inference task processing method provided in Embodiment 2 of the present invention. The relationship between this embodiment and the above embodiments is that different interruption strategies are executed in the P cluster or the D cluster, such as... Figure 2 As shown, the method specifically includes: S201. In response to detecting that the online task reserve of the pre-filled cluster is greater than a first preset threshold, or the online task reserve of the decoding cluster is greater than a second preset threshold, obtain the task type of each offline inference task.

[0036] S202. Based on the task type of each offline inference task, each offline inference task is assigned to a matching target cluster according to cluster performance constraints; wherein, the target cluster includes a pre-filled cluster and a decoding cluster.

[0037] S203. If an abnormality is detected in the performance indicators of the target cluster, the target offline inference task with the lowest interruption cost is obtained based on the computation completion rate, migration overhead, and remaining completion time.

[0038] S204. If it is determined that the target cluster is a pre-filled cluster and the computation completion rate of the target offline inference task is greater than a preset ratio threshold, the cluster resources occupied by the target offline inference task are released after the current computation result is saved.

[0039] For the P cluster, interruption is performed at the pre-fill request level. Since the pre-fill request is a one-time calculation, resources can be freed up after the current pre-fill request is completed, or the current pre-fill request can be interrupted directly in an emergency. For interrupted offline inference tasks, if the calculation completion rate is high, that is, greater than the preset ratio threshold (e.g., 70%), the current calculation result is saved in memory or solid-state drive to avoid the loss of computing power caused by task interruption.

[0040] S205. If it is determined that the target cluster is a pre-filled cluster, and the computational completion rate of the target offline inference task is less than or equal to a preset ratio threshold, release the cluster resources occupied by the target offline inference task.

[0041] If the computation completion rate of the target offline inference task is low, i.e. less than or equal to a preset percentage threshold, the cluster resources occupied by the target offline inference task are released directly to avoid consuming storage resources. This differentiated interrupt handling logic, tailored to the load characteristics of the P cluster, not only adapts to the operating characteristics of computationally intensive tasks but also balances data storage overhead and redundant computation costs by processing intermediate computation results differently based on a preset percentage threshold, while ensuring that online inference tasks have priority access to resources.

[0042] S206. If the target cluster is determined to be a decoding cluster and the predicted anomaly time is less than or equal to the first time threshold, the calculation result of the target offline inference task is saved to the graphics processor's video memory.

[0043] If the predicted anomaly time is short, i.e. less than or equal to the first time threshold, a hot-save strategy is adopted to save the calculation results of the target offline inference task to the graphics processor's video memory. When the online traffic peak lasts for a short period of time, the overhead of copying data from video memory to main memory is eliminated, ensuring that GPU resources are released as soon as possible. At the same time, saving the calculation results to the graphics processor's video memory ensures that the offline inference task can quickly resume operation after the online inference task peak.

[0044] S207. If the target cluster is determined to be a decoding cluster, and the predicted anomaly time is greater than the first time threshold and less than the second time threshold, the calculation result of the target offline inference task is saved to the memory of the central processing unit.

[0045] If the predicted anomaly time is relatively long, i.e., the predicted anomaly time is greater than the first time threshold but less than the second time threshold, a warm storage strategy is adopted. In this case, the online inference task continuously occupies GPU resources for a relatively long period of time, but it is not necessary to occupy them for an extremely long time. The calculation results of the target offline inference task are moved out of the GPU's video memory, and the video memory is released to supply the online inference task. The read and write latency of the central processing unit (CPU) memory is much better than that of solid-state drives. When the task is restored later, the key-value cache can be copied back to the video memory relatively quickly, balancing the speed of video memory release and the performance of task restoration.

[0046] S208. If the target cluster is determined to be a decoding cluster and the predicted anomaly time is greater than or equal to the second time threshold, the calculation result of the target offline inference task is saved to the solid-state drive.

[0047] If the prediction anomaly time is too long, i.e. greater than or equal to the second time threshold, a cold storage strategy is adopted. At this time, the peak load of the online inference task lasts for a very long time, and the offline task cannot be recovered for a long time. Therefore, the GPU memory and CPU memory are completely released, and all resources are prioritized for the online inference task. At the same time, since the solid-state drive has a large capacity, it can carry a large number of key-value caches for offline inference tasks, thereby reducing the recovery speed by a small part in exchange for the complete release of high-speed storage resources.

[0048] The technical solution of this invention, with its differentiated interrupt handling logic for the load characteristics of the P cluster, not only adapts to the operating characteristics of computationally intensive tasks, but also balances data storage overhead and redundant computation costs by processing intermediate calculation results differently based on a preset ratio threshold, while ensuring that online inference tasks have priority access to resources. For the differentiated interrupt handling logic for the load characteristics of the D cluster, a three-level hierarchical storage of key-value cache is implemented, dynamically balancing recovery latency and storage resource consumption, while reducing the computational waste caused by repeated decoding calculations.

[0049] Example 3 Figure 3 This is a flowchart of an offline inference task processing method provided in Embodiment 3 of the present invention. The relationship between this embodiment and the above embodiments is that the hybrid task is distributed as a whole to a P cluster or a D cluster for processing, such as... Figure 3 As shown, the method specifically includes: S301. In response to detecting that the online task reserve of the pre-filled cluster is greater than a first preset threshold, or the online task reserve of the decoding cluster is greater than a second preset threshold, obtain the task type of each offline inference task.

[0050] S302. If the current offline inference task is a hybrid task, obtain the online task reserve ratio and load ratio; wherein, the online task reserve ratio is the ratio of the online task reserve of the pre-filled cluster to the online task reserve of the decoding cluster; the load ratio is the ratio of the load requirement of the pre-filling stage to the load requirement of the decoding stage of the current offline inference task.

[0051] The ratio of the online task reserve of the pre-filled cluster to the online task reserve of the decoding cluster is used as the online task reserve ratio; the ratio of the load demand of the pre-filled stage of the current offline inference task to the load demand of the decoding stage is used as the load ratio.

[0052] S303. If it is determined that the online task surplus allocation ratio is greater than the first allocation threshold, the current offline inference task is allocated to the pre-filled cluster based on the cluster performance constraints; wherein, the first allocation threshold is obtained based on the load allocation ratio, and the first allocation threshold is greater than the load allocation ratio.

[0053] The first allocation threshold is the product of the load allocation and the first multiple, where the first multiple is a value greater than 1. If the online task surplus allocation is greater than the first allocation threshold, it indicates that the idle computing power of cluster P is relatively sufficient, while the idle resources of cluster D are relatively scarce. Specifically, the abundance of relatively idle resources in the pre-filled cluster is significantly higher than the relative demand of the task for pre-filled computing power. In this case, after the performance constraint verification of cluster P passes, the hybrid task is allocated to the pre-filled cluster as a whole, and the pre-filling and decoding stages are executed completely. This fully utilizes the surplus computing power resources of the pre-filled cluster when the current resource structure is not suitable for cross-cluster splitting, avoiding pipeline bottlenecks, key-value cache accumulation, and cross-cluster transmission resource waste caused by insufficient surplus in cluster D.

[0054] S304. If it is determined that the online task surplus allocation ratio is less than the second allocation threshold, the current offline inference task is allocated to the decoding cluster based on the cluster performance constraints; wherein, the second allocation threshold is calculated based on the load allocation ratio, and the second allocation threshold is less than the load allocation ratio.

[0055] The second allocation threshold is the product of the load allocation and the second multiple, where the second multiple is a positive value less than 1. If the online task reserve allocation is less than the second allocation threshold, it indicates that the D cluster has sufficient idle memory resources, while the P cluster has limited computing power reserves. Specifically, the relative abundance of idle resources in the decoding cluster is significantly higher than the relative demand for decoding bandwidth of the task. In this case, after the performance constraint verification of the D cluster passes, the hybrid task is allocated entirely to the decoding cluster to fully execute the pre-filling and decoding stages. This leverages the large-capacity video memory and sufficient memory access bandwidth of the decoding cluster to support the entire process of the hybrid task, avoiding further deterioration of online pre-filling latency caused by deploying the task on the computationally strained P cluster.

[0056] S305. If an abnormality is detected in the performance indicators of the target cluster, the target offline inference task with the lowest interruption cost is obtained based on the computation completion rate, migration overhead, and remaining completion time, so as to release the cluster resources occupied by the target offline inference task.

[0057] Optionally, in this embodiment of the invention, after determining that the online task surplus allocation ratio is greater than a first allocation threshold, the method further includes: dividing the current offline inference task into pre-filling tasks and decoding tasks, and allocating the pre-filling tasks to a pre-filling cluster, and dividing the decoding tasks into multiple decoding sub-tasks, so as to allocate one or more decoding sub-tasks to the decoding cluster according to the online task surplus allocation ratio; after determining that the online task surplus allocation ratio is less than a second allocation threshold, the method further includes: dividing the current offline inference task into pre-filling tasks and decoding tasks, and allocating the decoding tasks to the decoding cluster, and dividing the pre-filling tasks into multiple pre-filling sub-tasks, so as to allocate one or more pre-filling sub-tasks to the pre-filling cluster according to the online task surplus allocation ratio.

[0058] Specifically, as described in the above technical solution, when the online task reserve ratio is greater than the first ratio threshold, the idle computing power of the P cluster is relatively sufficient, and the pre-filling task is directly allocated to the P cluster. However, the idle resources of the D cluster are relatively scarce and cannot bear the complete decoding load at once. At this time, the decoding task is divided into multiple decoding sub-tasks. Based on the online task reserve ratio, the load required by the pre-filling task is calculated by the ratio of the load required by the pre-filling task to the online task reserve ratio. Based on this load, one or more decoding sub-tasks are selected and allocated to the D cluster for execution. In this way, by dividing the decoding task into multiple sub-tasks and gradually deploying them according to the carrying capacity of the D cluster, the load injection rate of the D cluster is controlled, and the limited reserve of the D cluster is matched. This ensures that the key-value cache is continuously produced in the pre-filling stage, and the decoding sub-tasks are consumed in an orderly manner, preventing the accumulation of a large number of key-value caches from causing task blocking.

[0059] When the online task reserve ratio is less than the second ratio threshold, the idle computing power of cluster D is relatively sufficient, and the decoding task is directly assigned to cluster D. However, the idle resources of cluster P are relatively scarce and cannot bear the complete pre-filling load at once. At this time, the pre-filling task is divided into multiple pre-filling sub-tasks. Based on the online task reserve ratio, the load that the pre-filling cluster can bear is calculated by multiplying the load required by the decoding task by the online task reserve ratio. Based on this load, one or more pre-filling sub-tasks are selected and assigned to cluster P for execution. In this way, the pre-filling task is split into multiple sub-tasks, and rate limiting and batch scheduling are used to avoid a large number of pre-filling requests flooding into cluster P and raising the TTFT of the online inference task. This ensures that the pre-filling output rate matches the downstream decoding consumption rate and balances the throughput of the pipeline.

[0060] The technical solution of this invention, when the online task reserve ratio is greater than the first ratio threshold, fully utilizes the surplus computing resources of the pre-filled cluster to avoid pipeline bottlenecks, key-value cache accumulation, and wasted cross-cluster transmission resources in the decoding stage due to insufficient reserve in the D cluster. When the online task reserve ratio is less than the second ratio threshold, the large-capacity video memory and sufficient memory access bandwidth of the decoding cluster support the entire task process, avoiding further deterioration of online pre-filling latency caused by deploying tasks on the computationally strained P cluster.

[0061] Example 4 Figure 4 This is a structural block diagram of an offline inference task processing device provided in Embodiment 4 of the present invention. The device specifically includes: The task type acquisition module 401 is used to acquire the task type of each offline inference task in response to detecting that the online task reserve of the pre-filled cluster is greater than a first preset threshold, or the online task reserve of the decoding cluster is greater than a second preset threshold. The target cluster allocation module 402 is used to allocate each offline inference task to a matching target cluster based on the task type of each offline inference task and cluster performance constraints; wherein, the target cluster includes a pre-filled cluster and a decoding cluster; The cluster resource management module 403 is used to, if an abnormality is detected in the performance indicators of the target cluster, select the target offline inference task with the lowest interruption cost based on the computation completion rate, migration overhead and remaining completion time, so as to release the cluster resources occupied by the target offline inference task.

[0062] The technical solution of this invention, in response to detecting that the online task reserve of the pre-filled cluster is greater than a first preset threshold, or that the online task reserve of the decoding cluster is greater than a second preset threshold, obtains the task type of each offline inference task; according to the task type of each offline inference task, and based on cluster performance constraints, assigns each offline inference task to a matching target cluster; if an abnormality is detected in the performance indicators of the target cluster, the target offline inference task with the lowest interruption cost is obtained based on the computation completion rate, migration overhead, and remaining completion time, so as to release the cluster resources occupied by the target offline inference task. By allocating offline inference tasks to the idle time slots of the online inference task cluster, the resource utilization of the online inference task cluster is improved, and the configuration of a separate cluster for offline inference tasks is avoided, thus improving the overall utilization of graphics processor resources; at the same time, through low-cost offline inference task interruption, the resource demand during surges in online traffic is met, while reducing the computational waste of offline inference tasks.

[0063] Optionally, if the target cluster is determined to be a pre-filled cluster, the cluster resource management module 403 is specifically used to release the cluster resources occupied by the target offline inference task after saving the current calculation result if the computation completion rate of the target offline inference task is determined to be greater than a preset ratio threshold; and to release the cluster resources occupied by the target offline inference task if the computation completion rate of the target offline inference task is determined to be less than or equal to the preset ratio threshold.

[0064] Optionally, if the target cluster is determined to be a decoding cluster, the cluster resource management module 403 is specifically used to: if the predicted anomaly time is determined to be less than or equal to a first time threshold, save the calculation result of the target offline inference task to the graphics processor's video memory; if the predicted anomaly time is determined to be greater than the first time threshold and less than a second time threshold, save the calculation result of the target offline inference task to the central processing unit's memory; if the predicted anomaly time is determined to be greater than or equal to the second time threshold, save the calculation result of the target offline inference task to a solid-state drive.

[0065] Optionally, the target cluster allocation module 402 is specifically used to obtain the online task reserve ratio and load ratio if the current offline inference task is a mixed task; wherein, the online task reserve ratio is the ratio of the online task reserve of the pre-filled cluster to the online task reserve of the decoding cluster; the load ratio is the ratio of the load requirement of the pre-filling stage to the load requirement of the decoding stage of the current offline inference task; if it is determined that the online task reserve ratio is greater than a first ratio threshold, the current offline inference task is allocated to the pre-filled cluster based on cluster performance constraints; wherein, the first ratio threshold is calculated based on the load ratio and is greater than the load ratio; if it is determined that the online task reserve ratio is less than a second ratio threshold, the current offline inference task is allocated to the decoding cluster based on cluster performance constraints; wherein, the second ratio threshold is calculated based on the load ratio and is less than the load ratio.

[0066] Optionally, the target cluster allocation module 402 is further configured to divide the current offline inference task into pre-filling tasks and decoding tasks, allocate the pre-filling tasks to the pre-filling cluster, and divide the decoding tasks into multiple decoding sub-tasks, so as to allocate one or more decoding sub-tasks to the decoding cluster according to the online task surplus allocation ratio;

[0067] Optionally, the task type acquisition module 401 is specifically used to acquire the computing power requirement and bandwidth requirement of the offline inference task through a task prediction model; if it is determined that the computing power requirement of the offline inference task is greater than a first preset threshold and the bandwidth requirement of the offline inference task is less than a second preset threshold, the task type of the offline inference task is determined to be a pre-filling task; if it is determined that the computing power requirement of the offline inference task is less than a third preset threshold and the bandwidth requirement of the offline inference task is greater than a fourth preset threshold, the task type of the offline inference task is determined to be a decoding task; if it is determined that the computing power requirement of the offline inference task is greater than a fifth preset threshold and the bandwidth requirement of the offline inference task is greater than a sixth preset threshold, the task type of the offline inference task is determined to be a hybrid task; wherein, the fifth preset threshold is less than or equal to the first preset threshold and greater than or equal to the third preset threshold; the sixth preset threshold is less than or equal to the fourth preset threshold and greater than or equal to the second preset threshold.

[0068] The above-described apparatus can execute the offline inference task processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment can be found in the offline inference task processing method provided in any embodiment of the present invention.

[0069] Example 5 Figure 5 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, electronic devices, blade electronic devices, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0070] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0071] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0072] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as offline inference task processing methods.

[0073] In some embodiments, the offline inference task processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on a heterogeneous hardware accelerator via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the offline inference task processing method described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform the offline inference task processing method by any other suitable means (e.g., by means of firmware).

[0074] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0075] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0076] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0077] To provide user interaction, the systems and techniques described herein can be implemented on a heterogeneous hardware accelerator, which includes: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the heterogeneous hardware accelerator. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback); and input from the user can be received in any form (including sound input, voice input, or haptic input).

[0078] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0079] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0080] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0081] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An offline inference task processing method, characterized in that, include: In response to detecting that the online task reserve of the pre-filled cluster is greater than a first preset threshold, or the online task reserve of the decoding cluster is greater than a second preset threshold, the task type of each offline inference task is obtained; Based on the task type of each offline inference task, and according to cluster performance constraints, each offline inference task is assigned to a matching target cluster; wherein, the target cluster includes a pre-filled cluster and a decoding cluster; If an anomaly is detected in the performance metrics of the target cluster, the target offline inference task with the lowest interruption cost is selected based on the computation completion rate, migration overhead, and remaining completion time, so as to release the cluster resources occupied by the target offline inference task.

2. The offline reasoning task processing method according to claim 1, characterized in that, If the target cluster is determined to be a pre-populated cluster, releasing the cluster resources occupied by the target offline inference task includes: If it is determined that the computation completion rate of the target offline inference task is greater than a preset ratio threshold, the cluster resources occupied by the target offline inference task are released after the current computation result is saved. If the computational completion rate of the target offline inference task is determined to be less than or equal to a preset percentage threshold, the cluster resources occupied by the target offline inference task are released.

3. The offline reasoning task processing method according to claim 1, characterized in that, If the target cluster is determined to be a decoding cluster, releasing the cluster resources occupied by the target offline inference task includes: If the predicted anomaly time is determined to be less than or equal to the first time threshold, the calculation result of the target offline inference task is saved to the graphics processor's video memory. If it is determined that the predicted anomaly time is greater than the first time threshold and less than the second time threshold, the calculation result of the target offline inference task is saved to the memory of the central processing unit. If the predicted anomaly time is determined to be greater than or equal to the second time threshold, the calculation result of the target offline inference task is saved to the solid-state drive.

4. The offline reasoning task processing method according to claim 1, characterized in that, The step of allocating each offline inference task to a matching target cluster based on the task type of each offline inference task and cluster performance constraints includes: If the current offline inference task is a mixed task, obtain the online task reserve ratio and load ratio; wherein, the online task reserve ratio is the ratio of the online task reserve of the pre-filled cluster to the online task reserve of the decoding cluster; the load ratio is the ratio of the load requirement of the pre-filling stage to the load requirement of the decoding stage of the current offline inference task. If it is determined that the online task reserve ratio is greater than the first ratio threshold, the current offline inference task is allocated to the pre-filled cluster based on the cluster performance constraints; wherein, the first ratio threshold is obtained based on the load ratio calculation, and the first ratio threshold is greater than the load ratio; If it is determined that the online task reserve ratio is less than the second ratio threshold, the current offline inference task is allocated to the decoding cluster based on the cluster performance constraints; wherein, the second ratio threshold is calculated based on the load ratio, and the second ratio threshold is less than the load ratio.

5. The offline reasoning task processing method according to claim 4, characterized in that, After determining that the online task reserve ratio is greater than the first ratio threshold, the method further includes: The current offline inference task is divided into pre-filling tasks and decoding tasks. The pre-filling tasks are assigned to the pre-filling cluster, and the decoding tasks are divided into multiple decoding sub-tasks. One or more decoding sub-tasks are assigned to the decoding cluster according to the online task surplus allocation ratio. After determining that the online task surplus allocation ratio is less than the second allocation threshold, the method further includes: The current offline inference task is divided into pre-filling tasks and decoding tasks. Decoding tasks are assigned to the decoding cluster, and pre-filling tasks are divided into multiple pre-filling sub-tasks. One or more pre-filling sub-tasks are then assigned to the pre-filling cluster according to the online task surplus allocation ratio.

6. The offline reasoning task processing method according to claim 1, characterized in that, The task types for obtaining each offline inference task include: The computing power and bandwidth requirements of the offline inference task are obtained through the task prediction model. If it is determined that the computing power requirement of the offline inference task is greater than the first preset threshold and the bandwidth requirement of the offline inference task is less than the second preset threshold, the task type of the offline inference task is determined to be a pre-filling task. If it is determined that the computing power requirement of the offline inference task is less than the third preset threshold and the bandwidth requirement of the offline inference task is greater than the fourth preset threshold, the task type of the offline inference task is determined to be a decoding task. If it is determined that the computing power requirement of the offline inference task is greater than a fifth preset threshold and the bandwidth requirement of the offline inference task is greater than a sixth preset threshold, the task type of the offline inference task is determined to be a hybrid task; wherein, the fifth preset threshold is less than or equal to the first preset threshold and greater than or equal to the third preset threshold; the sixth preset threshold is less than or equal to the fourth preset threshold and greater than or equal to the second preset threshold.

7. An offline reasoning task processing device, characterized in that, include: The task type acquisition module is used to acquire the task type of each offline inference task in response to the detection that the online task reserve of the pre-filled cluster is greater than a first preset threshold, or the online task reserve of the decoding cluster is greater than a second preset threshold. The target cluster allocation module is used to allocate each offline inference task to a matching target cluster based on the task type of each offline inference task and cluster performance constraints; wherein, the target cluster includes a pre-filled cluster and a decoding cluster; The cluster resource management module is used to, if an abnormality is detected in the performance indicators of the target cluster, select the target offline inference task with the lowest interruption cost based on the computation completion rate, migration overhead, and remaining completion time, so as to release the cluster resources occupied by the target offline inference task.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that is executed by the at least one processor, such that the at least one processor is able to perform the offline reasoning task processing method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the offline inference task processing method according to any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the offline inference task processing method according to any one of claims 1-6.