Cross-layer collaborative scheduling method and system for multi-model reasoning service

Abstract

The invention provides a multi-model reasoning service-oriented cross-layer collaborative scheduling method, which comprises the following steps of: firstly, determining the number of sub-graphs and an initial division point, then iteratively optimizing sub-graph segmentation by using a coordinate descent method, and selecting an optimal scheme in combination with a historical request weight and an actual time delay. Then, the sub-graph is divided into sub-graph blocks in a recursive mode, parallel scheduling schemes of different batch processing sizes are generated, and online candidate strategies are screened out; and meanwhile, new establishment, stretching and splitting operations are configured to realize dynamic batch processing, so that an optimal parallel strategy is scheduled and selected on line based on the service quality. The invention further provides a multi-model reasoning service-oriented cross-layer collaborative scheduling system, a storage medium and computer equipment. Therefore, the online model service performance can be improved, the computing resource utilization rate can be improved, online scheduling and offline optimization cooperative scheduling can be realized, and the service performance can be greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a cross-layer cooperative scheduling method and system for multi-model inference services, a storage medium and a computer device. BACKGROUND

[0002] With the rapid development of artificial intelligence, deep learning models have been widely applied in image recognition, speech processing, natural language understanding and other fields. In order to meet the needs of real-time and response speed of intelligent applications, more and more deep learning models are deployed on high-performance servers to provide inference capabilities in a service-oriented manner. These applications usually provide capabilities in the form of services, and users can obtain inference results by sending requests. With the continuous rise in application complexity, the complexity of model structure is continuously increasing. At the same time, a single model cannot meet the diversified intelligent task requirements, and more and more systems begin to introduce multi-DNN inference to realize composite functions. For example, in augmented reality and virtual reality applications, a model may be needed to complete face detection, another model to complete head pose estimation, and a third model to perform image semantic understanding; in a life log system, objects, scenes and behaviors in an image need to be analyzed simultaneously, and each task often corresponds to an independent neural network model. By connecting or running multiple models in parallel, more powerful intelligent capabilities can be achieved, and the scalability and modularity of the system are improved. Single-model inference can be regarded as a special case of multi-model inference, and multi-model inference has become the mainstream technology development direction in intelligent inference systems.

[0003] A multi-model inference service system mainly includes three levels. The service framework level, the inference system level and the operator optimization level.

[0004] The service framework level is mainly responsible for sending and receiving user requests and scheduling; this layer is usually responsible for interacting with the internal and external environment of the system; including receiving, parsing, request queue management and other basic functions. At the same time, the existing framework will also flexibly implement a batch processing mechanism on this basis, dynamically adjust the batch of requests according to the current hardware usage status while ensuring the stability of security functions, improve the hardware resource utilization rate of model services by scheduling requests, and improve the overall throughput of the system and reduce request delay.

[0005] The inference system layer focuses on model execution optimization, which is the bridge connecting service logic and underlying computing resources. At this layer, the system needs to handle the concurrent scheduling problem of multi-model tasks, reasonably arrange the order of model loading, memory allocation, and execution queuing, and avoid resource conflicts and redundant calculations. For example, multiple models may share hardware resources in space, and the parallelism between operators can be reasonably planned by scheduling operators within subgraphs. Some current high-performance inference engines have begun to introduce task graph scheduling, phased execution, and cross-model collaboration mechanisms to improve the throughput and parallelism of the system.

[0006] The operator optimization layer is the foundation of performance improvement for the entire system, involving low-level computation optimization within deep learning models. Each deep neural network is composed of several operators, such as convolution, normalization, and matrix multiplication. Optimization at the operator level directly affects the latency and resource consumption of inference. In the multi-model scenario, different models may use different architectures and input dimensions, and operator behavior is highly dynamic. Operator optimization usually includes fusion, intra-operator parallelization, and static scheduling at compile time. In recent years, some hardware-aware compilers, such as TVM, TensorRT, and XLA, have been integrated into inference systems to provide optimal execution paths for different devices. Especially in multi-model tasks, by scheduling multiple operator execution processes uniformly, reducing intermediate data copying, and dynamically adjusting thread binding strategies, the overall inference performance can be effectively improved.

[0007] Currently, there is no unified deep neural network service optimization system in academia and industry. Traditional service systems represented by NVIDIA Triton, Clipper, and TensorFlow Serving use fixed time windows and maximum batch size request collection mechanisms, which are difficult to dynamically adapt to online load changes and lack flexibility. Existing optimization techniques mainly have the following problems:

[0008] (1) The batch processing strategy of existing inference service systems has optimization blind spots. Although systems such as LazyBatch and DVAbatch use multi-subgraph architecture to achieve dynamic batch processing adjustment, they are still limited to model-level optimization and cannot go deep into operator and subgraph-level fine-grained scheduling. The execution of serial operators in the model is highly uneven in terms of hardware resource utilization. Calculation-intensive operators can make more efficient use of hardware resources with the help of service systems. However, operators with low resource requirements will waste hardware resources if executed alone. This method can only improve system throughput by adjusting the batch size at runtime, but it ignores the diversity of resource requirements caused by the diversity of operators, and still has the problem of insufficient utilization of computing resources. Especially when facing multi-model mixed deployment scenarios, existing batch processing strategies cannot achieve cross-model operator-level resource allocation.

[0009] (2) Current lack of task scheduling and operator optimization collaborative design mechanism. Although frameworks such as TensorRT, TVM have made significant progress in operator fusion and parallel computing (such as TASO expanding the optimization space through mathematical equivalent transformation, and Nimble using multi-stream scheduling to improve parallelism), these optimizations are mainly for offline scenarios. In the online service environment, there is no unified framework to combine real-time task scheduling and underlying operator optimization, making it difficult for the system to achieve the optimal balance between service delay and resource utilization. Although recent research has attempted to improve multi-model inference efficiency through model fusion (such as AdaShare) or hardware acceleration (such as dedicated DNN accelerators), these solutions either rely on specific model structures or require custom hardware support, lacking universality. SUMMARY

[0010] To overcome the above-mentioned defects, the purpose of the present application is to provide a cross-layer collaborative scheduling method and system for multi-model inference service, which can realize online scheduling and offline optimization collaborative scheduling and greatly improve service performance.

[0011] To solve the above technical problems, on the one hand, the present application provides a cross-layer collaborative scheduling method for multi-model inference service, comprising the steps of:

[0012] determining the number of subgraphs that the multi-model structure needs to be divided into according to system requirements, and setting initial division points in each sub-model;

[0013] using the coordinate descent method to iteratively optimize the division points for each sub-model, fixing the optimal division points of other sub-models in each iteration optimization, and randomly generating multiple division methods for the current sub-model; according to the actual execution delay of the subgraph under a certain batch, and combining the weight generated by the historical request data to perform weighted calculation on the performance of the division method, the final subgraph division scheme is selected for subgraph division;

[0014] For each subgraph after switching, the subgraph is recursively divided into multiple subgraph blocks while maintaining the inter-operator dependency relationship; the recursive method is used to decompose the subgraph parallel optimization problem into a subgraph block parallel optimization sub-problem, all legal subgraph blocks are enumerated and the optimal solution is calculated to obtain the parallel scheduling scheme of the subgraph under different batch sizes;

[0015] record the parallel scheduling scheme, and based on the strategy similarity analysis, select a fixed number of target strategies as online candidate strategies;

[0016] configure task-level operations for implementing dynamic batching, the task-level operations including new operation, stretching operation and splitting operation, the new operation is to create a new batch for collecting online requests, the stretching operation is to add new requests to the existing batch, and the splitting operation is to split a large batch into multiple small batches;

[0017] The task-level operation is used to process the request online based on the service quality-aware scheduling strategy, and an optimal parallel scheduling scheme is selected from the online candidate strategy according to the current batch size.

[0018] Optionally, the task-level operation is used to process the request online based on the service quality-aware scheduling strategy, and an optimal parallel scheduling scheme is selected from the online candidate strategy according to the current batch size.

[0019] The scheduling effect is evaluated based on the service quality target, and if the additional waiting time of the stretching operation does not exceed the upper limit of the delay allowed by the service quality, the stretching operation is performed; if the total delay of the splitting operation does not exceed the original batch delay, the splitting operation is performed.

[0020] Optionally, the strategy similarity analysis is used to select a fixed number of target strategies as online candidate strategies.

[0021] The similarity of different parallel scheduling schemes is calculated, and a fixed number of target strategies are selected as online candidate strategies based on the similarity.

[0022] Optionally, each subgraph after switching is recursively divided into multiple subgraph blocks while maintaining the inter-operator dependency relationship.

[0023] The subgraph is recursively divided into subgraph blocks by introducing multiple groups of hardware synchronization events.

[0024] Optionally, the new operation specifically is: when the system receives a new request and there is no batch waiting, a preset number of online requests are collected to create a new batch;

[0025] The stretching operation specifically is: when the existing batch has not started execution or the execution is not completed, the newly arrived request is added to the batch, and the number of new requests does not exceed the maximum processing capacity of the subgraph corresponding to the batch;

[0026] The splitting operation specifically is: a large batch that is being executed or waiting to be executed is split into at least two small batches, and the size of each small batch is not less than the minimum batch threshold set by the system.

[0027] Optionally, the number of subgraphs is determined to satisfy the following conditions:

[0028] The internal division points of each sub-module are consistent in order;

[0029] Each subgraph is composed of model calculation blocks of different sub-modules.

[0030] On the other hand, based on the same invention, the present application also provides a cross-layer collaborative scheduling system for a multi-model inference service for implementing the method as described above, which comprises:

[0031] An initialization module is configured to determine the number of subgraphs into which the multi-model structure needs to be divided according to system requirements, and set initial division points in each sub-model;

[0032] A subgraph division module is configured to perform iterative optimization of the division points one by one for the sub-models by using a coordinate descent method, fix the optimal division points of other sub-models in each iteration, randomly generate multiple division manners for the current sub-model, perform weighted calculation on the performance of the division manners according to the actual execution time delay of the subgraph under a specific batch and the weight generated based on historical request data, and select a final subgraph division scheme for subgraph division;

[0033] A parallel strategy optimization module is configured to divide each subgraph after switching into multiple subgraph blocks while maintaining the inter-operator dependency relationship, decompose the subgraph parallel optimization problem into subgraph block parallel optimization sub-problems in a recursive manner, enumerate all legal subgraph blocks and calculate the optimal solution, and obtain the parallel scheduling scheme of the subgraph under different batch sizes;

[0034] A record and screening module is configured to record the parallel scheduling scheme, and screen a fixed number of target strategies as online candidate strategies based on strategy similarity analysis;

[0035] A dynamic batch processing module is configured to configure task-level operations for implementing dynamic batch processing, the task-level operations including a new operation, a stretching operation and a splitting operation, the new operation being to create a new batch for collecting online requests, the stretching operation being to add a new request to an existing batch, and the splitting operation being to split a large batch into multiple small batches;

[0036] An online collaborative scheduling module is configured to schedule strategies based on quality of service awareness, process requests online through the task-level operations, and select an optimal parallel scheduling scheme from the online candidate strategies according to the size of the current batch.

[0037] Optionally, the subgraph division module further includes a historical data processing unit configured to statistically analyze the frequency of different input sizes in the historical request data, and generate a weight coefficient for weighted performance evaluation of subgraph division.

[0038] In addition, a storage medium and a computer device are also provided, the storage medium being configured to store a computer program for executing the cross-layer collaborative scheduling method for multi-model inference services.

[0039] The computer device includes a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, and the processor implements the cross-layer collaborative scheduling method for multi-model inference services when executing the computer program.

[0040] The cross-layer collaborative scheduling method for multi-model inference services provided by the application cuts the multi-model computation graph into a fixed number of subgraphs by using the coordinate descent method based on historical request data in the offline stage, generates operator-level parallel strategies of different batch sizes for each subgraph through dynamic programming, and filters out a representative strategy set through similarity analysis. In the online stage, the request batch is dynamically adjusted through three operations of new creation, stretching, and splitting: stretching is triggered when the throughput improvement of the merged batch meets the QoS target, splitting is performed when the total delay after splitting is less than or equal to the original delay, and the optimization strategy is dynamically switched according to the real-time batch size at the subgraph boundary. In this way, the application has the following beneficial effects: operator-level fine-grained scheduling improves hardware utilization; historical data-driven subgraph cutting adapts to multi-size input; dynamic batch processing and strategy switching collaboration improve system throughput while ensuring single-request delay within the QoS range. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The structural schematic diagram of the cross-layer collaborative scheduling system for multi-model inference services provided by another embodiment of the application;

[0042] Figure 2 The working principle schematic diagram of the system offline stage of the cross-layer collaborative scheduling method for multi-model inference services described by the application;

[0043] Figure 3 The working principle schematic diagram of the online scheduling stage of the cross-layer collaborative scheduling method for multi-model inference services described by the application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the application clearer and more apparent, the application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0045] It should be noted that the references to "one embodiment", "an embodiment", "example embodiment" and the like in the present specification indicate that the described embodiment can include a particular feature, structure or characteristic, but not every embodiment must necessarily include the particular feature, structure or characteristic. In addition, such expressions are not necessarily referring to the same embodiment. Further, when a particular feature, structure or characteristic is described in connection with an embodiment, it is indicated that such a feature, structure or characteristic is incorporated into other embodiments within the knowledge of those skilled in the art, whether or not it is explicitly described.

[0046] Furthermore, certain terms used in the specification and following claims are to be construed in accordance with their general usage, unless a different meaning is clearly intended. The description and the following claims are not limited to the described embodiments, but can be implemented in various ways. The terminology used in the description is for the purpose of describing the particular versions of the application only and is not intended to limit the scope of the application. The singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises", "comprising", "includes", "including" and the like are to be construed in an open-ended way, meaning that they are used to include, but not to limit, the objects or options as described. The terms "connected" and "coupled" are to be construed in an open-ended way, meaning that they are used to include, but not to limit, the objects or options as described. The terms "connected" and "coupled" are to be construed in an open-ended way, meaning that they are used to include, but not to limit, the objects or options as described.

[0047] In the existing inference service system, the scheduling framework and the inference engine are generally optimized separately. The engine only cares about offline inference performance and cannot perceive the characteristics of online requests. The online scheduling framework can only complete its own scheduling logic and ignores the engine. This leads to low utilization of GPU resources in the system, and the system is difficult to complete a large number of user requests within a specified time. The above problems bring great challenges to the design of a new multi-model inference service system.

[0048] In view of the problems existing in the prior art, the present application provides a cross-layer cooperative scheduling method for multi-model inference services, the technical concept of which is: through the joint design and optimization of online inference framework and offline inference optimization. Offline, by combining the historical request law, guiding the offline inference optimizer to dynamically split the multi-model computation graph, and optimizing the online frequently used operator parallel mode in each subgraph. In online inference, the online scheduler collects online requests, merges or separates the batch size in advance before entering each subgraph calculation, dynamically adjusts the batch size, selects the best performance optimization configuration of the subgraph under the batch size, and switches online, so as to obtain the best online inference performance, further improve the system throughput, and improve the request service quality.

[0049] The specific principle of the cross-layer cooperative scheduling method for multi-model inference services of the present application will be described below in conjunction with specific embodiments.

[0050] The cross-layer cooperative scheduling method for multi-model inference services provided by an embodiment of the present application includes an offline stage and an online stage, wherein the offline stage is step S101-S104, and the online stage is step S105-S106; the specific steps are as follows:

[0051] S101: Determine the number of subgraphs into which the multi-model structure needs to be divided based on system requirements, and set initial partitioning points in each sub-model. The validity of the partitioning scheme is ensured through these initial partitioning points. The determination of the number of subgraphs satisfies the following conditions: the partitioning point order within each sub-module is consistent; each subgraph is composed of model computation blocks from different sub-modules.

[0052] S102: Using the coordinate descent method, the partitioning points of each sub-model are iteratively optimized. In each iteration, the optimal partitioning points of other sub-models are fixed, and multiple partitioning methods are randomly generated for the current sub-model. Based on the actual execution latency of the subgraph in a specific batch, and combined with the weights generated from historical request data, the performance of the partitioning methods is weighted and calculated to select the final subgraph partitioning scheme. In other words, the partitioning points of each sub-model are iteratively optimized, the optimal partitioning points of other sub-models are fixed, multiple partitioning methods are randomly generated, and the importance of different input sizes is weighted and evaluated based on historical request data to select the best-performing partitioning method.

[0053] After obtaining the multi-model structure required for the service, in order to meet the requirements of a multi-stage service system, the multi-model inference needs to be divided into a fixed number of subgraphs. However, due to its complex computational graph structure, accurately finding the optimal partition that satisfies the performance requirements for different input sizes within a huge search space is not a simple task. Enumeration methods consume a lot of time, while random methods struggle to find the best-performing partitioning method.

[0054] To achieve rapid searching for the optimal segmentation method, this embodiment adopts the following approach: First, based on the system's preset number of subgraphs, corresponding partitioning points are set in each sub-model. A valid partitioning scheme must ensure the consistency of the order of partitioning points within each sub-model. After partitioning, the original model is decomposed into several subgraphs, each composed of model computation blocks from different sub-models. Since multi-model inference has a multi-branch characteristic, one branch can be selected each time while the others are fixed. For the selected branch, multiple partitioning methods are randomly generated, and the specific latency for each input size is measured. Furthermore, the importance of different input sizes is differentiated based on historical data, and the performance of each partitioning method is weighted and calculated. The best-performing partitioning method is selected. This process is repeated for each remaining branch, and finally, a subgraph partitioning method with near-optimal performance is obtained within a limited number of experiments.

[0055] Specifically, firstly, based on the system's preset number of subgraphs, corresponding partitioning points are set in each sub-model; a valid partitioning scheme must ensure the consistency of the order of partitioning points within each sub-model. Through the partitioning scheme, the original model is decomposed into several subgraphs, each subgraph being composed of model computation blocks from different sub-models; the task of executing the subgraph in a specific batch size is used as the basic evaluation unit, and the optimization goal is to maximize the overall performance improvement of the system by adjusting the partitioning scheme.

[0056] In an optional implementation, the performance evaluation employs a weighted comprehensive evaluation method: first, the time ratio of sequential execution to parallel execution under different batch sizes is calculated; then, a weighted sum is performed using weight coefficients generated from historical request distributions. Thus, compared to theoretical modeling, this example directly uses actual execution latency for measurement, ensuring that the evaluation results more closely reflect the real-world operating environment.

[0057] To address the issue of an excessively large search space for partitioning schemes, this scheme employs a coordinate descent search strategy: iteratively optimizing the partitioning points for each sub-model, fixing the optimal partitioning of other sub-models each time, and gradually approximating the global optimum by randomly sampling candidate schemes and evaluating their performance. To improve search efficiency, a greedy scheduling strategy is used in the initial stage to quickly estimate the parallel execution cost. The overall design is as follows: Figure 2 As shown, a multi-model structure is input into the system. First, in the offline part, based on historical information, subgraph tasks are optimized and partitioned. The optimal subgraph segmentation for online operation is obtained using coordinate descent. Each subgraph is then input into a parallel optimization module within the subgraph. Operator parallelism is tuned for different input sizes for each subgraph, and multiple parallel configurations are output for each subgraph to cope with dynamic changes in online service load. This improves the performance of the online model service: through detailed analysis of historical data, the dynamic range of online tasks is analyzed; based on computational speed, the coordinate descent method searches for the subgraph partitioning strategy with the best overall online performance, which can improve the online service performance by approximately 10%.

[0058] S103: For each subgraph after the switch, recursively divide it into multiple subgraph blocks while maintaining the dependencies between operators; decompose the subgraph parallel optimization problem into subproblems of subgraph block parallel optimization using a recursive approach, enumerate all legal subgraph blocks and calculate the optimal solution to obtain the parallel scheduling scheme of the subgraph under different batch sizes. Specifically, recursively dividing each subgraph after the switch into multiple subgraph blocks while maintaining the dependencies between operators includes: introducing multiple sets of hardware synchronization events to recursively divide the subgraph into subgraph blocks.

[0059] After obtaining an optimized subgraph partitioning scheme, the system generator first explores parallelization strategies between operators for each subgraph, and then selects candidate strategies suitable for the online phase.

[0060] In its implementation, inspired by the IOS (a static operator scheduling technique for single-model inference) method, this embodiment employs a dynamic programming-based inter-operator parallel strategy generation method. This method considers the dynamic characteristics of different batch sizes, recursively dividing the subgraph into multiple subgraph blocks by introducing multiple sets of hardware synchronization events, while maintaining the dependencies between operators. The original problem is decomposed into subproblems recursively, and the optimal solution is found by enumerating all legal subgraph blocks, recording the optimal scheduling scheme for different batch sizes. Unlike IOS, this embodiment also considers the dynamic batch size to find the optimal scheduling scheme, adapting to the highly dynamic nature of multi-model inference services.

[0061] S104: Record the parallel scheduling scheme and, based on policy similarity analysis, select a fixed number of target policies as online candidate policies.

[0062] Furthermore, the step of selecting a fixed number of target strategies as online candidate strategies based on strategy similarity analysis includes: calculating the similarity of different parallel scheduling schemes, and selecting a fixed number of target strategies as online candidate strategies based on the similarity.

[0063] The system selects several representative candidate strategies for online deployment, reducing runtime overhead while ensuring performance. Analysis of the generated strategies reveals a high degree of similarity between optimal scheduling schemes for different batch sizes. These schemes can be grouped and share a unified scheduling scheme with only a minor performance penalty. Therefore, this embodiment uses a weighted evaluation method to select the optimal strategy set from all possible strategy combinations based on strategy similarity analysis, and stores it in a strategy file for online scheduling. This evaluation process utilizes pre-computed and cached results to achieve rapid strategy selection. For example, it analyzes the structural similarity of optimal scheduling schemes for different batch sizes; groups similar strategies and shares a unified scheduling scheme; and selects a fixed number of representative strategy sets from all strategy combinations through weighted evaluation. Thus, this embodiment optimizes the parallelism between operators for subgraphs with specific input sizes, improving the utilization of hardware resources.

[0064] S105: Configure task-level operations for implementing dynamic batch processing. These operations include new, stretch, and split operations. The new operation collects online requests to create a new batch; the stretch operation adds new requests to an existing batch; and the split operation divides a large batch into multiple smaller batches. Specifically, the system implements three task-level operations—New, Stretch, and Split—to support flexible dynamic batch processing and integrates these operations into a task-operator collaborative scheduling mechanism. The stretch operation specifically adds newly arrived requests to an existing batch before it has started or finished executing, ensuring that the number of new requests does not exceed the maximum processing capacity of the subgraph corresponding to that batch. The split operation specifically divides a large batch that is currently executing or waiting to be executed into at least two smaller batches, with each smaller batch being no smaller than the minimum batch threshold set by the system.

[0065] S106: Based on the quality-of-service (QoS)-aware scheduling strategy, requests are processed online through the task-level operations, and the optimal parallel scheduling scheme is selected from the online candidate strategies according to the current batch size. In specific implementation, a QoS-aware scheduling strategy is adopted, taking into account changes in batch size and the performance of operator-level parallelization strategies corresponding to different task scheduling operations. By evaluating throughput improvement and latency reduction, the task scheduling strategy is dynamically adjusted to achieve efficient multi-model inference service.

[0066] The service quality-aware scheduling strategy, which processes requests online through task-level operations, includes: evaluating the scheduling effect based on service quality targets; if the additional waiting time of the stretching operation does not exceed the service quality-allowed delay limit, then the stretching operation is executed; if the total delay of the splitting operation does not exceed the original batch delay, then the splitting operation is executed.

[0067] In the task and operator collaborative scheduling section, this embodiment references the existing DVABatch design, incorporating three task-level operations—New, Stretch, and Split—to support flexible dynamic batch processing, and combining them with a task-operator collaborative scheduling mechanism. The New operation is a basic operation that creates a new batch by collecting online requests. Newly arriving requests are organized into a new batch through the New operation. The Stretch operation adds additional requests to the current batch, while the Split operation divides the ongoing batch into multiple independent batches. Because the subgraph tasks generated by these operations have dynamic batch sizes, the system dynamically adjusts the operator-level scheduling according to a pre-determined optimal parallelization strategy.

[0068] To support precise scheduling decisions, the system employs a service quality-aware scheduling strategy, considering variations in batch size and the performance of operator-level parallelization strategies for different task scheduling operations. Through new batch creation, online requests from different periods are aggregated into batch inputs for multi-model inference tasks. When a new batch arrives, the system evaluates the throughput improvement resulting from merging it into the current batch. If the throughput improvement after merging meets the service quality target, a stretching operation is performed. While this operation increases the inference latency of the previous batch, it improves the overall throughput. Therefore, the latency introduced by the stretching operation is guaranteed to be within the service quality target range.

[0069] Splitting involves dividing a batch into two smaller batches. The system estimates the reduction in average latency, and if the latency of the split tasks meets a certain condition, the split operation is performed. This operation allows one batch to complete earlier but introduces additional waiting time for the other batch. Therefore, the system only performs a split operation when the sum of the split latencies is less than or equal to the original latency, thereby reducing the average inference latency of tasks without reducing system throughput. Through these strategies, the system provides an efficient multi-model inference service system, supporting effective collaborative scheduling of tasks and operators. See also Figure 3 This invention demonstrates how the present invention dynamically utilizes three task operations to achieve optimal online service performance. When requests continuously arrive, the system combines multiple requests into new batches. During the processing of these batches, subsequent newly added batches can be integrated for a stretching operation, increasing system resource utilization. If a large batch takes too long to process in the queue, the system dynamically splits the large batch to ensure that some requests meet the service's QoS, achieving optimal resource allocation.

[0070] The online service system in this embodiment can dynamically select the parallel optimization method between operators based on the input size of the online dynamic task, achieving a combination of online scheduling and offline optimization. Furthermore, by dynamically combining batches online using batch operation primitives and selecting the most efficient parallel method, the system's utilization of hardware resources is improved, significantly increasing system throughput.

[0071] Figure 1 This invention illustrates a cross-layer collaborative scheduling system 100 for multi-model inference services, according to another embodiment of the present invention. The system 100 is used to implement the methods described in the above embodiments and includes an initialization module 10, a subgraph partitioning module 20, a parallel strategy optimization module 30, a recording and filtering module 40, a dynamic batch processing module 50, and an online collaborative scheduling module 60, wherein:

[0072] Initialization module 10 is used to determine the number of subgraphs to be divided into based on system requirements, and to set initial partitioning points in each subgraph. Subgraph partitioning module 20 is used to iteratively optimize the partitioning points for each subgraph using the coordinate descent method, fixing the optimal partitioning points of other subgraphs in each iteration, and randomly generating multiple partitioning methods for the current subgraph. Based on the actual execution latency of the subgraph in a specific batch, and combined with the weights generated from historical request data, the performance of the partitioning methods is weighted and calculated to select the final subgraph partitioning scheme for subgraph partitioning. Parallel strategy optimization module 30 is used to recursively divide each subgraph after switching into multiple subgraph blocks while maintaining the dependencies between operators. The subgraph parallel optimization problem is decomposed into sub-problems of subgraph block parallel optimization using a recursive approach, and the subgraph block parallel optimization problem is enumerated. The system obtains parallel scheduling schemes for subgraphs with valid subgraph blocks and calculates optimal solutions, under different batch sizes. A recording and filtering module 40 records these parallel scheduling schemes and, based on policy similarity analysis, filters a fixed number of target policies as online candidate policies. A dynamic batch processing module 50 configures task-level operations for implementing dynamic batch processing. These task-level operations include creation, stretching, and splitting operations. The creation operation collects online requests to create new batches, the stretching operation adds new requests to existing batches, and the splitting operation splits large batches into multiple smaller batches. An online collaborative scheduling module 60 processes requests online through these task-level operations based on a service quality-aware scheduling strategy and selects the optimal parallel scheduling scheme from the online candidate policies according to the current batch size.

[0073] In an optional implementation, the subgraph partitioning module 20 further includes a historical data processing unit, which is used to statistically analyze the frequency of occurrence of different input sizes in the historical request data and generate weight coefficients for weighted performance evaluation of subgraph partitioning.

[0074] This invention also provides a storage medium for storing a computer program, such as the cross-layer collaborative scheduling method for multi-model inference services, as described above. For example, computer program instructions, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. The program instructions for invoking the methods of this application may be stored in a fixed or removable storage medium, and / or transmitted via data streams in broadcast or other signal carrying media, and / or stored in the storage medium of a computer device operating according to the program instructions. Here, one embodiment according to this application includes, as... Figure 1The computer device shown is a cross-layer collaborative scheduling system for multi-model inference services. The computer device preferably includes a storage medium for storing computer programs and a processor for executing computer programs. When the computer program is executed by the processor, the computer device is triggered to execute the methods and / or technical solutions based on the foregoing embodiments.

[0075] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0076] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that works with a processor to perform the various steps or functions.

[0077] The method according to the invention can be implemented on a computer as a computer-implemented method, or in dedicated hardware, or a combination of both. Executable code or portions thereof for the method according to the invention can be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Preferably, the computer program product includes non-transitory program code components stored on a computer-readable medium so as to execute the method according to the invention when the program product is executed on a computer.

[0078] In a preferred embodiment, the computer program includes computer program code components adapted to perform all the steps of the method according to the invention when the computer program is run on a computer. Preferably, the computer program is embodied on a computer-readable medium.

[0079] In summary, the cross-layer collaborative scheduling method and system for multi-model inference services provided by this invention divides the multi-model computation graph into a fixed number of subgraphs based on historical request data during the offline phase using coordinate descent. It then generates operator-level parallel strategies with different batch sizes for each subgraph through dynamic programming, and selects representative strategy sets through similarity analysis. During the online phase, request batches are dynamically adjusted through three operations: creation, stretching, and splitting. Stretching is triggered when the throughput improvement of merging batches meets the QoS target; splitting is performed when the total latency after splitting is less than or equal to the original latency; and optimization strategies are dynamically switched at the subgraph boundaries based on the real-time batch size. Thus, this invention has the following advantages: fine-grained operator-level scheduling improves hardware utilization; historical data-driven subgraph segmentation adapts to multiple input sizes; and dynamic batch processing and strategy switching work together to improve system throughput while ensuring that single request latency remains within the QoS range.

[0080] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0081] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A cross-layer collaborative scheduling method for multi-model inference services, characterized in that, Including the following steps: Determine the number of subgraphs into which the multi-model structure needs to be divided based on system requirements, and set initial partitioning points in each sub-model; The coordinate descent method is used to iteratively optimize the partitioning points for each sub-model. In each iteration, the optimal partitioning points of other sub-models are fixed, and multiple partitioning methods are randomly generated for the current sub-model. Based on the actual execution latency of the subgraph in a specific batch, the performance of the partitioning methods is weighted and calculated in combination with the weights generated from historical request data. The final subgraph partitioning scheme is then selected for subgraph partitioning. For each subgraph after switching, it is recursively divided into multiple subgraph blocks while maintaining the dependencies between operators; the subgraph parallel optimization problem is decomposed into sub-problems of subgraph block parallel optimization in a recursive manner, all legal subgraph blocks are enumerated and the optimal solution is calculated to obtain the parallel scheduling scheme of the subgraph under different batch sizes. Record the parallel scheduling scheme, and based on policy similarity analysis, select a fixed number of target policies as online candidate policies; The configuration is used to implement task-level operations for dynamic batch processing. The task-level operations include new operation, stretch operation and split operation. The new operation is to collect online requests and create a new batch. The stretch operation is to add new requests to an existing batch. The split operation is to split a large batch into multiple smaller batches. Based on the quality of service-aware scheduling strategy, requests are processed online through the task-level operations, and the optimal parallel scheduling scheme is selected from the online candidate strategies according to the current batch size.

2. The cross-layer collaborative scheduling method for multi-model inference services according to claim 1, characterized in that, The service quality-aware scheduling strategy, through the task-level operation for online request processing, includes: The scheduling effect is evaluated based on the service quality target. If the additional waiting time of the stretching operation does not exceed the upper limit of the delay allowed by the service quality, the stretching operation is executed; if the total delay of the splitting operation does not exceed the original batch delay, the splitting operation is executed.

3. The cross-layer collaborative scheduling method for multi-model inference services according to claim 1, characterized in that, The process of selecting a fixed number of target strategies as online candidate strategies based on strategy similarity analysis includes: Calculate the similarity between different parallel scheduling schemes, and select a fixed number of target strategies as online candidate strategies based on the similarity.

4. The cross-layer collaborative scheduling method for multi-model inference services according to claim 1, characterized in that, The recursive division of each subgraph after the switch into multiple subgraph blocks while maintaining the dependencies between operators includes: Multiple sets of hardware synchronization events are introduced to recursively divide the subgraph into subgraph blocks.

5. The cross-layer collaborative scheduling method for multi-model inference services according to claim 1, characterized in that, The specific operation of creating a new batch is as follows: when the system receives a new request and there is no batch waiting, it collects a preset number of online requests to create a new batch. The stretching operation specifically involves adding newly arrived requests to an existing batch when the batch has not yet started execution or has not yet finished execution, provided that the number of new requests does not exceed the maximum processing capacity of the subgraph corresponding to the batch. The splitting operation specifically involves splitting a large batch that is being executed or waiting to be executed into at least two smaller batches, with each smaller batch being no smaller than the minimum batch threshold set by the system.

6. The cross-layer collaborative scheduling method for multi-model inference services according to claim 1, characterized in that, The determination of the number of subgraphs satisfies the following condition: The order of the division points within each of the sub-modules is consistent; Each of the subgraphs is composed of model computation blocks from different submodules.

7. A cross-layer collaborative scheduling system for implementing the method described in any one of claims 1 to 6 for multi-model inference services, characterized in that, Including: The initialization module is used to determine the number of subgraphs into which the multi-model structure needs to be divided based on system requirements, and to set the initial partitioning points in each sub-model; The subgraph partitioning module is used to iteratively optimize the partitioning points for each submodel using the coordinate descent method. In each iteration, the optimal partitioning points of other submodels are fixed, and multiple partitioning methods are randomly generated for the current submodel. Based on the actual execution latency of the subgraph in a specific batch, the performance of the partitioning methods is weighted and calculated in combination with the weights generated from historical request data, and the final subgraph partitioning scheme is selected for subgraph partitioning. The parallel strategy optimization module is used to recursively divide each subgraph after switching into multiple subgraph blocks while maintaining the dependencies between operators. The subgraph parallel optimization problem is decomposed into sub-problems of subgraph block parallel optimization in a recursive manner. All legal subgraph blocks are enumerated and the optimal solution is calculated to obtain the parallel scheduling scheme of the subgraph under different batch sizes. The recording and filtering module is used to record the parallel scheduling scheme and filter a fixed number of target strategies as online candidate strategies based on strategy similarity analysis.

Images