Scheduling method and system for inference tasks

Abstract

This application provides a method and system for scheduling inference tasks. The method includes: upon receiving an inference request, acquiring inference task information corresponding to the inference request and status information of multiple nodes in a cluster, wherein the inference request is used to request the execution of an inference task; based on the inference task information and status information, predicting at least one execution path of the inference request in the cluster and determining the execution cost of the at least one execution path; based on the execution cost, determining a target execution path from the at least one execution path; scheduling nodes on the target execution path to execute the inference task and obtaining the execution result of the inference task. This application solves the technical problem of low resource utilization in the scheduling of inference tasks in related technologies.

Description

Technical Field

[0001] This application relates to the field of task scheduling, and more specifically, to a scheduling method and system for inference tasks. Background Technology

[0002] With the widespread application of deep learning models in online service scenarios such as recommendation systems, autonomous driving, and speech recognition, how to efficiently implement inference tasks in a cluster is an urgent problem to be solved.

[0003] Static configuration is often used in related technologies to implement task scheduling with a fixed strategy. However, with increasingly complex inference tasks, using a fixed scheduling strategy can lead to uneven resource allocation and waste of resources. Summary of the Invention

[0004] This application provides a scheduling method and system for inference tasks, which at least solves the technical problem of low resource utilization in the scheduling of inference tasks in related technologies.

[0005] According to one aspect of the embodiments of this application, a method for scheduling inference tasks is provided, comprising: upon receiving an inference request, obtaining inference task information corresponding to the inference request and status information of multiple nodes in a cluster, wherein the inference request is used to request the execution of an inference task; based on the inference task information and the status information, predicting at least one execution path of the inference request in the cluster, and determining the execution cost of the at least one execution path; based on the execution cost, determining a target execution path from the at least one execution path; scheduling nodes on the target execution path to execute the inference task, and obtaining the execution result of the inference task.

[0006] Optionally, obtain the inference task information corresponding to the inference request, including: based on the inference request, determine the model identifier of the inference model, the evaluation index of the inference task, and the source information of the inference request, wherein the inference model is deployed on multiple nodes and is used to execute the inference task; based on the model identifier, obtain the historical operation information of the inference model; and based on the model identifier, evaluation index, source information, and historical operation information, obtain the inference task information.

[0007] Optionally, based on inference task information and state information, predicting at least one execution path for the inference request in the cluster and determining the execution cost of at least one execution path includes: predicting at least one execution path based on inference task information and state information; when at least one execution path includes a first execution path, a second execution path, and a third execution path, determining the first execution cost corresponding to the first execution path, the first execution cost corresponding to the second execution path, and the third execution cost corresponding to the third execution path based on the inference task information and state information, wherein the first execution path is the CPU instance path of the local node, the second execution path is the GPU instance path of the local node, and the third execution path is the cross-node GPU instance path; preferably, determining the first execution cost corresponding to the first execution path based on inference task information and state information includes: determining the processing time and the waiting time based on inference task information and state information, wherein the processing time is the time required for the local node to execute the inference task, and the waiting time is the time required for the inference request to complete in the cluster. The time spent waiting for processing in the queue; determining the first execution cost based on the sum of the processing time and the waiting time; preferably, determining the second execution cost corresponding to the second execution path based on inference task information and status information, including: determining the model loading time, processing time, and waiting time based on inference task information and status information, wherein the processing time is the time required for this node to execute the inference task, and the waiting time is the time for the inference request to wait for processing in the node queue; determining the second execution cost based on the sum of the model loading time, processing time, and waiting time; preferably, determining the third execution cost corresponding to the third execution path based on inference task information and status information, including: determining the cross-node transmission time, model loading time, processing time, and waiting time based on inference task information and status information, wherein the processing time is the time required for this node to execute the inference task, and the waiting time is the time for the inference request to wait for processing in the node queue; determining the third execution cost based on the sum of the cross-node transmission time, model loading time, processing time, and waiting time.

[0008] Optionally, based on execution cost, a target execution path is determined from at least one execution path, including: determining the lowest execution cost among the execution costs; and determining the execution path corresponding to the lowest execution cost as the target execution path, provided that the execution cost is within the threshold range corresponding to the task processing level of the inference request.

[0009] Optionally, based on execution cost, a target execution path is determined from at least one execution path, including: inputting inference task information and state information as input features, and execution cost as a label for the execution path, into a prediction model, and predicting the delay time of at least one execution path through the prediction model; and determining the target execution path based on the delay time.

[0010] According to another aspect of the embodiments of this application, a scheduling system for inference tasks is also provided, comprising: a client connected to a scheduling platform for initiating an inference request, wherein the inference request is used to request the execution of an inference task; multiple nodes deployed in a cluster and connected to the scheduling platform for sending status information of the multiple nodes to the scheduling platform and for executing the inference task; and a scheduling platform connected to the client for receiving the inference request sent by the client, obtaining inference task information and status information corresponding to the inference request, predicting at least one execution path of the inference request in the cluster based on the inference task information and status information, determining the execution cost of at least one execution path, determining a target execution path from the at least one execution path based on the execution cost, scheduling nodes on the target execution path to execute the inference task, and obtaining the execution result of the inference task.

[0011] Optionally, the scheduling platform includes: a gateway for receiving inference requests; a decision module for obtaining inference task information and status information corresponding to the inference request based on the inference request, predicting at least one execution path of the inference request in the cluster based on the inference task information and status information, determining the execution cost of at least one execution path, and determining the target execution path in at least one execution path based on the execution cost; and a scheduling module for scheduling nodes on the target execution path to execute the inference task and obtain the execution result of the inference task.

[0012] Optionally, the scheduling platform may also include a monitoring module, which connects to multiple nodes to collect the operating parameters of the multiple nodes in order to evaluate the target execution path.

[0013] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0017] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0018] In this embodiment, upon receiving an inference request, the system acquires inference task information corresponding to the request and status information of multiple nodes in the cluster. Based on the inference task information and status information, it predicts at least one execution path for the inference request in the cluster and determines the execution cost of at least one execution path. Based on the execution cost, it determines a target execution path from the at least one execution path. Then, it schedules nodes on the target execution path to execute the inference task and obtain the execution result of the inference task. This application, upon receiving an inference request, comprehensively perceives the inference task information and node status information by acquiring the inference task information corresponding to the request and the status information of multiple nodes in the cluster, thereby predicting at least one execution path. Based on the execution cost, it analyzes the resource utilization of each execution path, selects a target execution path, and schedules nodes on the target execution path to collaboratively complete the inference task and return the execution result. This improves the resource utilization of the inference task, achieving the purpose of dynamic scheduling and thus realizing the technical effect of improving resource utilization. This solves the technical problem of low resource utilization in the scheduling of inference tasks in related technologies. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a schematic diagram of a scheduling method for inference tasks according to an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of an optional inference task scheduling method according to an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of a scheduling system for inference tasks according to an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of an optional scheduling architecture according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of an optional hybrid scheduling method according to an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of a scheduling device for a reasoning task according to an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] According to an embodiment of this application, a method embodiment for scheduling inference tasks is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0030] This embodiment provides a method for scheduling inference tasks. Figure 1 This is a flowchart of a scheduling method for inference tasks according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps:

[0031] Step S102: Upon receiving an inference request, obtain the inference task information corresponding to the inference request, as well as the status information of multiple nodes in the cluster.

[0032] Among them, the reasoning request is used to request the execution of a reasoning task.

[0033] The aforementioned inference requests can be prediction calls initiated by the user or automatically by the system to the online inference service. Inference requests can be encapsulated and transmitted. Inference requests can reflect metadata such as model identifier, input data size (e.g., feature vector dimension or image resolution), SLA (Service Level Agreement) level, and tenant identifier. Each inference request can represent an independent model inference task, and the computational complexity of the task is strongly correlated with the input size. The scheduling system can perform lightweight feature extraction on each inference request to determine whether it is suitable for processing on the CPU or needs to be scheduled to the GPU to meet latency requirements.

[0034] The aforementioned inference task can be a logical aggregation of a set of inference requests with the same model attributes and SLA requirements by the scheduling system, serving as the fundamental object for resource allocation and cost estimation. Inference tasks may reflect a persistent service behavior pattern, such as "the inference task of a recommendation model on a user profile." Task-level modeling is used to identify long-term behavioral trends. For example, a task might historically have an average latency of 15ms on a GPU and 80ms on a CPU, and be extremely sensitive to latency. This task-level abstraction allows scheduling strategies to reuse experience from multiple inference requests, improving decision-making efficiency.

[0035] The aforementioned inference task information can serve as a metadata set supporting the intelligent decision-making of the scheduling system. This information reflects the static attributes and runtime dynamic characteristics of the model corresponding to the inference request. Inference task information may include, but is not limited to, model identifier, version number, input data dimension, historical average inference latency, standard deviation, SLA level, tenant priority, model structural complexity, and parameter size. Inference task information can be pre-loaded by the model registry or offline training pipeline, or dynamically obtained upon the first request. The scheduling system can use this inference task information to estimate the theoretical execution time on different hardware platforms. For example, large models need to be executed on GPUs (Graphics Processing Units), while lightweight models can run efficiently on CPUs (Central Processing Units). Simultaneously, tenant identifiers can be used to achieve resource isolation and differentiated SLA guarantees. Through structured storage and indexing, inference task information can be quickly matched with scheduling strategies, enabling one-model-one-strategy management. Inference task information can be linked with monitoring data to achieve dynamic adjustment of model-level resource quotas.

[0036] The cluster described above is the underlying infrastructure implemented in this embodiment. A cluster can refer to a distributed computing environment composed of multiple nodes, uniformly managing CPU and GPU resource pools and supporting multi-tenant, multi-model online inference services. The cluster can achieve elastic scaling and high availability of services through components such as scheduling, networking, and storage. The cluster can provide a unified resource abstraction, enabling the scheduler to be aware of the global resource status, rather than being limited to a single node. Nodes within the cluster may be heterogeneous, such as some being GPU nodes (equipped with A100 processors) and others being CPU nodes (high core count, large memory). The scheduling system can coordinate across nodes to achieve load balancing. The cluster also provides capabilities such as service discovery, automatic scaling, and health checks, providing environmental guarantees for scheduling decisions.

[0037] The nodes mentioned above are physical or virtual computing hosts in the cluster, serving as the physical carriers for resource execution. Each node can be equipped with multi-core CPUs, large-capacity memory, and GPU accelerators to form a hybrid computing unit. A node can also be a single computing unit, such as configured with a CPU or a GPU. Node status can influence scheduling decisions; for example, whether the GPU is idle, whether the video memory is occupied by other models, whether the CPU load is too high, and whether the target model weights have been pre-loaded are all real-time decision bases. The scheduler can rely on the metrics reported by the nodes to determine "whether execution can be performed on this node," avoiding network latency caused by cross-node migration. By deploying lightweight intelligent agents on the nodes, the system can collect millisecond-level performance data, making scheduling decisions more timely.

[0038] The aforementioned status information reflects the scheduling system's awareness of the cluster's runtime status. This status information can include, but is not limited to, real-time performance metrics at the node and service levels, such as CPU utilization, GPU utilization, memory usage, request queue length, latency, whether the model is preloaded in GPU memory, number of restarts, and error rate. Status information can be reported periodically by nodes or triggered by inference requests. Unlike static node configurations, status information reflects the node's current actual load. For example, even if a GPU node is idle, the actual scheduling cost may still be high because the model is not loaded. By fusing status information with inference task information, the scheduler can avoid allocating requests simply because the GPU is idle, ignoring the 300ms loading time. The accuracy of the status information determines the scheduling effect and is the feedback source for achieving a closed loop.

[0039] In one optional embodiment, the model metadata registry can be used to query the inference task information corresponding to the inference request. This metadata is dynamically injected into the request header of the inference request to form a structured task context. This task context flows with the inference request to the backend service for direct consumption by the scheduling system. Thus, upon receiving an inference request, the request header is parsed to obtain the inference task information, and the node is triggered to report its status information.

[0040] In another optional embodiment, Prometheus periodically retrieves metrics exposed by custom Exporters deployed on each node. The Prometheus Exporter is a service or agent in the scheduling system that monitors the nodes. The Exporter can be deployed on each node to periodically collect resource usage data for locally running inference tasks, including GPU utilization, memory usage, queue length, number of currently processed requests, and average latency for each model. Simultaneously, the Exporter can read model identifiers and SLA levels from environment variables, associate node status with inference tasks, and report them uniformly. The scheduler uses Prometheus queries to aggregate the "model-state" mapping of nodes and construct a global view.

[0041] In another alternative embodiment, an embedded intelligent agent, upon receiving an inference request, extracts the inference task information from the request in real time and combines it with the local resource status, then sends it to the centralized scheduling service via a streaming channel. Simultaneously, the node-level monitoring intelligent agent publishes metrics such as GPU status and network latency. The scheduling service, acting as a consumer, performs real-time stream processing, associating and aggregating the inference task information with the node's status information in memory.

[0042] Step S104: Based on the inference task information and state information, predict at least one execution path of the inference request in the cluster, and determine the execution cost of at least one execution path.

[0043] The execution paths described above can be complete processing routes selected for a single inference request, representing the direct output of the scheduling decision. Each execution path may represent a possible execution scheme, such as a local CPU instance (on the same node), a local GPU instance (on the same node, but a different service), a remote GPU instance (across nodes, requiring network forwarding), or a combined instance of local and remote GPUs. Each execution path has a different cost structure; for example, the local CPU path has no transmission overhead but is computationally slow. The local GPU path has no network latency but may require model loading. The remote GPU path is computationally fast but has bandwidth overhead. By balancing speed and cost, a better path is selected. Path selection is also constrained by service topology, such as GPU services being deployed only on specific nodes. Through path abstraction, the complex scheduling problem is transformed into a path cost problem, enabling flexible, scalable, and interpretable inference scheduling strategies.

[0044] The aforementioned execution cost can be a quantitative metric for evaluating the merits of each execution path. Execution cost may include, but is not limited to, processing time, such as the time required for the model to perform inference on the CPU or GPU (processing time can be determined by the model structure and input size); waiting time, latency caused by service queue congestion, which can be estimated by the current number of requests and average processing speed; model loading time, such as the time it takes to load model weights from disk to GPU memory or RAM; data transfer time, such as the time it takes to copy input data between host memory and GPU memory; and cross-node transfer time, the time incurred due to latency in forwarding requests across nodes. Execution cost can be estimated in milliseconds. Determining the execution cost of each execution path facilitates subsequent path selection.

[0045] In one optional embodiment, a statistical model of the average execution time, queuing latency, and migration cost for each model on different execution paths can be constructed by offline analysis of historical inference request logs. For each inference request, the system looks up the corresponding possible execution path and baseline execution cost based on the inference task information, and estimates the execution cost by combining this with the current node's status information.

[0046] In another optional embodiment, execution paths are determined based on inference task information and state information using path determination rules, resulting in at least one execution path. Then, a sliding time window (e.g., the most recent 100 inference requests) is introduced to dynamically weight the execution cost of each execution path. The system maintains a lightweight memory structure to record the execution time sequence of each execution path. For new inference requests, an exponentially weighted moving average is used to calculate the execution cost.

[0047] In another alternative embodiment, an interpretable decision tree model is used, where tree nodes filter execution paths and assign costs based on preset thresholds. Each leaf node corresponds to an execution path and an execution cost. Thus, after obtaining inference task information and state information, the input to the decision tree model outputs at least one execution path and its corresponding execution cost.

[0048] In another alternative embodiment, an online prediction model based on a lightweight neural network predicts the execution path and execution cost. After inputting inference task information and state information into the online prediction model, the model can capture non-linear relationships and output at least one execution path and its corresponding execution cost. The online prediction model is trained offline using historical execution data and continuously fine-tuned through small-batch online incremental learning after going online.

[0049] Step S106: Based on execution cost, determine the target execution path from at least one execution path.

[0050] The target execution path described above is the execution plan selected for inference requests after evaluating the execution costs of various execution paths. The target execution path is a solution that integrates latency constraints, resource availability, and cost-effectiveness. The target path must be executable, meaning the target instance exists, resources are sufficient, and the network is reachable. The target execution path can forward inference requests to the corresponding container group through service discovery mechanisms or gateway routing rules. The accuracy of the target execution path directly affects the service SLA achievement rate and is a direct reflection of intelligent scheduling capabilities.

[0051] In one optional embodiment, selection rules are set to obtain the target execution path. The selection rules may involve sorting the execution costs of each execution path in ascending order and selecting the execution path with the lowest execution cost that also satisfies the requirement of being the target execution path. If multiple execution costs are the same, the local execution path is selected first (to avoid network overhead), followed by an instance with a pre-loaded model (to reduce the risk of cold start). This strategy is simple to implement, responds quickly, and is suitable for real-time inference scenarios that are sensitive to scheduling latency.

[0052] In another alternative embodiment, the Service Level Agreement (SLA) level is introduced as a hard constraint weight based on the execution cost. The final evaluation of each execution path is defined as: Weighted Cost = Execution Cost × (1 + SLA Penalty Factor), where the higher the SLA level, the larger the penalty factor. Even if a path has a low execution cost, if its predicted completion time may violate a high-priority SLA, the weighted cost will increase significantly, thus excluding it. This allows for the selection of execution paths with lower execution costs from among those that meet the SLA. This ensures that high-value requests always receive priority resource guarantees, reducing timeout rates, and is particularly suitable for high-real-time scenarios such as autonomous driving.

[0053] In another alternative embodiment, the scheduling objective is modeled as a multi-objective analysis problem, such as breaking down execution costs into execution latency, resource consumption, and cross-node transmission. For each execution path, a three-dimensional cost vector (latency, resource consumption, network overhead) is generated. A Pareto front solution set is selected using non-dominated sorting, and then weights are assigned to each objective using entropy weighting or expert weighting to calculate a comprehensive score. Finally, the execution path with the highest score is selected as the target execution path.

[0054] Step S108: Schedule nodes on the target execution path to execute inference tasks and obtain the execution results of the inference tasks.

[0055] The execution results described above can be the actual feedback data after the inference request has been processed on the target execution path. Execution results may include, but are not limited to: actual response time, whether a timeout occurred, actual CPU or GPU consumption, error code, request source, and processing node identifier. Execution results can be asynchronously reported to the scheduling management service to update cost model parameters and improve the accuracy of the next prediction. Execution results can also be used to trigger elastic scaling: if the timeout rate of a GPU service increases, the system can automatically increase the number of replicas. By continuously learning from historical execution results, the system can adapt to dynamic changes such as model updates, traffic surges, and hardware aging, thereby achieving adaptive scheduling.

[0056] In one optional embodiment, a proxy is deployed in the inference service of the node along the target execution path. The target execution path is used as the routing target, and load balancing is achieved through the proxy. Inference requests are accurately delivered to the target instance without modifying the client logic, and the instance calls its local model engine to perform inference, returning the encapsulated execution result.

[0057] In another optional embodiment, the inference request is serialized into a binary message and sent to each node, thereby triggering each node to execute the inference task. After execution, the execution result is sent back to the scheduling service of the original request via a callback interface, or written to a unified result store. The scheduling service listens for callbacks or polls for execution results. This approach supports multi-site active-active deployment, load balancing, and fault-tolerant retries, and is suitable for distributed inference clusters such as those used in connected vehicles.

[0058] In this embodiment of the application, upon receiving an inference request, the inference task information corresponding to the inference request and the status information of multiple nodes in the cluster are obtained. Based on the inference task information and the status information, at least one execution path of the inference request in the cluster is predicted, and the execution cost of at least one execution path is determined. Based on the execution cost, a target execution path is determined from the at least one execution path. Then, the nodes on the target execution path are scheduled to execute the inference task to obtain the execution result of the inference task.

[0059] Upon receiving an inference request, this application comprehensively perceives the inference task information and node status information by acquiring the inference task information corresponding to the inference request and the status information of multiple nodes in the cluster. This allows for the prediction of at least one execution path. Based on the execution cost, the application analyzes the resource utilization of each execution path, selects the target execution path, and schedules the nodes on the target execution path to collaboratively complete the inference task and return the execution result. This improves the resource utilization of the inference task, achieving the purpose of dynamic scheduling and thus realizing the technical effect of improving resource utilization. Consequently, it solves the technical problem of low resource utilization in the scheduling of inference tasks in related technologies.

[0060] Optionally, obtain the inference task information corresponding to the inference request, including: based on the inference request, determine the model identifier of the inference model, the evaluation index of the inference task, and the source information of the inference request, wherein the inference model is deployed on multiple nodes and is used to execute the inference task; based on the model identifier, obtain the historical operation information of the inference model; and based on the model identifier, evaluation index, source information, and historical operation information, obtain the inference task information.

[0061] The aforementioned inference model can refer to a model deployed across multiple nodes to perform inference tasks, such as recommendation tasks, image recognition, speech-to-text conversion, and autonomous driving perception. A model identifier can be a string or number that uniquely identifies an inference model, used to distinguish between different models or different versions of the same model.

[0062] In one optional embodiment, obtaining the inference task information corresponding to the inference request is a prerequisite for realizing intelligent scheduling. A specific inference request can be abstracted into an inference task with semantics and historical context, thereby providing an accurate basis for subsequent cost prediction and path selection.

[0063] First, based on the inference request, the model identifier of the inference model, the evaluation metrics of the inference task, and the source information of the inference request can be determined. The model identifier is used to uniquely identify the inference model invoked by the inference request and is the basis for associating historical data and resource configuration. Evaluation metrics (such as SLA level and latency requirements) define the performance constraints of the inference request and determine the priority and fault tolerance threshold during scheduling. Source information (such as tenant identifier, service system, and device type) is used to distinguish different users or application scenarios, supporting multi-tenant resource isolation and policy differentiation. This step can transform the inference request into structured task metadata, enabling the system to understand the task meaning of the inference request.

[0064] Then, based on the model identifier, the historical execution information of the model is obtained. This historical execution information may include, but is not limited to, the average execution time, latency distribution, success rate, memory usage, cold start frequency, and resource consumption fluctuations of the inference model on CPU and GPU in the past. This data comes from monitoring systems or log analysis platforms. Through historical execution information, the system no longer relies on static assumptions but predicts the execution path and cost of the current inference request based on actual performance, thus improving prediction accuracy.

[0065] Therefore, complete inference task information can be generated by integrating model identifiers, evaluation metrics, source information, and historical operational information. Static configurations, dynamic metrics, and historical model behavior are unified into a single task profile to reflect the task characteristics corresponding to the inference request. This inference task information can serve as input, directly driving subsequent cost calculations and scheduling decisions, ensuring that scheduling behavior possesses service awareness and data-driven characteristics, and avoiding resource waste or service degradation caused by fixed scheduling.

[0066] Optionally, based on inference task information and state information, predict at least one execution path of the inference request in the cluster, and determine the execution cost of at least one execution path, including: predicting at least one execution path based on inference task information and state information; and, if at least one execution path includes a first execution path, a second execution path, and a third execution path, determining the first execution cost corresponding to the first execution path, the first execution cost corresponding to the second execution path, and the third execution cost corresponding to the third execution path based on the inference task information and state information, wherein the first execution path is the CPU instance path of the local node, the second execution path is the GPU instance path of the local node, and the third execution path is the cross-node GPU instance path.

[0067] In one optional embodiment, the possible execution paths of the current inference request are evaluated by combining inference task information that reflects the characteristics of the task and state information of the real-time status of the cluster, so as to narrow the decision space, eliminate unreasonable execution paths (such as idle nodes without models, nodes with no network connectivity), and ensure that the execution path has actual executability.

[0068] Then, when the execution path includes the CPU instance path of this node, the GPU instance path of this node, and the GPU instance path across nodes, the corresponding execution cost is determined for each execution path to perform quantitative evaluation, realize the comparison between different hardware resources, provide an objective basis for scheduling decisions, avoid blind scheduling, and achieve a global balance between resource utilization and service performance.

[0069] Preferably, determining the first execution cost corresponding to the first execution path based on inference task information and status information includes: determining the processing time and waiting time based on inference task information and status information, wherein the processing time is the time required for this node to execute the inference task, and the waiting time is the time for the inference request to wait for processing in the node queue; and determining the first execution cost based on the sum of the processing time and the waiting time.

[0070] In one optional embodiment, the execution cost of an inference request on the local node's CPU instance is accurately estimated to provide a quantitative basis for scheduling decisions. Specifically, using inference task information and status information, the time required for the inference request to complete the actual computation on the local node's CPU, i.e., the processing time, is predicted. Simultaneously, combined with the status information of this type of inference service on the node, the waiting time before the request can be scheduled for execution is estimated, i.e., the waiting time. Decomposing the abstract execution cost into two measurable and predictable dimensions improves the accuracy and interpretability of the estimation.

[0071] The processing time in this embodiment can be estimated through offline statistics or real-time utilization, reflecting the actual computation time required for the inference model to perform one forward inference operation on the CPU. The waiting time in this embodiment refers to the time required for execution to begin after a reasoning request arrives and the queued requests have been processed. The waiting time can be estimated using the number of requests or the number of historical reasoning requests.

[0072] Then, the processing time and waiting time are added together to form the first execution cost of the execution path. This first execution cost reflects the end-to-end time taken for the inference request from submission to completion. It is a key indicator for evaluating the execution path, avoiding suboptimal scheduling caused by resource idleness or simple load balancing, and achieving cost-sensitive and latency-controllable scheduling.

[0073] Preferably, determining the second execution cost corresponding to the second execution path based on inference task information and status information includes: determining the model loading time, processing time, and waiting time based on inference task information and status information, wherein the processing time is the time required for this node to execute the inference task, and the waiting time is the time for the inference request to wait for processing in the node queue; and determining the second execution cost based on the sum of the model loading time, processing time, and waiting time.

[0074] In one optional embodiment, the focus is on calculating the execution cost of an inference request executed on the local node's GPU instance. Inference task information and state information are used to determine whether the model has been preloaded in the local node's GPU memory: if not, the weights need to be loaded from disk or main memory, and the time taken is recorded as the model loading time. The processing time on the local node's GPU and the waiting time in the local node's queue are also determined. These three factors together constitute the complete latency profile of GPU execution.

[0075] The processing time in this embodiment can refer to the processing time on the GPU, which can be estimated through historical statistics or current utilization. The waiting time can refer to the waiting time in the node's GPU queue, that is, the time waiting for the currently queued task of the GPU instance to be processed, which can be estimated through queue length or historical waiting time.

[0076] This sums up the model loading time, processing time, and waiting time overhead to form the end-to-end expected latency of the inference request executed on the local GPU. This second execution cost not only reflects computing power performance but also incorporates the effects of cold start and queuing, avoiding blind scheduling based solely on the fact that GPUs are faster. It enables the assessment of GPU resource usage, thereby preventing latency deterioration caused by frequent model loading or high queue backlog and improving the reliability and stability of scheduling decisions.

[0077] Preferably, the third execution cost corresponding to the third execution path is determined based on the inference task information and the status information, including: determining the cross-node transmission time, model loading time, processing time, and waiting time based on the inference task information and the status information, wherein the processing time is the time required for the node to execute the inference task, and the waiting time is the time for the inference request to wait for processing in the node queue; and the third execution cost is determined based on the sum of the cross-node transmission time, model loading time, processing time, and waiting time.

[0078] In one alternative embodiment, a third execution cost of inference requests executed across cross-node GPU instances is precisely quantified to support decisions on whether to remotely schedule inference requests to other node GPUs.

[0079] By utilizing inference task information and state information, the four major overheads of cross-node execution are estimated respectively. For example, cross-node transmission time can be estimated from the size of the requested data and the network latency and bandwidth from the source node to the target node. The model loading time, processing time and waiting time are also determined, thereby comprehensively identifying the sources of latency in cross-node scheduling and avoiding focusing only on computing power while ignoring communication and loading costs.

[0080] The processing time in this embodiment can also refer to the processing time on the GPU, which can be estimated through historical statistics or current utilization. The waiting time can also refer to the waiting time in the node's GPU queue, that is, the time waiting for the currently queued task of the GPU instance to be processed, which can be estimated through queue length or historical waiting time.

[0081] This sums up the four types of overhead to form the end-to-end expected total latency cost of the cross-node GPU path, i.e., the third execution cost. This avoids the deterioration of service performance caused by network latency spikes and frequent cold starts due to blind remote scheduling, and achieves global adjustment of resource scheduling and controllable latency.

[0082] By determining the execution cost described above, the complete link cost can be determined from an end-to-end perspective, rather than evaluating computing power in isolation. By explicitly distinguishing whether the model is already loaded into GPU memory, misjudgments of cold start penalties are avoided, achieving a clear distinction between hot and cold starts. Explicitly introducing cross-node transmission time reflects the impact of network status on the task.

[0083] Optionally, based on execution cost, a target execution path is determined from at least one execution path, including: determining the lowest execution cost among the execution costs; and determining the execution path corresponding to the lowest execution cost as the target execution path, provided that the execution cost is within the threshold range corresponding to the task processing level of the inference request.

[0084] In one alternative embodiment, a target execution path is selected from multiple execution paths by combining execution cost and quality of service constraints.

[0085] Among the predicted execution paths, the path with the lowest execution cost is selected, prioritizing efficient and fast-responding execution solutions to improve resource utilization. Service Level Agreement (SLA) constraints are introduced because different tasks (such as high-priority autonomous driving inference and low-priority log analysis) have different latency tolerance thresholds. Even if an execution path has a low cost, it cannot be selected if the predicted latency exceeds the maximum allowable value for that task, preventing the pursuit of efficiency at the expense of service quality. This ensures that, while meeting the SLA, the lowest-cost execution path is selected as the actual execution solution, achieving a balance between superior performance and service availability. It ensures that high-requirement tasks do not time out, and low-requirement tasks do not consume valuable resources, achieving constraint-driven intelligent scheduling.

[0086] Optionally, based on execution cost, a target execution path is determined from at least one execution path, including: inputting inference task information and state information as input features, and execution cost as a label for the execution path, into a prediction model, and predicting the delay time of at least one execution path through the prediction model; and determining the target execution path based on the delay time.

[0087] In one optional embodiment, inference task information and state information are used as inputs to the prediction model to ensure that the prediction model can fully perceive the scheduling context. Execution cost is used as a label for the execution path to supervise the prediction model in learning the complex nonlinear relationship between input and latency. This allows the trained prediction model to predict the latency of the current inference request on each execution path, thereby more accurately capturing patterns such as model differences and load coupling. Based on the latency, the target execution path is determined, improving scheduling accuracy and reducing latency jitter.

[0088] The technical solution proposed in this application will be described below with reference to an optional embodiment. This application proposes a CPU and GPU scheduling method suitable for Kubernetes online inference clusters. Kubernetes is a container orchestration system that can schedule inference tasks.

[0089] Because cluster nodes are often equipped with multiple CPU cores and one or more GPUs simultaneously, the scheduling methods of online inference services in related technologies have the following main problems: coarse scheduling granularity and difficulty in accurately controlling computing resources. For example, the scheduler mainly schedules based on resource request volume and node remaining resources, making it difficult to perform fine-grained control over the allocation of model inference tasks between CPUs and GPUs. Furthermore, there is a lack of real-time awareness and loop control of latency metrics. Related technologies often rely on static configurations (e.g., fixed number of replicas, fixed GPU configuration), making it difficult to dynamically adjust based on real-time request load, CPU and GPU utilization, and queue length, easily leading to timeouts during peak periods and resource waste during off-peak periods. Moreover, data migration and cold start overhead are not incorporated into unified scheduling decisions in related technologies. Since switching execution between CPUs and GPUs involves overhead such as loading model weights and transferring feature data between main memory and GPU memory, failing to assess whether the migration cost is worthwhile can lead to frequent switching, which in turn increases overall latency. Within a cluster, multiple inference models, multiple versions, and multi-tenant tasks often coexist. Furthermore, the computational characteristics of different inference models differ significantly, making it difficult for a single static strategy to simultaneously balance latency, throughput, and resource utilization.

[0090] Therefore, to address the above-mentioned problems, the scheduling method in this embodiment can comprehensively consider task characteristics, node status, and network and data overhead at the cluster and service levels, and realize dynamic scheduling of inference tasks between CPU and GPU, thereby improving the overall resource utilization while meeting latency constraints.

[0091] This method introduces inference task awareness, node status monitoring, and a unified cost model into the cluster to dynamically decide the execution location of inference requests between CPU and GPU, achieving the following goals: improving GPU utilization and overall throughput while meeting interface latency constraints; automatically adjusting the ratio of CPU and GPU usage during traffic fluctuations to reduce resource waste; and reducing additional latency caused by model cold starts and data migration.

[0092] The scheduling method is as follows: Figure 2 As shown, the process includes the following steps: collecting inference requests and model feature information, collecting cluster node status information, constructing a unified cost model, making scheduling decisions, implementing scheduling, and providing feedback on execution results and adaptive model updates.

[0093] Specifically, inference request and model feature information is collected. By collecting information such as model identifier, model version, input data size, target latency level, and request source for each inference request in the entry gateway, and combining it with the model's pre-configuration or offline statistics, inference task information is formed.

[0094] Collect cluster node status information. For example, you can periodically collect information such as CPU utilization, GPU utilization, GPU memory usage, number of inference requests being processed, request queue length, and historical average latency of each node through a custom controller or monitoring component to form node status information.

[0095] Constructing a unified cost model allows for the estimation of the execution cost of inference requests on CPU and GPU instances based on inference task information and node state information that reflect task characteristics. Execution costs may include, but are not limited to: processing time, queuing time, model weight loading time, input data transfer time between main memory and video memory, and cross-node network transfer time.

[0096] The scheduler makes scheduling decisions, such as selecting the execution path with the shortest expected completion time and that meets latency constraints for each inference request based on a unified cost model. The execution path includes: executing on the local CPU instance, executing on the GPU instance on the local node, or routing the request to the GPU instance on other nodes.

[0097] Implement scheduling, such as routing requests at the gateway or service layer to select the appropriate CPU or GPU service. Alternatively, use a custom scheduler or scheduling extension. Dynamically adjust the number of replicas and resource quotas for CPU and GPU services based on long-term statistical results.

[0098] The execution result feedback and model adaptive update, such as using the actual response time of the inference request, the CPU and GPU resources used, and the error rate to update the cost model parameters, form a closed loop and improve the accuracy of subsequent scheduling decisions.

[0099] Through the above embodiments, timeout rates and latency jitter can be reduced in online inference scenarios. By jointly sensing the SLA level, model characteristics, and node load of inference requests, interface latency is explicitly constrained in scheduling decisions. This ensures that high-priority, real-time requests receive priority access to high-performance resources such as GPUs, reducing latency and the proportion of timeout requests, and improving the stability of online services and user experience. Furthermore, by embedding a unified cost model into the service routing and scheduling process of the Kubernetes cluster, not only resource requests are considered, but also the current CPU utilization, GPU utilization, queue length, and other operating states of nodes are taken into account. This enables scheduling at the granularity of single inference requests, thereby achieving collaborative work between CPUs and GPUs across the cluster and avoiding situations where some GPU nodes are severely idle while other nodes are overloaded.

[0100] Furthermore, the unified cost model in this embodiment covers model loading time, input data copying time between main memory and GPU memory, and cross-node network transmission time. It can identify scenarios where migration costs outweigh acceleration benefits, thereby avoiding ineffective cross-node or cross-device migrations. This reduces end-to-end latency jitter caused by frequent migrations and cold starts, improving the overall service time determinism. By differentially scheduling requests for different models and SLA levels, GPUs can handle computationally intensive, latency-sensitive tasks, while lightweight, non-real-time, or latency-insensitive tasks are assigned to CPUs. Under the same hardware configuration, this improves the actual effective utilization of GPUs, reduces the need to reserve large amounts of idle GPUs to meet peak latency requirements, and thus lowers the overall computing cost of the cluster.

[0101] The scheduling method in this embodiment can also adaptively adapt to complex environments with multiple models, versions, and tenants. For example, by continuously collecting and updating historical request execution data, it can dynamically correct the time estimates and scheduling parameters of different models on CPU and GPU, allowing the cost model to continuously improve with the evolution of task types and traffic characteristics. This ensures that it can maintain superior latency and resource utilization performance even under task changes and traffic fluctuations. Furthermore, it can work in conjunction with existing service discovery, monitoring and alerting, and elastic scaling components, which facilitates rapid deployment on existing cloud-native infrastructure and makes it easy to subsequently expand scheduling strategies or introduce machine learning prediction models based on task requirements.

[0102] According to an embodiment of this application, an embodiment of a scheduling system for inference tasks is also provided.

[0103] Figure 3 This is a schematic diagram of a scheduling system for inference tasks according to an embodiment of this application, such as... Figure 3 As shown, the system includes:

[0104] Client 300 connects to the scheduling platform and is used to initiate inference requests, which are used to request the execution of inference tasks.

[0105] Multiple nodes 302, deployed in cluster 400, are connected to the scheduling platform to send status information of the multiple nodes to the scheduling platform, and to execute inference tasks. Figure 3 (This example uses three nodes.)

[0106] The scheduling platform 304 is connected to the client and is used to receive inference requests sent by the client, obtain inference task information and status information corresponding to the inference request, predict at least one execution path of the inference request in the cluster based on the inference task information and status information, determine the execution cost of at least one execution path, determine the target execution path from at least one execution path based on the execution cost, schedule the nodes on the target execution path to execute the inference task, and obtain the execution result of the inference task.

[0107] The aforementioned client can refer to a terminal or application system that initiates inference requests, such as an autonomous driving perception module, a recommendation engine front-end, or a speech recognition program. The client can involve network services or internal microservices. The client can provide users with an inference operation terminal to trigger inference tasks and receive execution results. The client can generate inference requests and consume results.

[0108] The aforementioned scheduling platform can allocate inference service instances to appropriate nodes. The scheduling platform can include, but is not limited to, native schedulers, custom schedulers, and scheduling extension plugins. The scheduling platform can complete the initial deployment based on resource requests, affinity, and other rules, ensuring high availability and load balancing.

[0109] The scheduling platform in this scheduling system receives inference requests through a gateway, extracts the corresponding inference task information and cluster status information based on the inference request, dynamically predicts multiple feasible execution paths, and quantifies the execution cost of each path, including resource consumption, scheduling latency, network transmission overhead, and energy efficiency ratio. Then, it selects the target execution path, and finally, multiple nodes in the cluster execute the inference task. This breaks through the limitations of traditional static or simple load balancing scheduling, and realizes the transformation from passive allocation to active path selection driven by multi-dimensional costs, achieving a synergistic improvement in inference efficiency and system resource utilization.

[0110] Optionally, the scheduling platform includes: a gateway for receiving inference requests; a decision module for obtaining inference task information and status information corresponding to the inference request based on the inference request, predicting at least one execution path of the inference request in the cluster based on the inference task information and status information, determining the execution cost of at least one execution path, and determining the target execution path in at least one execution path based on the execution cost; and a scheduling module for scheduling nodes on the target execution path to execute the inference task and obtain the execution result of the inference task.

[0111] The gateway mentioned above can serve as a unified entry point for cluster ingress traffic. Gateways can be reverse proxies, service meshes, edge gateways, etc. Gateways can receive external requests, perform authentication and authorization, route requests, and extract task features. Gateways can also forward inference requests to the scheduling module or directly distribute them to CPU and GPU services.

[0112] The aforementioned decision-making module is the core service of the scheduling platform. It enables scheduling decisions and instructs nodes to perform inference tasks. The decision-making module can operate based on rules or machine learning models. It comprehensively evaluates inference task information and state information, predicts end-to-end latency costs under different execution paths, and comprehensively weighs migration overhead, cold start costs, and resource contention to output execution path suggestions, making scheduling decisions perceptive, predictive, and adaptive.

[0113] The aforementioned scheduling module enables node scheduling and serves as the component connecting decision-making intent with resource execution. The scheduling module transforms the target execution path output by the decision-making module into controllable operations on the actual resources of the cluster. This can be achieved through a controller or scheduling extender. The scheduling module can also maintain the states of preheating node caches, fault isolation lists, and dynamic adjustments to resource quotas, ensuring that scheduling behavior is consistent with the actual state of the cluster. The scheduling module is the final execution unit that ensures the scheduling strategy is implemented in a real cloud environment.

[0114] After receiving inference requests through the gateway, the scheduling platform dynamically analyzes and predicts multiple possible execution paths based on the inference task information corresponding to the request and the real-time status information of the cluster nodes. It comprehensively evaluates the execution cost of each execution path in terms of resource consumption, latency, energy consumption, and hardware compatibility, and then selects the target execution path from the candidate paths. The scheduling module then precisely schedules the matching CPU or GPU nodes on the execution path to execute the inference task. This enables dynamic decision-making on computing resource allocation based on task characteristics and the actual operating environment of the nodes, breaking through the limitations of traditional static scheduling rules and effectively improving the cluster resource utilization and inference service response efficiency.

[0115] Optionally, the scheduling platform may also include a monitoring module, which connects to multiple nodes to collect the operating parameters of the multiple nodes in order to evaluate the target execution path.

[0116] The scheduling platform can establish connections with multiple nodes in the cluster through the newly added monitoring module, and collect real-time operating parameters such as CPU and GPU resource utilization, temperature, load balancing status, and task queuing status of each node. This structured data is then used directly to evaluate the execution cost of inference tasks on the CPU or GPU path, thus forming a closed-loop mechanism from data collection to path evaluation. This makes scheduling decisions no longer dependent on static or lagging information, but rather on the execution path dynamically quantified based on the current real load of the nodes, effectively improving the accuracy and response speed of inference task scheduling, and ultimately achieving efficient utilization of cluster resources and dynamic adjustment of inference latency.

[0117] The technical solution proposed in this application is described below with reference to an optional embodiment. This application proposes a rule-based hybrid scheduling method. The scheduling component used in this hybrid scheduling method can be deployed on a cluster. The scheduling component may include, but is not limited to, a gateway that receives external inference requests; inference services such as CPU inference services and GPU inference services; a scheduling management service that makes scheduling decisions by reading monitoring data and service metrics; and a monitoring and metric collection component that collects real-time metrics of nodes and services.

[0118] This hybrid scheduling method can adopt a scheduling architecture as shown in Figure 4. Figure 4 As shown, after the client initiates an inference task, the output is sent to the scheduling platform, which instructs the cluster to schedule GPU nodes and CPU nodes.

[0119] The hybrid scheduling method is as follows: Figure 5 As shown, the process includes the following steps: inference task feature modeling, node state acquisition, unified cost model design, scheduling decision and request routing, and implementation of scheduling feedback and adaptive adjustment.

[0120] Inference task feature modeling refers to the process where, after an inference request arrives at the gateway, the gateway adds task features to the request context, including: model identifier, version number, latency level, input data size, and task identifier, to differentiate between different tenant strategies. For each model identifier and version number, the system can pre-calculate the average processing time and variance on CPU and GPU offline for initial cost model estimation.

[0121] Node status collection refers to periodically collecting data on each node by deploying a monitoring intelligent agent or using a monitoring system, including: CPU utilization, number of available cores; GPU utilization, memory usage ratio, temperature, etc.; the current number of queued requests and recent latency for each inference instance; and whether the model has been fully loaded into the GPU's memory. Node status data can be periodically reported to the scheduling management service, or retrieved by the scheduling management service periodically.

[0122] The unified cost model design refers to the system calculating the execution cost on different execution paths for a given scheduled request. For example, the execution cost on the CPU instance of this node is calculated using the following formula:

[0123] ;

[0124] in, As the first execution cost, The processing time required for the model to perform inference on the local CPU can be inferred from historical statistics and current utilization. , The time it takes for an inference request to wait in the local CPU inference service queue for processing can be estimated from the current queue length and the average processing time. .

[0125] The execution cost on the GPU instance of this node is calculated using the following formula:

[0126] ;

[0127] in, As the second execution cost, This refers to the time required to load model weights from storage to the video memory of the node device in the target execution path, when the model has been preloaded. It can be approximated as 0. The processing time for a model to perform inference on a GPU can be inferred from historical statistics and current utilization. , The waiting time for an inference request in the GPU instance queue in the target execution path can be estimated from the current queue length and average processing time, or from the current backlog of inference requests. .

[0128] The execution cost on the remote node GPU instance is calculated using the following formula:

[0129] ;

[0130] in, As the third execution cost, The time consumed by requesting data to be transmitted over the network between different nodes can be estimated based on the network latency between nodes and the size of the input data. , This refers to the time required to load model weights from storage to the video memory of the node device in the target execution path. This refers to the processing time for the model to perform inference on the GPU. The waiting time for an inference request in the GPU instance queue in the target execution path.

[0131] The scheduling decision compares the execution cost C of each execution path and combines it with the pre-defined Service Level Agreement (SLA) constraints to select the execution path that meets the latency requirements and has the lowest cost. For inference requests with strict SLAs, a penalty for violation is added to the cost. If the estimated completion time of an execution path exceeds the SLA, the cost increases significantly or the path is eliminated.

[0132] Scheduling decisions and request routing refer to the scheduling management service selecting the target execution path from candidate execution paths based on the cost model, generating a routing decision. For example, the gateway might call the scheduling management service's interface to obtain the routing target for the current request. Alternatively, it can use service discovery and load balancing to directly forward inference requests to the corresponding nodes. Or, it can maintain a pre-warming node list for the GPU service and select nodes where existing models are loaded.

[0133] Feedback and adaptive adjustment refer to the process whereby, after processing an inference request, the inference service reports the actual processing time, whether a timeout occurred, and node resource usage to the scheduling management service. Based on the feedback data, the scheduling management service updates the following: such as updating the average processing time and fluctuation range of different models on CPU and GPU; dynamically adjusting the number of GPU service replicas and resource quotas based on periodic statistics; or fine-tuning the coefficients in the cost model to improve prediction accuracy.

[0134] Alternatively, the cost model can also be implemented using a prediction model. For example, task characteristics of historical inference requests (model identifier, input size, SLA, etc.) and node states (such as CPU and GPU utilization, queue length, etc.) can be used as input features, while actual execution time, timeout status, and resource consumption can be used as labels to train a regression or classification model to predict expected latency on different execution paths. This prediction model can be deployed online, and the prediction interface can be called during each scheduling session to select the target execution path.

[0135] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0136] According to an embodiment of this application, an embodiment of a scheduling device for inference tasks is provided. It should be noted that the device can be used to execute the above-described scheduling method for inference tasks.

[0137] Figure 6 This is a schematic diagram of a scheduling device for inference tasks according to an embodiment of this application, such as... Figure 6 As shown, the device includes:

[0138] The acquisition module 50 is used to acquire the inference task information corresponding to the inference request and the status information of multiple nodes in the cluster when an inference request is received. The inference request is used to request the execution of the inference task.

[0139] The prediction module 52 is used to predict at least one execution path of the inference request in the cluster based on inference task information and state information, and to determine the execution cost of at least one execution path.

[0140] The determination module 54 is used to determine the target execution path from at least one execution path based on the execution cost.

[0141] The scheduling module 56 is used to schedule nodes on the target execution path to execute inference tasks and obtain the execution results of the inference tasks.

[0142] Optionally, the acquisition module is also used to: determine the model identifier of the inference model, the evaluation index of the inference task, and the source information of the inference request based on the inference request, wherein the inference model is deployed on multiple nodes and is used to execute the inference task; obtain the historical operation information of the inference model based on the model identifier; and obtain the inference task information based on the model identifier, evaluation index, source information, and historical operation information.

[0143] Optionally, the prediction module is further configured to: predict at least one execution path based on inference task information and state information; and, when at least one execution path includes a first execution path, a second execution path, and a third execution path, determine, based on the inference task information and state information, the first execution cost corresponding to the first execution path, the first execution cost corresponding to the second execution path, and the third execution cost corresponding to the third execution path, respectively, wherein the first execution path is a CPU instance path of the local node, the second execution path is a GPU instance path of the local node, and the third execution path is a cross-node GPU instance path; preferably, the prediction module is further configured to: determine the processing time and the waiting time based on the inference task information and state information, wherein the processing time is the time required for the local node to execute the inference task, and the waiting time is the time for the inference request to wait for processing in the node queue. The prediction module is further configured to: determine a first execution cost based on the sum of processing time and waiting time; preferably, the prediction module is also configured to: determine model loading time, processing time, and waiting time based on inference task information and status information, wherein processing time is the time required for the node to execute the inference task, and waiting time is the time for the inference request to wait for processing in the node queue; determine a second execution cost based on the sum of model loading time, processing time, and waiting time; preferably, the prediction module is also configured to: determine cross-node transmission time, model loading time, processing time, and waiting time based on inference task information and status information, wherein processing time is the time required for the node to execute the inference task, and waiting time is the time for the inference request to wait for processing in the node queue; and determine a third execution cost based on the sum of cross-node transmission time, model loading time, processing time, and waiting time.

[0144] Optionally, the determining module is further configured to: determine the lowest execution cost among the execution costs; and, provided that the execution cost is within the threshold range corresponding to the task processing level corresponding to the inference request, determine the execution path corresponding to the lowest execution cost as the target execution path.

[0145] Optionally, the determination module is also used to: input inference task information and state information as input features, and execution cost as a label for the execution path, input them into the prediction model, predict the delay time of at least one execution path through the prediction model, and determine the target execution path based on the delay time.

[0146] This application also provides an electronic device 90, please refer to... Figure 7 It includes a memory 910 and a processor 920, wherein the memory 910 is used to store computer programs; and the processor 920 is used to execute the programs stored in the memory 910 to implement the methods in the various embodiments of this application.

[0147] Methods in Various Embodiments of This Application Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods in various embodiments of this application.

[0148] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0149] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0150] Embodiments of this application also provide a computer program that, when executed by a processor, implements the methods described in the various embodiments of this application.

[0151] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0152] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0153] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0155] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0156] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A scheduling method for inference tasks, characterized in that, include: Upon receiving an inference request, obtain the inference task information corresponding to the inference request, as well as the status information of multiple nodes in the cluster, wherein the inference request is used to request the execution of the inference task; Based on the inference task information and the state information, predict at least one execution path of the inference request in the cluster, and determine the execution cost of the at least one execution path; Based on the execution cost, a target execution path is determined from the at least one execution path; The nodes on the target execution path are scheduled to execute the inference task, and the execution result of the inference task is obtained.

2. The method according to claim 1, characterized in that, Obtaining the inference task information corresponding to the inference request includes: Based on the inference request, the model identifier of the inference model, the evaluation index of the inference task, and the source information of the inference request are determined, wherein the inference model is deployed on the multiple nodes and is used to execute the inference task; Based on the model identifier, obtain the historical operation information of the inference model; The inference task information is obtained based on the model identifier, the evaluation index, the source information, and the historical operation information.

3. The method according to claim 1, characterized in that, Based on the inference task information and the state information, predict at least one execution path for the inference request in the cluster, and determine the execution cost of the at least one execution path, including: Based on the inference task information and the state information, predict the at least one execution path; When the at least one execution path includes a first execution path, a second execution path, and a third execution path, based on the inference task information and the state information, the first execution cost corresponding to the first execution path, the first execution cost corresponding to the second execution path, and the third execution cost corresponding to the third execution path are determined respectively, wherein the first execution path is the CPU instance path of this node, the second execution path is the GPU instance path of this node, and the third execution path is the cross-node GPU instance path; Preferably, determining the first execution cost corresponding to the first execution path based on the inference task information and the state information includes: Based on the inference task information and the status information, the processing time and the waiting time are determined, wherein the processing time is the time required for this node to execute the inference task, and the waiting time is the time for the inference request to wait for processing in the node queue; The first execution cost is determined based on the sum of the processing time and the waiting time; Preferably, determining the second execution cost corresponding to the second execution path based on the inference task information and the state information includes: Based on the inference task information and the status information, the model loading time, processing time, and waiting time are determined, wherein the processing time is the time required for this node to execute the inference task, and the waiting time is the time for the inference request to wait for processing in the node queue; The second execution cost is determined based on the sum of the model loading time, the processing time, and the waiting time. Preferably, determining the third execution cost corresponding to the third execution path based on the inference task information and the state information includes: Based on the inference task information and the status information, the cross-node transmission time, model loading time, processing time, and waiting time are determined, wherein the processing time is the time required for this node to execute the inference task, and the waiting time is the time for the inference request to wait for processing in the node queue. The third execution cost is determined based on the sum of the cross-node transmission time, the model loading time, the processing time, and the waiting time.

4. The method according to claim 1, characterized in that, Determining a target execution path from the at least one execution path based on the execution cost includes: Determine the lowest execution cost among the aforementioned execution costs; If the execution cost is within the threshold range corresponding to the task processing level of the inference request, the execution path with the lowest execution cost is determined as the target execution path.

5. The method according to claim 1, characterized in that, Determining a target execution path from the at least one execution path based on the execution cost includes: The inference task information and the state information are used as input features, and the execution cost is used as the label of the execution path. These are input into the prediction model, and the prediction model is used to predict the delay time of the at least one execution path. The target execution path is determined based on the delay time.

6. A scheduling system for inference tasks, characterized in that, include: A client, connected to the scheduling platform, is used to initiate inference requests, wherein the inference requests are used to request the execution of inference tasks; Multiple nodes, deployed in a cluster, are connected to the scheduling platform and are used to send status information of the multiple nodes to the scheduling platform, as well as to execute the inference task; A scheduling platform, connected to the client, is used to receive the inference request sent by the client, obtain the inference task information corresponding to the inference request and the status information, predict at least one execution path of the inference request in the cluster based on the inference task information and the status information, determine the execution cost of the at least one execution path, determine a target execution path from the at least one execution path based on the execution cost, schedule the nodes on the target execution path to execute the inference task, and obtain the execution result of the inference task.

7. The scheduling system according to claim 6, characterized in that, The scheduling platform includes: A gateway is used to receive the inference request; The decision module is used to obtain the inference task information corresponding to the inference request and the status information based on the inference request; predict at least one execution path of the inference request in the cluster based on the inference task information and the status information; determine the execution cost of the at least one execution path; and determine the target execution path among the at least one execution path based on the execution cost. The scheduling module is used to schedule nodes on the target execution path to execute the inference task and obtain the execution result of the inference task.

8. The scheduling system according to claim 7, characterized in that, The scheduling platform also includes: The monitoring module is connected to the multiple nodes and is used to collect the operating parameters of the multiple nodes in order to evaluate the target execution path.

9. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the method according to any one of claims 1 to 5.

Images