An edge-cloud collaborative AI computing power and detection task integrated scheduling method

Abstract

The application discloses an AI computing power and detection task integrated scheduling method based on end-edge-cloud cooperation, and belongs to the technical field of detection task scheduling. The application comprises the following steps: constructing a global task queue, constructing a directed acyclic graph of a detection task, adaptively dividing and dynamically distributing, batch processing scheduling, and degradation execution control. The application collects the end-side load, edge node parameters and task priority in real time, constructs a unified state vector and a global task queue, analyzes the detection task into a directed acyclic graph and labels the priority, precision lower limit, time delay upper limit, power consumption upper limit and data volume to form a quality of service constraint model, divides the AI model into end, edge and cloud sub-models according to the sub-task time delay threshold, and dynamically adjusts the division point and distribution path according to the network quality, dynamically adjusts the node computing power capacity according to the queue blocking rate, determines the execution order and queue priority according to the batch processing size, triggers cross-layer migration and degradation execution when the network is interrupted or the edge fails, and reschedules the remaining sub-tasks.

Description

Technical Field

[0001] This invention relates to the field of detection task scheduling technology, and in particular to an integrated scheduling method for AI computing power and detection tasks that combines edge-cloud collaboration. Background Technology

[0002] In the edge-cloud collaborative architecture, edge nodes are limited by computing resources and are prone to problems such as excessive inference latency and decreased detection accuracy when handling high-difficulty, high-concurrency detection tasks. While the cloud has sufficient computing power, it is constrained by network transmission latency and queue congestion in high-parallel request scenarios.

[0003] Most existing edge-cloud collaborative AI (Artificial Intelligence) computing power and detection task scheduling solutions adopt a design approach that decouples computing power resource scheduling from AI detection tasks. Although cloud computing resources are abundant, they are constrained by network links. Existing scheduling strategies are mostly static task allocation or resource allocation based on only a single-dimensional indicator. It is difficult to perceive in real time the remaining computing power, memory usage, network bandwidth, link jitter, and multi-dimensional status information such as the priority, accuracy requirements, and latency thresholds of currently queued detection tasks of each node. This easily leads to load imbalance problems such as some edge nodes being overloaded, causing inference timeouts, missed detections, and false detections, while the computing power resources of other nodes are idle and wasted. AI detection tasks have different task granularities, ranging from lightweight preliminary screening tasks that require millisecond-level local response to high-precision detection tasks that rely on large models for fine-grained defect identification and multi-frame correlation analysis. Existing technologies cannot dynamically divide AI inference models, distribute sub-tasks, and orchestrate pipelines based on the business indicators of detection tasks. In weak networks or network interruptions, the problem persists. In scenarios like these, the lack of dynamic task migration and degradation execution mechanisms means that when network quality deteriorates, the transmission of large amounts of raw image and video stream data to the cloud can cause bandwidth congestion and a surge in transmission latency, making it impossible to balance detection real-time performance and detection accuracy. Existing scheduling frameworks lack a multi-objective joint optimization mechanism for detection task inference accuracy, end-to-end latency, device power consumption, and computing power utilization. In the scheduling decision-making process, they often prioritize one aspect over another, making it difficult to meet the differentiated QoS requirements of different detection tasks. Furthermore, the lack of a unified closed-loop feedback mechanism for computing power status and detection task execution status makes it impossible to continuously iterate and adjust scheduling strategies based on detection result feedback and changes in model inference performance. Task scheduling decisions between layers are independent, and scheduling silos exist in the edge-cloud collaborative inference link. When task execution fails or node failures occur, it is difficult to quickly complete the cross-layer migration and recovery of detection tasks. In large-scale concurrent detection scenarios, a large number of heterogeneous AI inference tasks concurrently compete for resources, which can easily lead to task queue blocking and inference performance jitter, making it difficult to achieve a globally integrated optimal match between the computing power resources of the edge-cloud three layers and multiple types of AI detection tasks. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an integrated scheduling method for AI computing power and detection tasks in edge-cloud collaboration. This method can construct a unified state vector and a global task queue by real-time collection of edge-side load, edge node parameters, and task priorities; it parses detection tasks into directed acyclic graphs and labels them with priorities, accuracy lower limits, latency upper limits, power consumption upper limits, and data volume to form a service quality constraint model; it divides the AI ​​model into edge, cloud, and terminal sub-models based on sub-task latency thresholds, and dynamically adjusts the division points and distribution paths according to network quality; it dynamically adjusts node computing power capacity based on queue blocking rate, and determines the execution order and queue priority according to batch size; and it triggers cross-layer migration and degraded execution in the event of network interruption or edge failure, rescheduling unfinished remaining sub-tasks.

[0005] On the one hand, a method for integrated scheduling of AI computing power and detection tasks in edge-cloud collaboration is provided, which includes: Step S1, real-time collection of load parameters of edge devices and edge node parameters, and acquisition of the priority of AI detection tasks to be scheduled, forming a unified state vector of edge-cloud and global detection task queue; Step S2, parsing each AI detection task into a directed acyclic graph, and labeling each sub-task node with priority, accuracy lower limit, latency upper limit, power consumption upper limit and data volume, forming a directed acyclic graph of detection tasks with service quality constraints; Step S3, determining the splitting point of the AI ​​model according to the latency threshold of each sub-task in the directed acyclic graph of AI detection tasks, and dividing the AI ​​model into segments. The model is divided into end-side sub-model, edge sub-model, and cloud sub-model, and the model splitting point and sub-task distribution path are dynamically adjusted when the network quality changes; Step S4: The computing power capacity of each node is dynamically adjusted according to the task queue blocking rate to suppress task queue blocking. After dynamically adjusting the computing power capacity of each node, the execution order and queue priority are determined based on the batch size; Step S5: During the execution order and queue priority process, when a network interruption or edge node failure is detected, the cross-layer dynamic migration and degraded execution control of the detection task is triggered, the cloud sub-task is migrated to the edge node or end-side device, and the remaining unfinished sub-tasks in the directed acyclic graph of the detection task are rescheduled.

[0006] Beneficial effects The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: By constructing a unified state vector across edge and cloud and a global detection task queue, real-time, multi-dimensional perception of the computing power status, network status, and detection task requirements of edge devices, edge nodes, and cloud servers was achieved. By parsing detection tasks into a directed acyclic graph with service quality constraints, each sub-task node was precisely labeled with priority, accuracy lower limit, latency upper limit, power consumption upper limit, and data volume, achieving fine-grained mapping of detection task business metrics to scheduling constraints. Model splitting points were adaptively determined based on sub-task latency thresholds, enabling millisecond-level lightweight screening tasks to be executed on the edge, medium-complexity tasks to be executed at the edge, and high-precision large-model inference tasks to be executed in the cloud. The splitting points and distribution paths were dynamically adjusted when network quality changed, effectively avoiding bandwidth congestion and transmission latency spikes in weak network scenarios. The computing power capacity of each node was dynamically adjusted by the task queue blocking rate, enabling overloaded nodes to be managed. The closed-loop adaptive matching of computing power expansion and idle node computing power reduction suppresses task queue blocking and inference performance jitter; based on batch size to determine execution order and queue priority, adaptive scheduling is achieved to prioritize the execution of small batches of high-urgency tasks and aggregate the execution of large batches of low-urgency tasks, improving inference throughput and computing power utilization; when the network is interrupted or the edge node fails, cross-layer dynamic migration and degraded execution control are triggered to migrate cloud subtasks to edge nodes or end-side devices and reschedule unfinished remaining subtasks, ensuring the continuous execution of detection tasks in weak network or fault scenarios; through closed-loop feedback and cross-layer collaborative recovery mechanisms, the islands of edge-cloud layered scheduling are eliminated, enabling the system to gradually recover to the globally integrated optimal scheduling state after fault recovery, thus improving the overall utilization of computing power resources, load balancing, end-to-end completion rate of detection tasks, and system reliability of the three layers of edge-cloud computing power. Attached Figure Description

[0007] Figure 1 A flowchart illustrating an integrated scheduling method for AI computing power and detection tasks in edge-cloud collaboration, as provided in this application embodiment; Figure 2 The flowchart shows the sequence and queue priority of an integrated scheduling method for AI computing power and detection tasks in edge-cloud collaboration, as provided in the embodiments of this application. Detailed Implementation

[0008] The following provides explanations for some of the terms used in this application. It should be noted that these explanations are for the convenience of those skilled in the art and do not constitute a limitation on the scope of protection claimed in this application.

[0009] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0010] like Figure 1The diagram shown is a flowchart of an integrated scheduling method for AI computing power and detection tasks in edge-cloud collaboration, provided in an embodiment of this application. The method includes the following steps: Step S1: Real-time collection of load parameters of edge devices and edge node parameters, and acquisition of the priority of AI detection tasks to be scheduled, forming a unified state vector of edge and cloud and a global detection task queue; among which, load parameters include at least computing power reserve, memory usage, power consumption, and storage reserve; edge node parameters include at least NPU, CPU, GPU utilization, memory usage, queue length and power consumption, and cloud server available computing power, storage capacity, task queue and service instance status.

[0011] Status acquisition agents are deployed on edge devices, edge nodes, and cloud servers. Each agent is configured with a unified clock source, acquisition cycle, event triggering conditions, data reporting format, and status identifier. Event triggering conditions include changes in node computing power exceeding preset thresholds, changes in memory usage exceeding preset thresholds, changes in power consumption exceeding preset thresholds, changes in task queue length exceeding preset thresholds, changes in network link quality exceeding preset thresholds, or the arrival and completion of detection tasks. Each agent collects local status information in real time according to the acquisition cycle or event triggering conditions, and appends a collection timestamp, node identifier, node hierarchy identifier, and status reliability identifier. Real-time acquisition of edge device load parameters is also performed. These parameters include at least the edge device's computing power reserve, memory usage, power consumption, storage reserve, battery level, acquisition load, processor utilization, neural network processing unit utilization, task queue length, number of currently running detection tasks, data acquisition rate, data cache usage, and edge device online status. For scenarios with multiple edge devices, load parameters are collected for each device separately and stored in association according to device identifier and collection timestamp. Real-time collection of edge node parameters, including at least the utilization rate of neural network processing units, CPU utilization, GPU utilization, memory usage, storage capacity, task queue length, power consumption, temperature, cached model fragments, inference executor status, loaded model version, number of currently concurrent inference tasks, network interface status, and edge node online status. When an edge node contains multiple computing units, the utilization rate, computing power capacity, memory usage, and power consumption of each computing unit are collected separately and aggregated according to the computing unit identifier. Real-time collection of cloud server parameters, including at least the available computing power, storage capacity, task queue length, service instance status, GPU utilization, CPU utilization, memory usage, power consumption, model version, inference service concurrency, task queuing latency, service health status, and cloud server online status. When the cloud server provides computing power in the form of containers, virtual machines, or service instances, the computing power quota, actual utilization rate, task queue status, and health status of each service instance are collected separately. Real-time collection of network status parameters for edge links, edge-cloud links, and edge-cloud links. Network status parameters include at least available bandwidth, round-trip latency, link jitter, packet loss rate, connection stability, link switching status, network interruption status, and transmission queue length. For multi-path or multi-link scenarios, network status parameters for each link are collected separately and associated and stored according to link identifier, source node identifier, and target node identifier.Obtain the attributes and priorities of the AI ​​detection tasks to be scheduled. The attributes of each task should include at least the task identifier, business type, business level, whether it is safety-critical, end-to-end accuracy requirement, end-to-end latency threshold, task arrival time, data source type, original data volume, data privacy requirements, model dependencies, subtask dependencies, migration allowance flag, degradation allowance flag, and a candidate set of degradation models. Calculate the priority of each AI detection task based on its business level, safety-criticality, end-to-end latency threshold, end-to-end accuracy requirement, and the number of downstream dependent nodes. Classify the priorities as high, medium, and low, or represent them as normalized numerical values. Preprocess the collected data, including at least timestamp alignment, unit normalization, outlier removal, missing value imputation, duplicate data deduplication, state credibility assessment, and data format standardization. Mark state data that has not been updated for a preset time as expired. Mark data with a state credibility lower than a preset credibility threshold as low-credibility, and reduce its weight or trigger re-collection during subsequent state vector construction. Based on the preprocessed load parameters of the edge devices, edge node parameters, cloud server parameters, network status parameters, and attributes of the AI ​​detection tasks to be scheduled, a unified edge-cloud state vector is constructed. This unified state vector includes at least node status sub-vectors, link status sub-vectors, task requirement sub-vectors, and a timestamp field. The node status sub-vectors encode the computing power reserve, memory usage, power consumption, storage reserve, task queue length, inference executor status, and online status of each node at three levels: edge devices, edge nodes, and cloud servers. The link status sub-vectors encode the available bandwidth, round-trip latency, link jitter, packet loss rate, and connection stability of the edge-cloud link, the edge-cloud link, and the end-to-end link. The task requirement sub-vectors encode the priority, accuracy requirements, latency threshold, data volume, portability, and degradability of the AI ​​detection tasks to be scheduled. These sub-vectors are then concatenated or weighted and merged according to a unified dimensional order to form the unified edge-cloud state vector, which is periodically updated according to the collection timestamp. A global detection task queue is constructed based on the priority, end-to-end latency threshold, arrival time, and dependencies of the AI ​​detection tasks to be scheduled. The global detection task queue includes at least a ready queue, a waiting queue, a blocked queue, a migration queue, and a degraded queue. Tasks in each queue are arranged in order of priority from high to low, latency threshold from low to high, and arrival time from early to late. For detection tasks with directed acyclic graph dependencies, their successor subtasks are only placed in the ready queue after all their predecessor subtasks have been completed. Subtasks that cannot be executed immediately due to network interruption, node failure, or insufficient resources are placed in the blocked queue or the migration queue. Subtasks that trigger degraded execution control are placed in the degraded queue.The unified state vector of edge-cloud and global detection task queue is updated and output synchronously. When the status of edge devices, edge nodes, cloud servers or network links changes, the unified state vector of edge-cloud is updated in real time. When the AI ​​detection task to be scheduled arrives, completes, fails, migrates or degrades, the global detection task queue is updated in real time. The updated unified state vector of edge-cloud and global detection task queue is output to the subsequent AI model segmentation, dynamic adjustment of computing power, batch processing scheduling, cross-layer migration and degrade execution control modules, and the scheduling results feedback of the above modules are received, forming a closed loop of state awareness, task queuing, scheduling decision and feedback update.

[0012] Step S2: Each AI detection task is parsed into a directed acyclic graph (DAG), and each subtask node is labeled with priority, accuracy lower limit, latency upper limit, power consumption upper limit, and data volume to form a DAG of detection tasks with quality of service constraints. The nodes of the DAG represent at least one of the following subtasks: lightweight preliminary screening subtask, fine-grained defect identification subtask, multi-frame correlation analysis subtask, feature extraction subtask, data preprocessing subtask, or result fusion subtask. The edges of the DAG represent the data flow dependency, model dependency, and execution order dependency between subtasks.

[0013] It is important to understand that the specific process of marking each subtask node with priority, lower limit of precision, upper limit of latency, upper limit of power consumption, and data volume is as follows: Obtain the end-to-end service quality requirements and task attributes for each AI detection task. The end-to-end service quality requirements and task attributes should include at least the task identifier, business type, business level, whether it is safety critical, end-to-end accuracy requirements, and end-to-end latency threshold. Each AI detection task is parsed as a directed acyclic graph (DAG). Based on the input data type, input tensor shape, number of channels, bit width, frame count, resolution, encoding bitrate, compression ratio, and feature map size of the sub-task nodes, the input data volume, output data volume, and cross-layer transmission data volume requiring cross-device, edge, and cloud transmission are calculated and written into the data volume attribute of the sub-task nodes. For video stream detection tasks, the data volume per unit time is calculated according to the frame extraction rate, single-frame resolution, single-frame bit depth, and encoding compression ratio. The directed acyclic graph is G=(V V = {v_1, v_2, ..., v_n} represents the set of subtask nodes, and each subtask node v_j is at least one of the following: data preprocessing subtask, lightweight preliminary screening subtask, feature extraction subtask, fine-grained defect identification subtask, multi-frame correlation analysis subtask, result fusion subtask, or post-processing subtask. E = {e_ij} represents the set of directed edges, and the directed edge e_ij represents the data flow dependency, model dependency, or execution order dependency between subtask node v_i and subtask node v_j. Based on the end-to-end accuracy requirements, the position of the sub-task nodes in the directed acyclic graph, and the series and parallel relationships between the sub-task nodes, the end-to-end accuracy requirements are decomposed to each sub-task node to obtain the lower limit of accuracy for each sub-task node, and the fusion detection accuracy of all sub-task nodes after execution according to the directed acyclic graph is not lower than the end-to-end accuracy requirements.

[0014] Each subtask node is labeled with its priority, lower limit of precision, upper limit of latency, upper limit of power consumption, and data volume, including: Based on the end-to-end latency threshold and the critical path of the directed acyclic graph, calculate the earliest start time, latest completion time, and remaining latency of the critical path for each subtask node to obtain the latency upper limit of the subtask node. Then, decompose the latency upper limit into the computation latency upper limit, the cross-layer transmission latency upper limit, and the queuing latency upper limit. Write the latency upper limit into the latency upper limit attribute of the subtask node. Obtain the battery capacity and power consumption of the terminal device, and mark the power consumption limit of the sub-task node according to the candidate execution location and whether it is portable; when the sub-task node can be executed in multiple locations in the terminal device, edge node and cloud server, mark the terminal power consumption limit, edge power consumption limit and cloud power consumption limit respectively; write the power consumption limit to the power consumption limit attribute of the sub-task node. Based on the service level to which the subtask node belongs, the priority of the subtask node is calculated. Based on the cross-layer data transmission volume, latency upper limit, power consumption upper limit, and accuracy lower limit, a set of candidate execution locations is marked for the subtask node. When the cross-layer data transmission volume is large and the latency upper limit is less than the preset latency threshold, the candidate execution location set will preferentially include edge devices. When the accuracy lower limit is higher than the preset accuracy threshold and the latency upper limit is greater than the preset latency threshold, the candidate execution location set will include cloud servers. When the subtask node involves data privacy requirements, the candidate execution location set is restricted to edge devices or edge nodes. The priority satisfies: Pr(v_j) = w1L_business(v_j) + w2(T_ref / upper limit of latency) + w3(lower limit of precision / A_ref) + w4N_down(v_j) + w5I_critical(v_j), where w1, w2, w3, w4, and w5 are preset weights, L_business(v_j) is the business level, T_ref is the reference latency, A_ref is the reference precision, N_down(v_j) is the number of downstream dependent nodes of subtask node v_j, and I_critical(v_j) is the critical path indicator of whether subtask node v_j is located on the critical path. Pr(v_j) is divided into high priority, medium priority, and low priority, or written into the priority attribute of subtask node v_j in the form of a normalized value. The candidate execution location set is a subset of the set composed of end devices, edge nodes, and cloud servers. The nodes are labeled with transferable flags, allowed degradation flags, and a set of candidate degradation models. The priority, lower limit of accuracy, upper limit of latency, upper limit of power consumption, data volume, candidate execution location set, transferable flags, allowed degradation flags, and candidate set of degradation models for each subtask node are written into a directed acyclic graph (DAG) to form a DAG of detection tasks with service quality constraints, G_qos=(V,E,Q), where Q is the set of service quality attributes for subtask nodes. G_qos is then used for feasibility verification to determine whether each subtask node has an execution scheme in the candidate execution location set that satisfies the lower limit of accuracy, upper limit of latency, upper limit of power consumption, and cross-layer data transmission volume constraints. If not, the AI ​​inference model splitting points are adjusted, a degradation model is enabled, or detection task admission control is triggered. If a scheme exists, the DAG of detection tasks with service quality constraints, G_qos, is output and AI inference model adaptive splitting and cross-layer pipeline orchestration are performed.

[0015] In this embodiment, the end-to-end service quality requirements and task attribute information corresponding to each AI detection task are obtained. These end-to-end service quality requirements and task attributes at least cover task identifier, business type, business level, whether it is a security critical identifier, end-to-end accuracy requirements, and end-to-end latency threshold. Subsequently, each complete AI detection task is decomposed and parsed into a directed acyclic graph. This directed acyclic graph contains a set of subtask nodes and a set of directed edges. Each subtask node in the set of subtask nodes can specifically be any one or more of the following: data preprocessing subtask, lightweight preliminary screening subtask, feature extraction subtask, fine-grained defect identification subtask, multi-frame correlation analysis subtask, result fusion subtask, and post-processing subtask. In this combination, each directed edge in the directed edge set represents the data flow dependency, model dependency, or execution order dependency between two subtask nodes. Then, based on the input data type, input tensor shape, number of channels, bit width, frame count, resolution, encoding bitrate, compression ratio, and feature map size of each subtask node, the input and output data volumes of each subtask node are calculated. Simultaneously, the cross-layer transmission data volume that needs to be exchanged between the edge node, cloud server, and the terminal is calculated, and the calculated data volume results are written into the data volume attribute field of the corresponding subtask node. For video stream AI detection tasks, the frame extraction rate, single frame resolution, and single frame bit depth are combined. The amount of data generated per unit time is calculated based on depth and encoding compression ratio. Then, considering the overall end-to-end accuracy requirements of the AI ​​detection task, the topological position of sub-task nodes within the directed acyclic graph (DAG), and the serial and parallel relationships between sub-task nodes, the overall end-to-end accuracy requirements are decomposed layer by layer. A lower precision limit is assigned to each sub-task node to ensure that all sub-task nodes execute sequentially according to the dependencies within the DAG. The final fused output's overall detection accuracy will not be lower than the original AI detection task's end-to-end accuracy requirements. Finally, combining the end-to-end latency threshold of the AI ​​detection task, the critical path of the DAG is solved, and the results are calculated. The earliest start time, latest completion time, and remaining critical path latency of each subtask node are used to determine the latency ceiling for each subtask node. This latency ceiling is further broken down into computation latency ceiling, cross-layer transmission latency ceiling, and queuing latency ceiling. These latency-related parameters are written into the latency ceiling attribute of the corresponding subtask node. Battery capacity and power consumption parameters of the edge device are collected, and combined with the candidate executable locations of each subtask node and whether the subtask node supports task migration, a corresponding power consumption ceiling is assigned to each subtask node. When a subtask node supports simultaneous computation execution at multiple locations including edge devices, edge nodes, and cloud servers...This involves marking the power consumption limits for the subtask node running on the device side, the edge node, and the cloud server, respectively. All power consumption limits corresponding to different execution locations are then written into the subtask node's power consumption limit attribute. Next, considering the business level of the subtask node, and taking into account factors such as business level, latency constraints, accuracy lower limit requirements, the number of downstream dependent subtask nodes, and whether the subtask node is on a critical path, the priority of the subtask node is calculated. This priority can be divided into high, medium, and low priority levels, or a normalized numerical form can be written into the corresponding subtask node's priority attribute. The cross-layer data transmission volume, latency upper limit, power consumption upper limit, and accuracy lower limit of a sub-task node determine its candidate execution location set. This candidate execution location set is a subset of the total set comprised of edge devices, edge nodes, and cloud servers. Specifically, the execution location determination logic is as follows: when the cross-layer data transmission volume of the sub-task node is large and the latency upper limit is less than the system's pre-set latency threshold, the edge device is prioritized as a candidate execution location. When the sub-task node's accuracy lower limit is higher than the system's pre-set accuracy threshold and its latency upper limit is greater than the system's pre-set latency threshold, the cloud server is included in the candidate execution location set. When the sub-task node has data privacy protection requirements, the candidate execution location is... The location set is limited to edge devices or edge nodes. Each sub-task node is labeled with a migrated flag, a allowed degradation flag, and a set of candidate degradation models. Then, the priority, lower limit of accuracy, upper limit of latency, upper limit of power consumption, data volume, candidate execution location set, migrated flag, allowed degradation flag, and candidate degradation model set of each sub-task node are uniformly written into the original directed acyclic graph (DAG). This constructs a DAG for the detection task with service quality constraints. This graph structure includes the original set of sub-task nodes, the set of directed edges, and the set of service quality attributes corresponding to all sub-task nodes. After completing the graph structure construction, for the detection task with service quality constraints... A feasibility check is performed on the directed acyclic graph (DAG). For each subtask node, it is determined whether a feasible execution plan exists within its own set of candidate execution positions that simultaneously satisfies the constraints of lower accuracy limits, upper latency limits, upper power consumption limits, and cross-layer data transmission volume. If the check determines that no feasible execution plan exists, the system's fault-tolerant processing flow is executed. This involves adjusting the AI ​​inference model's splitting points, enabling a degraded model, or triggering detection task admission control to complete task adaptation. If the check determines that a feasible execution plan exists, the DAG of the detection task with service quality constraints is output and sent to downstream processing stages for AI inference model adaptive splitting and cross-layer pipeline orchestration.

[0016] Step S3: Based on the latency threshold of each sub-task in the directed acyclic graph of the AI ​​detection task, determine the splitting point of the AI ​​model, and split the AI ​​model into edge sub-model, cloud sub-model, and terminal sub-model. Dynamically adjust the model splitting point and sub-task distribution path when the network quality changes.

[0017] It should be further explained that the specific steps for dynamically adjusting the model splitting point and subtask distribution path are as follows: The system acquires a directed acyclic graph (DAG) of the detection task with quality of service (QoS) constraints and its real-time network status. The DAG of the detection task with QoS constraints includes a set of subtask nodes, a set of directed edges, and a set of QoS attributes for the subtask nodes. The QoS attributes for the subtask nodes include at least the upper limit of latency, the lower limit of accuracy, the upper limit of power consumption, the amount of input data, the amount of output data, the amount of cross-layer data transmission, the set of candidate execution locations, the migration flag, and the allowable degradation flag for each subtask node. The real-time network status includes the available bandwidth, round-trip latency, link jitter, packet loss rate, and connection stability of the edge links and edge-cloud links, as well as the real-time computing power, memory, power consumption, and task queue blocking rate of the edge devices, edge nodes, and cloud servers. The AI ​​model is parsed into a divisible structure, which is then divided into an ordered sequence of model units. Each model unit is labeled with its computational cost, number of parameters, memory usage, input and output tensor shape and data volume, whether it can be executed independently, whether it supports quantization or pruning, and whether it supports cross-layer transmission. Based on the ordered model unit sequence and the upper limit of latency, lower limit of accuracy, amount of cross-layer data transmission, and set of candidate execution locations for each subtask node, a set of candidate splitting points is constructed. Each candidate split point includes a first split point between the edge sub-model and the edge sub-model, and a second split point between the edge sub-model and the cloud sub-model. The candidate split points satisfy the following conditions: the amount of cross-layer data transmitted from the edge sub-model to the edge node does not exceed the amount of data that the edge link can transmit within the corresponding latency budget; the amount of cross-layer data transmitted from the edge sub-model to the cloud server does not exceed the amount of data that the edge-cloud link can transmit within the corresponding latency budget; and the cumulative accuracy loss of the edge sub-model, the edge sub-model, and the cloud sub-model does not exceed the lower limit of the accuracy of the corresponding sub-task node.

[0018] Dynamically adjusting model split points and subtask distribution paths also includes: For each candidate segmentation point, the inference latency of the end-side sub-model, the transmission latency of the end-edge, the inference latency of the edge sub-model, the transmission latency of the edge-cloud, the inference latency of the cloud sub-model, and the queuing latency are calculated to obtain the total end-to-end latency. The detection accuracy, total power consumption, total memory usage, and total amount of cross-layer data transmission are also calculated after segmentation. Based on the latency upper limit, accuracy lower limit, power consumption upper limit, candidate execution location set, and directed edge dependencies of each subtask node, constraint verification is performed on each candidate split point to determine whether it meets the following conditions: the end-to-end total latency is not greater than the latency upper limit of the corresponding subtask node; the detection accuracy after splitting is not less than the accuracy lower limit of the corresponding subtask node; the total power consumption is not greater than the power consumption upper limit of the corresponding node; the deployment location of each sub-model belongs to the candidate execution location set; and the total amount of cross-layer data transmission does not exceed the product of the current available bandwidth and the remaining latency budget. Candidate split points that do not meet any of the constraints are removed from the candidate split point set. Among the candidate segmentation points that satisfy the constraints, at least two of the following objectives are used as optimization goals: minimum end-to-end total latency, highest detection accuracy, lowest total power consumption, highest computing power utilization, optimal load balancing, and minimum total amount of cross-layer data transmission. A multi-objective evolutionary algorithm is used to select the optimal segmentation point, and the artificial intelligence inference model is divided into edge sub-model, edge sub-model, and cloud sub-model based on the optimal segmentation point. Specifically, the model units corresponding to the millisecond-level lightweight preliminary screening sub-task are preferentially assigned to the edge sub-model, the model units corresponding to the intermediate feature extraction and medium-complexity detection sub-tasks are preferentially assigned to the edge sub-model, and the model units corresponding to the fine-grained defect identification, multi-frame correlation analysis, and large model inference sub-tasks are preferentially assigned to the cloud sub-model. Based on the optimal split point and the directed acyclic graph of the detection task with quality of service constraints, subtask distribution paths are generated, mapping each subtask node to an end-side device, edge node, or cloud server, and determining a cross-layer data transmission path for each directed edge. These cross-layer data transmission paths include at least one of end-to-end direct transmission, end-to-edge relay, edge-to-cloud relay, and cloud aggregation and backhaul. Simultaneously, the execution order, batch size, queue priority, inference executor reuse strategy, and resource quotas for each subtask node are set, forming a pipeline orchestration strategy across the end-side, edge, and cloud layers.

[0019] Dynamically adjusting model split points and subtask distribution paths also includes: During execution, the available bandwidth, round-trip latency, link jitter, packet loss rate, and connection stability of the edge-cloud link are monitored in real time. When any of the following conditions are met, specifically: available bandwidth is lower than the preset bandwidth threshold, round-trip latency is higher than the preset latency threshold, link jitter is higher than the preset jitter threshold, packet loss rate is higher than the preset packet loss threshold, network interruption occurs, link switching occurs, or edge node or cloud server fails, it is determined that the network quality has changed, triggering dynamic adjustment of the model splitting point and subtask distribution path. The system updates the candidate splitting point set and the optimal splitting point, and dynamically adjusts the model splitting points and subtask distribution paths based on the updated optimal splitting point. Specifically, when the quality of the edge-cloud link deteriorates, some model units in the cloud sub-model are moved forward to the edge sub-model or end-side sub-model to reduce the amount of cross-layer data transmission between the edge and cloud. When the quality of the end-edge link deteriorates, some model units in the end-side sub-model are moved backward to the edge sub-model, or the original image and video stream data are compressed, frame-extracted, cached, and then uploaded with a delay. When the network quality recovers, the corresponding model units are moved backward to restore high-precision cloud inference. Simultaneously, the execution order, batch size, queue priority, resource quota, and cross-layer data transmission path of each subtask node are updated. If the latency upper limit and accuracy lower limit of the subtask nodes still cannot be met after dynamic adjustment, a degraded execution mechanism will be activated. This mechanism will downgrade high-precision large-model inference to lightweight model inference, suspend or migrate non-critical subtasks, and prioritize the continuous execution of high-priority, low-latency threshold, and safety-critical detection subtasks. Specifically, the updated model splitting points, subtask distribution paths, network status, and task execution results will be fed back to the multi-objective joint optimization scheduling model and the closed-loop feedback and cross-layer collaborative recovery mechanism. This will update the unified state vector of the edge-cloud and resolve the subtask mapping relationship, resource quota, batch size, execution order, and queue priority.

[0020] In this embodiment, a directed acyclic graph (DAG) of the detection task with quality of service (QoS) constraints and the real-time network status are obtained. The DAG includes a set of sub-task nodes, a set of directed edges, and a set of QoS attributes for each sub-task node. The QoS attributes of each sub-task node at least cover the upper limit of latency, the lower limit of accuracy, the upper limit of power consumption, the amount of input data, the amount of output data, the amount of cross-layer data transmission, the set of candidate execution locations, the migration flag, and the allowable degradation flag. The real-time network status includes the available bandwidth, round-trip latency, link jitter, packet loss rate, and connection stability of the edge-to-edge link and the cloud-to-edge link. Simultaneously, the real-time computing power, memory usage, power consumption level, and task queue blocking rate of the edge devices, edge nodes, and cloud servers are collected. Subsequently, the AI ​​model's divisible structure is analyzed to process the complete AI... The model is decomposed and parsed into an ordered sequence of model units. Simultaneously, attribute annotations are completed for each model unit, including its computational cost, number of parameters, memory usage, input / output tensor shape and corresponding data volume, whether it can be executed independently, whether it supports quantization or pruning, and whether it supports cross-layer transmission. Then, combining the ordered sequence of model units, and referring to the latency upper limit, accuracy lower limit, cross-layer transmission data volume, and candidate execution position set for each sub-task node, a candidate split point set is constructed. Each candidate split point simultaneously includes the first split point between the edge sub-model and the edge sub-model, and the second split point between the edge sub-model and the cloud sub-model. Candidate split points must satisfy corresponding constraints: the cross-layer transmission data volume generated by the edge sub-model output to the edge node cannot exceed the data volume that the edge link can handle within the corresponding latency budget. The amount of cross-layer data transmitted to the cloud server cannot exceed the amount of data that the edge-cloud link can handle within the corresponding latency budget. The overall cumulative accuracy loss resulting from the superposition of the end-side sub-model, edge sub-model, and cloud sub-model cannot exceed the accuracy lower limit set for the corresponding sub-task node. For each candidate segmentation point within the candidate segmentation point set, the inference latency of the end-side sub-model, the edge transmission latency, the edge sub-model inference latency, the edge-cloud transmission latency, the cloud sub-model inference latency, and the queuing latency are calculated and summarized to obtain the end-to-end total latency of the corresponding scheme. At the same time, the detection accuracy, overall total power consumption, total memory usage, and total cross-layer data transmission volume under this segmentation scheme are calculated. Then, combined with the latency upper limit, accuracy lower limit, power consumption upper limit, candidate execution position set, and task dependency relationship represented by directed edges of each sub-task node, multi-dimensional constraint verification is carried out for each candidate segmentation point.The following checks are performed sequentially: whether the total end-to-end latency of the proposed solution is not greater than the latency upper limit of the corresponding sub-task node; whether the actual detection accuracy after segmentation is not less than the accuracy lower limit of the corresponding sub-task node; whether the overall total power consumption is not greater than the power consumption upper limit of the corresponding sub-task node; whether the deployment location of each sub-model falls within the candidate execution location set of the corresponding sub-task node; and whether the total amount of cross-layer data transmission does not exceed the data scale that can be carried, jointly determined by the available bandwidth of the current link and the remaining latency budget. If a candidate segmentation point does not meet any of the above constraints, it is directly removed from the candidate segmentation point set. Among the candidate segmentation points that pass all checks and meet all constraints, the one with the minimum end-to-end total latency, the highest detection accuracy, and the lowest total power consumption is selected. At least two of the following optimization objectives are considered: low latency, highest computing power utilization, optimal load balancing, and minimum total amount of cross-layer data transmission. A multi-objective evolutionary algorithm is used to screen and determine the optimal segmentation point. Based on the selected optimal segmentation point, the complete artificial intelligence inference model is segmented into edge sub-models, cloud sub-models, and terminal sub-models. In the actual segmentation process, the principle of adapting to business sub-tasks is followed. Model units corresponding to millisecond-level lightweight preliminary screening sub-tasks are prioritized for allocation to edge sub-models; model units corresponding to intermediate feature extraction and medium-complexity detection sub-tasks are prioritized for allocation to edge sub-models; and model units corresponding to fine-grained defect identification, multi-frame correlation analysis, and large-model inference sub-tasks are prioritized for allocation to cloud sub-models. After determining the model segmentation results, By combining the optimal split point and the directed acyclic graph of the detection task with service quality constraints, subtask distribution paths are generated. Each subtask node is mapped and deployed to edge devices, edge nodes, or cloud servers. Simultaneously, a corresponding cross-layer data transmission path is determined for each directed edge. This path can be any combination of end-to-end direct transmission, end-to-edge relay, edge-to-cloud relay, or cloud aggregation and backhaul. Furthermore, the actual execution order, batch size, queue priority, inference executor reuse strategy, and resource quotas for each subtask node are configured. This ultimately forms a pipeline orchestration strategy covering the edge, cloud, and end-to-end architectures. Throughout the entire task execution process, the edge and end-to-end nodes are continuously monitored in real time. The system monitors the available bandwidth, round-trip latency, link jitter, packet loss rate, and connection stability of the cloud and edge-cloud links. When any of the following occurs: available bandwidth falls below a preset bandwidth threshold, round-trip latency exceeds a preset latency threshold, link jitter exceeds a preset jitter threshold, packet loss rate exceeds a preset packet loss threshold, or a network outage, link switching, or device failure at an edge node or cloud server, it is determined that the current network quality has significantly changed. This triggers a dynamic readjustment process for the model splitting points and subtask distribution paths. The candidate splitting point set is updated, and the optimal splitting point is solved again. Based on the updated optimal splitting point, the model splitting position and subtask distribution paths are dynamically updated. Specifically, when the quality of the cloud and edge links deteriorates...Some model units originally belonging to the cloud sub-model are moved forward and adjusted to the edge sub-model or end-side sub-model to reduce the amount of cross-layer data transmission between the cloud and the edge. When the quality of the end-edge link deteriorates, some model units in the end-side sub-model are moved backward and adjusted to the edge sub-model. Alternatively, the original image and video stream data can be compressed, frame-stripped, and cached before delayed upload. After the network quality returns to normal, the corresponding model units that were previously moved forward are moved backward again to restore the high-precision inference capability of the cloud side. The execution order, batch size, queue priority, resource quota, and cross-layer data transmission path of each sub-task node are updated synchronously. If the entire solution remains stable after a round of dynamic adjustments, the solution will be optimized. If the latency upper limit and accuracy lower limit constraints of subtask nodes cannot be met, a degradation execution mechanism will be activated. This will downgrade the original high-precision, large-model inference to a lightweight model inference, pausing or migrating non-critical subtasks. Priority will be given to ensuring uninterrupted execution of high-priority, low-latency threshold, and safety-critical detection subtasks. Furthermore, the updated model splitting points, subtask distribution paths, real-time network status, and actual task execution results will be fed back to the multi-objective joint optimization scheduling model. Simultaneously, these will be fed into a closed-loop feedback and cross-layer collaborative recovery mechanism to update the unified state vector across edge, cloud, and endpoint. The subtask mapping relationships, resource quotas, batch sizes, execution order, and queue priorities will be recalculated, completing a closed-loop update process.

[0021] Step S4: Dynamically adjust the computing power capacity of each node according to the task queue blocking rate to suppress task queue blocking. After dynamically adjusting the computing power capacity of each node, determine the execution order and queue priority based on the batch size.

[0022] It should be understood that the specific steps for dynamically adjusting the computing power capacity of each node based on the task queue blocking rate are as follows: Real-time collection of task queue status information from scheduling nodes in end-side devices, edge nodes, and cloud servers. The task queue status information includes at least the current queue length, queue capacity, number of tasks arriving per unit time, number of tasks served per unit time, number of tasks waiting for timeout, total number of waiting tasks, and average waiting time for tasks. The task queue blocking rate of each scheduling node is calculated based on the task queue status information. The ratio of the current queue length to the queue capacity is used as the first blocking factor, the ratio of the number of timed-out tasks to the total number of waiting tasks is used as the second blocking factor, the ratio of the average waiting time of a task to the latency threshold of the detection tasks carried by the node is used as the third blocking factor, and the ratio of the number of tasks arriving per unit time to the number of tasks served per unit time is used as the fourth blocking factor. Preset weights are assigned to the first, second, third, and fourth blocking factors, and the sum of each preset weight is one. The task queue blocking rate of the scheduling node is obtained by weighted summation of each blocking factor. When the queue capacity, total number of waiting tasks, latency threshold, or number of service tasks per unit time is zero, it is corrected according to the preset default value to ensure that the task queue blocking rate calculation is continuous and does not produce a division-by-zero error.

[0023] Dynamically adjusting the computing power capacity of each node based on the task queue blocking rate also includes: Set a first blocking rate threshold, a second blocking rate threshold, and a third blocking rate threshold, with the first blocking rate threshold being less than the second blocking rate threshold, and the second blocking rate threshold being less than the third blocking rate threshold. Compare the task queue blocking rate of each scheduling node with the first blocking rate threshold, the second blocking rate threshold, and the third blocking rate threshold, respectively. When the task queue blocking rate is not greater than the first blocking rate threshold, it is determined to be in a normal state. When the task queue blocking rate is greater than the first blocking rate threshold but not greater than the second blocking rate threshold, it is determined to be in a warning state. When the task queue blocking rate is greater than the second blocking rate threshold but not greater than the third blocking rate threshold, it is determined to be in an overload state. When the task queue blocking rate is greater than the third blocking rate threshold, it is determined to be in a severely blocked state. When any scheduling node is overloaded, a computing power expansion operation is triggered; when any scheduling node is severely blocked, an emergency expansion and collaborative degradation operation is triggered; when the task queue blocking rate of any scheduling node is less than the first blocking rate threshold, a computing power reduction operation is triggered.

[0024] In this embodiment, the computing power expansion operation includes increasing the computing power quota of the neural network processing unit, central processing unit, or graphics processing unit of the scheduling node, increasing the number of inference threads or processes, increasing the batch processing limit, enabling reserved computing power resources, increasing the working frequency, and migrating the detection subtasks marked as migrateable and non-real-time in the scheduling node's queue to at least one of other end-side devices, edge nodes, or cloud servers whose task queue blocking rate is lower than a first blocking rate threshold. During migration, priority is given to migrating detection subtasks with small cross-layer data transmission volume, not located on the critical path, and allowed to be migrated. The emergency expansion and collaborative degradation operation includes pausing or delaying the execution of low-priority, non-real-time, and high-power detection subtasks. Prioritize ensuring computing power quotas for high-priority, low-latency threshold, and high-precision detection subtasks; downgrade cloud-based large-model inference subtasks to edge-based lightweight model inference subtasks; downgrade edge-based high-precision defect identification subtasks to edge-based preliminary screening subtasks; compress, extract, or cache raw image or video stream data before delayed upload; and trigger at least one of the following: computing power reduction operations include reducing the computing power quota of the scheduling node, reducing the number of inference threads or processes, lowering the batch processing limit, releasing cache resources, suspending some inference executors, and adding the released computing power resources to the unified edge-cloud computing power pool for use by other high-blocking-rate nodes or high-priority detection tasks. One less; when scaling down computing power, set a hysteresis interval and minimum hold time to avoid inference performance jitter caused by frequent adjustments to computing power capacity; when performing computing power expansion, scaling down, task migration, or collaborative degradation, use the power consumption limit, memory capacity, storage capacity, network bandwidth, link jitter, task dependencies in the directed acyclic graph of detection tasks, detection accuracy lower limit, and end-to-end latency threshold of each scheduling node as constraints, and ensure that the expanded computing power capacity does not exceed the physical limit of the scheduling node, and the scaled-down computing power capacity is not lower than the minimum computing power capacity required to ensure the normal execution of high-priority detection tasks; during task migration, maintain the data flow dependencies and model dependencies between sub-task nodes in the directed acyclic graph of detection tasks. The execution order dependency is not disrupted; the dynamically adjusted computing capacity of each scheduling node, task queue blocking rate, task migration results, and collaborative degradation execution results are fed back to the multi-objective joint optimization scheduling model and the closed-loop feedback and cross-layer collaborative recovery mechanism, updating the unified state vector of edge-cloud and end-to-end, and resolving the subtask mapping relationship, resource quota, batch size, execution order, and queue priority. This enables the task queue blocking rate to form a closed-loop adaptive match with the computing capacity of each scheduling node, thereby avoiding idle and wasted computing resources, reducing the task queue blocking rate and inference performance jitter in large-scale concurrent detection scenarios, and improving the utilization rate of computing resources, load balancing, and end-to-end completion rate of detection tasks in the edge-cloud three-layer computing power resource system.

[0025] It should be understood that, such as Figure 2The flowchart shown below illustrates the sequence and queue priority of an edge-cloud collaborative AI computing power and detection task integrated scheduling method provided in this application embodiment. The specific process is as follows: First, batch processing thresholds and latency thresholds are set. Tasks are then categorized into four batch sizes: small, medium, large, and ultra-large. Simultaneously, based on the latency upper limit, three latency urgency levels are determined: high, medium, and low. The batch type and latency urgency level are matched to generate the initial task queue priority and execution order. Then, the initial scheduling configuration is modified based on the queue blocking rate of the scheduling node. Large and ultra-large batch tasks are downgraded when the blocking rate is between the second and third blocking rate thresholds; otherwise, the blocking rate exceeds the third blocking rate threshold. Pause or delay low-priority large tasks, while maintaining the initial configuration in other scenarios. Then, dynamically adjust the batch processing scale based on the remaining computing power of the nodes and the queue blocking rate. When computing power is sufficient and the blocking rate is low, aggregate multiple small and medium batch tasks into large batch tasks. When computing power is insufficient or the blocking rate is high, split large and ultra-large batch tasks into small or medium batch tasks, and then recalculate the queue priority and execution order. The split small batch tasks are inserted into the queue according to the latency urgency level. The aggregated large batch tasks open an aggregation waiting window. If the waiting window ends or the number of aggregated tasks reaches the preset limit, batch inference is triggered. If a high-urgency task appears in the window, it is allowed to jump the queue for execution. The large batch tasks continue to wait, and finally the final queue scheduling scheme is output.

[0026] It should be further explained that the specific steps for determining the execution order and queue priority based on the batch size are as follows: Set thresholds for the first batch processing size, the second batch processing size, and the third batch processing size. Compare the batch processing size of each subtask node with these thresholds. Specifically, obtain the batch processing size, upper latency limit, lower precision limit, priority, input data volume, output data volume, model computation volume, candidate execution location set, and migration flags for each subtask node in the directed acyclic graph of the task to be scheduled. Simultaneously, obtain the current batch processing capacity, remaining computing power, memory usage, task queue blocking rate, number of aggregated tasks in the queue, and inference executor occupancy status of each scheduling node in the edge device, edge node, and cloud server. The first batch processing size threshold is less than the second batch processing size threshold, and the second batch processing size threshold is less than the third batch processing size threshold. When the batch size is not greater than the threshold of the first batch size, it is determined to be a small batch task; When the batch size is greater than the threshold for the first batch size but not greater than the threshold for the second batch size, it is determined to be a medium batch task. When the batch size is greater than the second batch size threshold but not greater than the third batch size threshold, it is judged as a large batch task. When the batch size exceeds the third batch size threshold, it is determined to be an ultra-large batch task; The latency urgency level of a subtask node is determined based on its latency cap, specifically as follows: Subtask nodes with a latency limit not greater than the first latency threshold are classified as high-urgency tasks; subtask nodes with a latency limit greater than the first latency threshold but not greater than the second latency threshold are classified as medium-urgency tasks; and subtask nodes with a latency limit greater than the second latency threshold are classified as low-urgency tasks; wherein the first latency threshold is less than the second latency threshold. Based on the batch size determination result and the latency urgency level, specifically: for subtask nodes of small batch tasks that are high-urgency tasks, their queue priority is set to the highest priority, and their execution order is arranged at the front of the current queue; For subtask nodes that are small-batch tasks and are medium- or low-urgency tasks, set their queue priority to high priority and arrange their execution order after the highest priority tasks. For subtask nodes of medium-batch tasks that are high-urgency tasks, set their queue priority to high priority and allow them to be aggregated with adjacent medium-batch tasks for execution, provided that the latency limit is met. For subtask nodes of medium-batch tasks that are medium-urgent or low-urgent tasks, set their queue priority to medium priority.

[0027] Determining the execution order based on batch size and queue priority also includes: For subtask nodes of large or very large batches of tasks that are low-urgency tasks, set their queue priority to low priority and place them at the end of the current queue for execution, waiting to be aggregated into a larger batch before execution. The initial queue priority and execution order are adjusted based on the task queue blocking rate of each scheduling node: when the task queue blocking rate of a scheduling node is greater than the second blocking rate threshold but not greater than the third blocking rate threshold, a downgrade operation is performed on the large batch and ultra-large batch tasks in the queue of that scheduling node, reducing their queue priority and shifting their execution order backward, while moving the small batch and high-urgency tasks forward; when the task queue blocking rate of a scheduling node is greater than the third blocking rate threshold, the execution of low-priority large batch and ultra-large batch tasks is suspended or delayed, and the execution of small batch, high-urgency, and high-priority tasks is prioritized. Based on the current batch processing capacity and remaining computing power of each scheduling node, the batch processing size of the subtask nodes is dynamically adjusted: when the remaining computing power of the scheduling node is sufficient and the task queue blocking rate is not greater than the first blocking rate threshold, multiple small batch tasks or medium batch tasks are aggregated into large batch tasks; when the remaining computing power of the scheduling node is insufficient or the task queue blocking rate is greater than the second blocking rate threshold, large batch tasks or ultra-large batch tasks are split into multiple small batch tasks or medium batch tasks. Based on the batch size adjustment results, the execution order and queue priority of subtask nodes are re-determined: For small batches of tasks obtained after splitting, they are inserted into the corresponding positions in the current queue according to their latency urgency level and precision lower limit; for large batches of tasks obtained after aggregation, their execution order is shifted to the back, and an aggregation waiting window is set in the queue. Within the waiting window, similar subtask nodes continue to be received. When the waiting window ends or the number of aggregated tasks reaches the preset upper limit, batch inference is triggered; if a high-urgency task appears within the waiting window, the high-urgency task is allowed to jump the queue for execution, and the large batch of tasks that are jumped the queue continue to wait. During execution, the system monitors the actual inference latency, batch execution time, memory usage, power consumption, and task queue blocking rate of each subtask node in real time. When the actual inference latency exceeds its latency limit or the task queue blocking rate continues to rise, the system dynamically adjusts the batch size, execution order, and queue priority. The adjustment results are then fed back to the multi-objective joint optimization scheduling model and the closed-loop feedback and cross-layer collaborative recovery mechanism. This updates the unified state vector of the edge-cloud and resolves the subtask mapping relationship, resource quota, batch size, execution order, and queue priority, thereby achieving dynamic collaborative optimization of the execution order and queue priority based on the batch size.

[0028] In this embodiment, three progressively increasing batch processing size thresholds are pre-set. First, the batch processing size, upper latency limit, lower precision limit, priority, input / output data volume, model computation volume, candidate execution location set, and transferability flag of each sub-task node in the directed acyclic graph of the task to be scheduled are obtained. Simultaneously, the batch processing capacity, remaining computing power, memory usage, task queue blocking rate, number of aggregated tasks in the queue, and inference executor occupancy status of each scheduling node at the edge, cloud, and endpoints are obtained. The batch processing size of the sub-task nodes is compared with the three thresholds to classify tasks into four categories: small batch, medium batch, large batch, and ultra-large batch. Then, based on the pre-set first and second latency thresholds related to the batch size, the batch processing size is further determined. Based on the latency limits of subtask nodes, three latency urgency levels are defined: high, medium, and low. Then, the initial queue priority and execution order are configured according to the batch processing type and latency urgency level. Small batches of high-urgency subtasks are set to the highest priority and placed at the front of the queue. Small batches of medium and low-urgency subtasks are set to high priority and placed after the highest-priority subtasks. Medium batches of high-urgency subtasks are set to high priority and, if the latency limits are met, can be aggregated with adjacent medium batches of subtasks for execution. Medium batches of medium and low-urgency subtasks are set to medium priority. Large or very large batches of low-urgency subtasks are set to low priority and placed at the end of the queue, waiting to be aggregated into larger batches before execution. Finally, the initial priority and execution order are adjusted based on the task queue blocking rate of the scheduling node. In this sequence, when the queue blocking rate of the scheduling node is between the second and third blocking rate thresholds, large and very large batches of tasks in the queue are downgraded and moved to the back, while small and high-urgency tasks are moved to the front. When the blocking rate exceeds the third blocking rate threshold, low-priority large and very large batches of tasks are paused or postponed, prioritizing the execution of small and high-urgency tasks. The batch size can also be dynamically adjusted based on the scheduling node's batch processing capacity and remaining computing power. When there is sufficient remaining computing power and the queue blocking rate does not exceed the first blocking rate threshold, multiple small and medium batches of tasks are aggregated into a large batch of tasks. When there is insufficient remaining computing power or the queue blocking rate exceeds the second blocking rate threshold, large and very large batches of tasks are split into several small or medium batches for batch processing. After the size adjustment is completed, the execution order and queue priority are redefined. The split small batch tasks are inserted into the corresponding queue positions according to latency urgency and precision lower limit. The aggregated large batch tasks are moved to the back of the execution order and an aggregation waiting window is configured. The waiting window continuously receives similar subtasks. When the window ends or the number of aggregated tasks reaches the preset limit, batch inference is started. If a high-urgency task arrives in the window, it is allowed to jump the queue, and the large batch tasks that are jumped the queue continue to wait. During the task execution phase, the actual inference latency, batch execution time, memory usage, power consumption, and queue blocking rate of each subtask are monitored in real time. Once the actual inference latency exceeds the latency limit or the queue blocking rate continues to rise, the batch size, execution order, and queue priority are dynamically adjusted again.Furthermore, the adjustment results are fed back to the multi-objective joint optimization scheduling model and the closed-loop feedback and cross-layer collaborative recovery mechanism to update the unified state vector of the edge-cloud system. This allows for the re-solution of subtask mapping relationships, resource quotas, batch size, execution order, and queue priority, ultimately achieving dynamic collaborative optimization of execution order and queue priority based on batch size.

[0029] In step S5, during the execution order and queue priority process, when a network interruption or edge node failure is detected, the detection task is dynamically migrated across layers and its execution is downgraded. The cloud subtasks are migrated to the edge nodes or end devices, and the remaining unfinished subtasks in the directed acyclic graph of the detection task are rescheduled.

[0030] It should be understood that during the process of determining the execution order and queue priority of each subtask node according to the batch size and executing them, the connection status, available bandwidth, round-trip latency, link jitter, packet loss rate, and connection stability of the edge-to-edge link and edge-to-cloud link are monitored in real time. Simultaneously, the heartbeat signals, computing power status, memory usage, power consumption, inference executor status, task queue blocking rate, and node online status of the edge nodes and cloud servers are monitored. When at least one of the following occurs: edge-to-cloud link interruption, edge-to-edge link interruption, cloud service unreachable, edge node heartbeat timeout, edge node computing power anomaly, memory overflow, inference executor crash, power consumption anomaly, or node offline, a network interruption or edge node failure event is determined. In response to determining the occurrence of a network interruption or edge node failure event... In the event of a failure, the affected scheduling nodes and the scope of affected detection tasks are identified. The task queues corresponding to the affected scheduling nodes are frozen, and the dispatch of new sub-task nodes to the failed nodes is suspended. Sub-task nodes currently executing on the failed node are marked as sub-tasks to be migrated or recovered, and sub-task nodes that have not yet started are marked as sub-tasks to be rescheduled. Simultaneously, the output results, intermediate features, checkpoint data, model version, accuracy information, and timestamps of completed sub-task nodes are recorded. The affected directed acyclic graph of detection tasks is located from the global detection task queue. The set of remaining unfinished sub-tasks in the directed acyclic graph is determined, and completed sub-task nodes, currently executing sub-task nodes, unstarted sub-task nodes, and recoverable sub-task nodes are distinguished. The data is then obtained. The system sets the latency upper limit, accuracy lower limit, power consumption upper limit, priority, batch size, candidate execution location set, migration flag, degradability flag, cross-layer data transfer volume, and location on the critical path for nodes with unfinished remaining subtasks. Based on these attributes, it filters migrated and degradable subtasks. For cloud-based subtasks, it prioritizes migrating them to edge nodes; if edge nodes are unavailable or have insufficient remaining computing power, it migrates them to edge devices. For non-real-time subtasks marked as migrateable but not located on the critical path, it allows delayed execution or migration to other nodes with a task queue blocking rate below the first blocking rate threshold. For subtask nodes that are not allowed to migrate or cannot meet the latency upper limit after migration, it enables degradative execution control. The system then constructs a migration... The set of candidate target nodes is moved, and the remaining computing power, memory capacity, power consumption margin, task queue blocking rate, network bandwidth, link jitter, inference executor occupancy status, and loaded model fragments of each candidate target node are evaluated. The optimal migration target node is selected with constraints or optimization objectives such as total migration latency, detection accuracy loss, device power consumption, task queue blocking rate, load balancing, and cross-layer data transmission volume. When selecting the migration target node, the data flow dependency, model dependency, and execution order dependency between sub-task nodes in the directed acyclic graph of the detection task are not disrupted. If the model fragment corresponding to the cloud sub-task has been pre-cached on the edge node or end device, the loaded inference executor is directly loaded and reused; if it is not cached, an equivalent lightweight model is delivered or loaded.The input data, intermediate features, or checkpoint data of the cloud-based subtasks are compressed and transmitted to the migration target node, where execution is resumed. The execution position, resource quota, batch size, execution order, and queue priority of the subtask node are updated. For high-priority, low-latency threshold subtask nodes and those located on the critical path, they are inserted at the front of the migration target node's queue for priority execution. For large or very large batches of migration subtasks, they are split or moved to the back for aggregation. The remaining unfinished subtasks in the directed acyclic graph of the detection task are rescheduled: the directed acyclic graph of the detection task is updated based on the migration results, completed subtask nodes are deleted, and the critical path, earliest start time, and latest completion time are recalculated. The process involves determining the completion time, remaining latency budget, and priority. Based on the updated priorities, latency urgency, batch size, and portability, the execution order and queue priority of each remaining unfinished subtask node are redefined. For cloud subtasks where edge nodes or end-devices lack sufficient computing power after migration, the AI ​​inference model is divided into sub-model fragments executable on edge nodes or end-devices according to the model splitting points, and unexecutable later-stage subtasks are cached or delayed. When the latency upper limit and accuracy lower limit of the subtask nodes still cannot be met after migration, high-precision large-model inference is downgraded to lightweight model inference, fine-grained defect identification is downgraded to preliminary screening, and multi-frame correlation analysis is downgraded to single-frame analysis, according to the downgraded model candidate set. The analysis involves reducing the resolution, frame rate, or batch size of the input image or video stream, decreasing the number of model layers or channels, and recording the accuracy loss, latency changes, and power consumption changes after degradation. For safety-critical, high-priority, or low-latency-threshold subtask nodes, their computing power quotas and execution order are prioritized. For low-priority, non-real-time, or degradation-permitted subtask nodes, execution is paused, upload is delayed, or they are migrated to other idle nodes. When the network interruption is recovered or the edge node failure is recovered, the status information of the edge-to-edge link, edge-to-cloud link, and recovered node is re-collected to determine whether the migration conditions are met. If they are met, the subtask nodes migrated to the edge node or end-side device are migrated back to the cloud server according to the reverse migration process to restore high accuracy. The algorithm performs reasoning and updates the execution order, queue priority, batch size, and resource quota. If these conditions are not met, it maintains the degraded or migrated execution state until the migration conditions are met. It feeds back network interruption or edge node failure events, migration target nodes, migration results, degraded execution results, rescheduled execution order, and queue priority to the unified edge-cloud state vector, multi-objective joint optimization scheduling model, and closed-loop feedback and cross-layer collaborative recovery mechanism. This updates subtask mapping relationships, resource quotas, batch size, execution order, and queue priority, ensuring that unfinished subtasks in the directed acyclic graph of the detection task are executed continuously in weak network or fault scenarios, and gradually recovers to the globally integrated optimal scheduling state after fault recovery.

[0031] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope and intent of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and variations.

Claims

1. A method for integrated scheduling of AI computing power and detection tasks through edge-cloud collaboration, characterized in that, Includes the following steps: Step S1: Collect load parameters of the end-side device and edge node parameters in real time, and obtain the priority of the AI ​​detection tasks to be scheduled to form a unified state vector of end-edge-cloud and a global detection task queue. Step S2: Each AI detection task is parsed into a directed acyclic graph, and each subtask node is labeled with priority, lower limit of accuracy, upper limit of latency, upper limit of power consumption and data volume, forming a directed acyclic graph of detection tasks with service quality constraints. Step S3: Based on the latency threshold of each sub-task in the directed acyclic graph of the AI ​​detection task, determine the splitting point of the AI ​​model, and split the AI ​​model into edge sub-model, cloud sub-model, and terminal sub-model. Dynamically adjust the model splitting point and sub-task distribution path when the network quality changes. Step S4: Dynamically adjust the computing power capacity of each node according to the task queue blocking rate to suppress task queue blocking. After dynamically adjusting the computing power capacity of each node, determine the execution order and queue priority based on the batch size. In step S5, during the execution order and queue priority process, when a network interruption or edge node failure is detected, the detection task is dynamically migrated across layers and its execution is downgraded. The cloud subtasks are migrated to the edge nodes or end devices, and the remaining unfinished subtasks in the directed acyclic graph of the detection task are rescheduled.

2. The edge-cloud collaborative AI computing power and detection task integrated scheduling method as described in claim 1, characterized in that: The specific process for labeling each subtask node with priority, accuracy lower limit, latency upper limit, power consumption upper limit, and data volume is as follows: Obtain the end-to-end service quality requirements and task attributes for each AI detection task. The end-to-end service quality requirements and task attributes include at least the task identifier, business type, business level, whether it is safety critical, end-to-end accuracy requirements, and end-to-end latency threshold. Each AI detection task is parsed as a directed acyclic graph. Based on the input data type, input tensor shape, number of channels, bit width, number of frames, resolution, encoding bit rate, compression ratio, and feature map size of the sub-task nodes, the input data volume, output data volume, and cross-layer transmission data volume that needs to be transmitted across devices, edges, and the cloud are calculated and written into the data volume attribute of the sub-task nodes. For video stream detection tasks, the data volume per unit time is calculated according to the frame extraction rate, single frame resolution, single frame bit depth, and encoding compression ratio. Based on the end-to-end accuracy requirements, the position of the sub-task nodes in the directed acyclic graph, and the series and parallel relationships between the sub-task nodes, the end-to-end accuracy requirements are decomposed to each sub-task node to obtain the lower limit of accuracy for each sub-task node, and the fusion detection accuracy of all sub-task nodes after execution according to the directed acyclic graph is not lower than the end-to-end accuracy requirements.

3. The edge-cloud collaborative AI computing and detection task integrated scheduling method of claim 2, wherein: The method also includes labeling each subtask node with priority, lower limit of precision, upper limit of latency, upper limit of power consumption, and data volume, and further includes: Based on the end-to-end latency threshold and the critical path of the directed acyclic graph, calculate the earliest start time, latest completion time, and remaining latency of the critical path for each subtask node to obtain the latency upper limit of the subtask node. Then, decompose the latency upper limit into the computation latency upper limit, the cross-layer transmission latency upper limit, and the queuing latency upper limit. Write the latency upper limit into the latency upper limit attribute of the subtask node. Obtain the battery capacity and power consumption of the terminal device, and mark the power consumption limit of the sub-task node according to the candidate execution location and whether it is portable; when the sub-task node can be executed in multiple locations in the terminal device, edge node and cloud server, mark the terminal power consumption limit, edge power consumption limit and cloud power consumption limit respectively; write the power consumption limit to the power consumption limit attribute of the sub-task node. Based on the service level to which the subtask node belongs, the priority of the subtask node is calculated. Based on the cross-layer data transmission volume, latency upper limit, power consumption upper limit, and accuracy lower limit, a set of candidate execution locations is marked for the subtask node. When the cross-layer data transmission volume is large and the latency upper limit is less than the preset latency threshold, the candidate execution location set will preferentially include edge devices. When the accuracy lower limit is higher than the preset accuracy threshold and the latency upper limit is greater than the preset latency threshold, the candidate execution location set will include cloud servers. When the subtask node involves data privacy requirements, the candidate execution location set is restricted to edge devices or edge nodes.

4. The edge-cloud collaborative AI computing power and detection task integrated scheduling method as described in claim 1, characterized in that: The specific steps for dynamically adjusting the model's splitting points and subtask distribution paths are as follows: Obtain the directed acyclic graph of the detection task with service quality constraints and the real-time network status; the directed acyclic graph of the detection task with service quality constraints includes a set of subtask nodes, a set of directed edges, and a set of service quality attributes of subtask nodes; The AI ​​model is parsed into a divisible structure, which is then divided into an ordered sequence of model units. Each model unit is labeled with its computational cost, number of parameters, memory usage, input and output tensor shape and data volume, whether it can be executed independently, whether it supports quantization or pruning, and whether it supports cross-layer transmission. Based on the ordered model unit sequence and the upper limit of latency, lower limit of accuracy, amount of cross-layer data transmission, and set of candidate execution locations for each subtask node, a set of candidate splitting points is constructed. Each candidate split point includes a first split point between the edge sub-model and the edge sub-model, and a second split point between the edge sub-model and the cloud sub-model; Candidate split points satisfy the following conditions: the amount of cross-layer data transmitted from the end-side sub-model to the edge node does not exceed the amount of data that can be transmitted within the corresponding latency budget of the end-edge link; the amount of cross-layer data transmitted from the edge sub-model to the cloud server does not exceed the amount of data that can be transmitted within the corresponding latency budget of the edge-cloud link; and the cumulative accuracy loss of the end-side sub-model, edge sub-model, and cloud sub-model does not exceed the lower limit of the accuracy of the corresponding sub-task node.

5. The edge-cloud collaborative AI computing power and detection task integrated scheduling method as described in claim 4, characterized in that: The dynamic adjustment of model splitting points and subtask distribution paths also includes: For each candidate segmentation point, the inference latency of the end-side sub-model, the transmission latency of the end-edge, the inference latency of the edge sub-model, the transmission latency of the edge-cloud, the inference latency of the cloud sub-model, and the queuing latency are calculated to obtain the total end-to-end latency. The detection accuracy, total power consumption, total memory usage, and total amount of cross-layer data transmission are also calculated after segmentation. Based on the latency upper limit, accuracy lower limit, power consumption upper limit, candidate execution location set, and directed edge dependencies of each subtask node, constraint verification is performed on each candidate split point to determine whether it meets the following conditions: the end-to-end total latency is not greater than the latency upper limit of the corresponding subtask node; the detection accuracy after splitting is not less than the accuracy lower limit of the corresponding subtask node; the total power consumption is not greater than the power consumption upper limit of the corresponding node; the deployment location of each sub-model belongs to the candidate execution location set; and the total amount of cross-layer data transmission does not exceed the product of the current available bandwidth and the remaining latency budget. Candidate split points that do not meet any of the constraints are removed from the candidate split point set. A multi-objective evolutionary algorithm is used to select the optimal split point, and the artificial intelligence inference model is divided into edge sub-model, edge sub-model and cloud sub-model according to the optimal split point; Based on the optimal split point and the directed acyclic graph of the detection task with quality of service constraints, a subtask distribution path is generated, and each subtask node is mapped to the end device, edge node or cloud server, and a cross-layer data transmission path is determined for each directed edge.

6. The edge-cloud collaborative AI computing power and detection task integrated scheduling method as described in claim 5, characterized in that: The dynamic adjustment of model splitting points and subtask distribution paths also includes: During execution, the available bandwidth, round-trip latency, link jitter, packet loss rate, and connection stability of the edge-cloud link are monitored in real time. When any of the following conditions are met, specifically: available bandwidth is lower than the preset bandwidth threshold, round-trip latency is higher than the preset latency threshold, link jitter is higher than the preset jitter threshold, packet loss rate is higher than the preset packet loss threshold, network interruption occurs, link switching occurs, or edge node or cloud server fails, it is determined that the network quality has changed, triggering dynamic adjustment of the model splitting point and subtask distribution path. Update the candidate split point set and the optimal split point, and dynamically adjust the model split points and subtask distribution paths based on the updated optimal split point; If the latency upper limit and accuracy lower limit of the subtask node still cannot be met after dynamic adjustment, the degradation execution mechanism will be activated, which will downgrade high-precision large model inference to lightweight model inference, suspend or migrate non-critical subtasks, and prioritize the continuous execution of high-priority, low-latency threshold and safety-critical detection subtasks.

7. The edge-cloud collaborative AI computing power and detection task integrated scheduling method as described in claim 1, characterized in that: The specific steps for dynamically adjusting the computing power capacity of each node based on the task queue blocking rate are as follows: Real-time collection of task queue status information from scheduling nodes in end-side devices, edge nodes, and cloud servers. The task queue status information includes at least the current queue length, queue capacity, number of tasks arriving per unit time, number of tasks served per unit time, number of tasks waiting for timeout, total number of waiting tasks, and average waiting time for tasks. The task queue blocking rate of each scheduling node is calculated based on the task queue status information. The ratio of the current queue length to the queue capacity is used as the first blocking factor, the ratio of the number of timed-out tasks to the total number of waiting tasks is used as the second blocking factor, the ratio of the average waiting time of a task to the latency threshold of the detection tasks carried by the node is used as the third blocking factor, and the ratio of the number of tasks arriving per unit time to the number of tasks served per unit time is used as the fourth blocking factor. Preset weights are assigned to the first, second, third, and fourth blocking factors, and the sum of each preset weight is one. The task queue blocking rate of the scheduling node is obtained by weighted summation of each blocking factor. When the queue capacity, total number of waiting tasks, latency threshold, or number of service tasks per unit time is zero, it is corrected according to the preset default value to ensure that the task queue blocking rate calculation is continuous and does not produce a division by zero error.

8. The edge-cloud collaborative AI computing power and detection task integrated scheduling method as described in claim 7, characterized in that: The method of dynamically adjusting the computing power capacity of each node based on the task queue blocking rate also includes: Set a first blocking rate threshold, a second blocking rate threshold, and a third blocking rate threshold, with the first blocking rate threshold being less than the second blocking rate threshold, and the second blocking rate threshold being less than the third blocking rate threshold. Compare the task queue blocking rate of each scheduling node with the first blocking rate threshold, the second blocking rate threshold, and the third blocking rate threshold, respectively. When the task queue blocking rate is not greater than the first blocking rate threshold, it is determined to be in a normal state. When the task queue blocking rate is greater than the first blocking rate threshold but not greater than the second blocking rate threshold, it is determined to be in a warning state. When the task queue blocking rate is greater than the second blocking rate threshold but not greater than the third blocking rate threshold, it is determined to be in an overload state. When the task queue blocking rate is greater than the third blocking rate threshold, it is determined to be in a severely blocked state. When any scheduling node is overloaded, a computing power expansion operation is triggered; when any scheduling node is severely blocked, an emergency expansion and collaborative degradation operation is triggered; when the task queue blocking rate of any scheduling node is less than the first blocking rate threshold, a computing power reduction operation is triggered.

9. The edge-cloud collaborative AI computing power and detection task integrated scheduling method as described in claim 1, characterized in that: The specific steps for determining the execution order and queue priority based on batch size are as follows: Set the first batch processing size threshold, the second batch processing size threshold, and the third batch processing size threshold, and compare the batch processing size of each subtask node with the first batch processing size threshold, the second batch processing size threshold, and the third batch processing size threshold, respectively. When the batch size is not greater than the threshold of the first batch size, it is determined to be a small batch task; When the batch size is greater than the threshold for the first batch size but not greater than the threshold for the second batch size, it is determined to be a medium batch task. When the batch size is greater than the second batch size threshold but not greater than the third batch size threshold, it is judged as a large batch task. When the batch size exceeds the third batch size threshold, it is determined to be an ultra-large batch task; The latency urgency level of a subtask node is determined based on its latency cap, specifically as follows: Subtask nodes with a latency limit not greater than the first latency threshold are classified as high-urgency tasks; subtask nodes with a latency limit greater than the first latency threshold but not greater than the second latency threshold are classified as medium-urgency tasks; and subtask nodes with a latency limit greater than the second latency threshold are classified as low-urgency tasks. The first delay threshold is less than the second delay threshold; Based on the batch size determination result and the latency urgency level, specifically: for subtask nodes of small batch tasks that are high-urgency tasks, their queue priority is set to the highest priority, and their execution order is arranged at the front of the current queue; For subtask nodes that are small-batch tasks and are medium- or low-urgency tasks, set their queue priority to high priority and arrange their execution order after the highest priority tasks. For subtask nodes of medium-batch tasks that are high-urgency tasks, set their queue priority to high priority and allow them to be aggregated with adjacent medium-batch tasks for execution, provided that the latency limit is met. For subtask nodes of medium-batch tasks that are medium-urgent or low-urgent tasks, set their queue priority to medium priority.

10. The edge-cloud collaborative AI computing power and detection task integrated scheduling method as described in claim 9, characterized in that: The method for determining the execution order and queue priority based on batch size also includes: For subtask nodes of large or very large batches of tasks that are low-urgency tasks, set their queue priority to low priority and place them at the end of the current queue for execution, waiting to be aggregated into a larger batch before execution. The initial queue priority and execution order are modified based on the task queue blocking rate of each scheduling node: when the task queue blocking rate of a scheduling node is greater than the second blocking rate threshold but not greater than the third blocking rate threshold, a downgrade operation is performed on the large batch and ultra-large batch tasks in the queue of that scheduling node; when the task queue blocking rate of a scheduling node is greater than the third blocking rate threshold, the execution of low-priority large batch and ultra-large batch tasks is paused or delayed. Based on the current batch processing capacity and remaining computing power of each scheduling node, the batch processing size of the subtask nodes is dynamically adjusted: when the remaining computing power of the scheduling node is sufficient and the task queue blocking rate is not greater than the first blocking rate threshold, multiple small batch tasks or medium batch tasks are aggregated into large batch tasks; when the remaining computing power of the scheduling node is insufficient or the task queue blocking rate is greater than the second blocking rate threshold, large batch tasks or ultra-large batch tasks are split into multiple small batch tasks or medium batch tasks. Based on the batch size adjustment results, the execution order and queue priority of subtask nodes are re-determined: For small batches of tasks obtained after splitting, they are inserted into the corresponding positions in the current queue according to their latency urgency level and precision lower limit; for large batches of tasks obtained after aggregation, their execution order is shifted to the back, and an aggregation waiting window is set in the queue. Within the waiting window, similar subtask nodes continue to be received. When the waiting window ends or the number of aggregated tasks reaches the preset upper limit, batch inference is triggered; if a high-urgency task appears within the waiting window, the high-urgency task is allowed to jump the queue for execution, and the large batch of tasks that are jumped the queue continue to wait.

Images