End-side cloud cooperation system based on hybrid scheduling strategy

Abstract

The invention relates to the technical field of artificial intelligence and edge computing, in particular to an end-side cloud collaboration system based on a hybrid scheduling strategy. The end-side cloud cooperation system based on the hybrid scheduling strategy comprises a model segmentation module, a state sensing module, a reasoning scheduling module, a model compression and deployment module, an edge optimization engine and a communication synchronization module. According to the end-side cloud cooperation system based on the hybrid scheduling strategy, elastic deployment and dynamic reasoning of a complex model in a multilayer heterogeneous environment are supported, the system state can be sensed in real time, a scheduling path can be optimized, and the robustness of the system is improved; and in combination with model compression and edge operator optimization, the reasoning efficiency is remarkably improved, the communication load is reduced, and then low-delay cooperation between the terminal and the cloud is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and edge computing technology, and particularly relates to an end-edge-cloud collaborative system based on a hybrid scheduling strategy. BACKGROUND

[0002] With the continuous growth of the size of deep learning models, the inference computing demand has far exceeded the processing capacity of most terminal devices. In order to improve the response speed and reduce the energy consumption, end-edge-cloud collaborative inference has become a research hotspot. Through the division and deployment of models on different levels of devices, combined with the unloading decision mechanism, efficient collaborative inference is realized.

[0003] However, most of the current end-edge-cloud inference systems only use static strategies or simple heuristic methods for unloading judgment, lack real-time perception and feedback optimization capability for dynamic resource state, and are difficult to cope with network fluctuations and computing power differences. In addition, how to balance model deployment optimization, edge inference acceleration and cross-device communication efficiency is still a technical difficulty to be solved.

[0004] In order to realize efficient deployment and inference of cross-level intelligent models and improve the intelligence of local decision and the overall performance of the system, the present application proposes an end-edge-cloud collaborative system based on a hybrid scheduling strategy. SUMMARY

[0005] The present application provides a simple and efficient end-edge-cloud collaborative system based on a hybrid scheduling strategy to overcome the shortcomings of the prior art.

[0006] The present application is realized by the following technical solutions:

[0007] An end-edge-cloud collaborative system based on a hybrid scheduling strategy comprises:

[0008] A model division module is used to analyze the completed deep learning model, perform static or dynamic structure division according to the network structure or computation graph, and send the divided sub-models to the model compression and deployment module for allocation to the terminal, edge or cloud for execution according to the computing resources and network conditions.

[0009] The model division module performs static division or dynamic division on the model structure of the deep learning model according to the level of computation intensity, parameter size and dependency relationship.

[0010] The static division refers to division according to fixed boundaries, and the dynamic division refers to real-time re-division according to the real-time state of the device.

[0011] After division, each slice is subjected to lightweight processing by the model compression and deployment module and is deployed to the corresponding nodes of the terminal, edge and cloud.

[0012] a state awareness module, configured to collect current resource state parameters in real time, including processor load, memory occupation, network bandwidth, network delay and battery power of terminal devices, edge nodes and cloud resources;

[0013] a reasoning scheduling module, configured to implement a hybrid scheduling strategy combining a static rule scheduling strategy and a dynamic AI scheduling strategy, to perform intelligent task allocation and intermediate result routing among terminals, edge nodes and clouds, so as to guarantee system stability and performance upper limit;

[0014] The static rule scheduling strategy is an initial offloading strategy, and the dynamic AI scheduling strategy is an offloading decision optimized in real time during task running.

[0015] The initial offloading strategy is generated based on predefined rules of platform capability, network topology and model structure.

[0016] The offloading decision is generated in real time during task running based on a scheduling engine of reinforcement learning or graph neural network.

[0017] The static rule scheduling strategy is based on preset rules, model structure and deployment environment, and performs initial offloading and resource allocation, including one or more of the following:

[0018] determining deployment depth of an edge level according to model hierarchical features;

[0019] dividing delay-sensitive paths and non-sensitive paths according to real-time requirements of tasks;

[0020] deciding whether to migrate part of reasoning to the cloud based on a network bandwidth threshold;

[0021] The dynamic AI scheduling strategy dynamically optimizes decisions based on state awareness and AI reasoning, specifically including:

[0022] current central processing unit (CPU), graphics processing unit (GPU) or neural network processing unit (NPU) utilization of a device;

[0023] network delay and jitter statistical information;

[0024] computational and communication overheads of each segment of a model;

[0025] historical reasoning results and task success rates.

[0026] The reasoning scheduling module performs scheduling strategy decisions, and the entire scheduling process is divided into three stages, namely, an initial deployment stage, a reasoning running stage and a feedback adjustment stage.

[0027] Step S1. Initial deployment stage (static scheduling)

[0028] The inference scheduling module generates an initial model offloading configuration according to a model structure, a network topology, and platform capabilities;

[0029] The model structure includes hierarchical computation and dependency analysis, the network topology includes terminal-edge-cloud connection relationships, and the platform capabilities include device computing power, memory, and power thresholds;

[0030] The static rule scheduling strategy divides the model into several sub-model segments, deploys the front sub-model including input preprocessing and convolution to the terminal, deploys the middle sub-model including feature extraction and fusion layers to the edge, and deploys the rear sub-model including fully connected layers and classifiers to the cloud.

[0031] Step S2. Inference running phase (dynamic scheduling)

[0032] During actual model running, the state perception module continuously perceives the current resource state and calls the inference scheduling module to dynamically execute strategy adjustment, and the process is as follows:

[0033] Step S2.1. The state perception module collects data

[0034] The central processing unit (CPU), graphics processing unit (GPU), and neural network processing unit (NPU) load, network bandwidth, network delay, cache occupancy, and battery power of the terminal device, edge node, and cloud resources are acquired in real time, and a current scheduling graph state input vector is constructed.

[0035] Step S2.2. The AI scheduling engine performs inference

[0036] Based on reinforcement learning or a graph neural network, the state input vector is input, and inference path optimization suggestions are output.

[0037] Step S2.3. Intermediate result routing and offloading switching

[0038] The inference scheduling module controls the transmission path of the intermediate features according to the inference path optimization suggestions; the inference scheduling module supports smooth migration within multiple frames to avoid frequent retransmission and failure.

[0039] Step S3. Feedback adjustment phase (adaptive learning)

[0040] The running results (such as delay, accuracy, and energy consumption) are fed back to the inference scheduling module, the AI model is continuously trained, and the scheduling strategy is updated to adapt to new scenarios and new models.

[0041] If an extreme network failure / node offline situation is encountered, the static rule scheduling strategy is automatically rolled back; after connection recovery, the dynamic AI scheduling strategy is used for dominant decision-making, and the initial offloading strategy is used as a bottom mechanism.

[0042] A model compression and deployment module is configured to implement differentiated compression processing on the deployed model of each layer based on the target platform capability, and deploy the model to a terminal, an edge or a cloud node.

[0043] The differentiated compression processing means include but are not limited to pruning, quantization and distillation, and the processed model retains a customized key precision to reduce resource consumption and model volume and adapt to the capability of a heterogeneous device.

[0044] An edge optimization engine is deployed on an edge server and is configured to deploy a special operator on an edge device, perform fusion optimization of a model graph structure, edge operator replacement and underlying instruction set mapping, so as to realize hardware acceleration and improve inference efficiency.

[0045] For an edge hardware platform, the edge optimization engine calls a special AI acceleration operator library and uses a graph compilation optimization tool for automatic optimization and compilation deployment.

[0046] A communication synchronization module is configured to realize low-delay transmission and synchronization of intermediate features and control information among multiple layers of heterogeneous devices.

[0047] The communication synchronization module supports end-side failover to an edge or local fault tolerance when an edge connection is disconnected, uses feature compression encoding, adaptive bit rate and a caching mechanism to reduce communication burden, and each sub-model segment is executed asynchronously in a dependent order and cooperates through lightweight intermediate feature transmission to realize low-delay transmission and synchronization of intermediate features among different devices and ensure controllable inference delay.

[0048] The end-edge-cloud collaborative system based on a hybrid scheduling strategy has the advantages that the system supports elastic deployment and dynamic inference of a complex model in a multi-layer heterogeneous environment, can perceive a system state in real time and optimize a scheduling path, improves system robustness, significantly improves inference efficiency in combination with model compression and edge operator optimization, reduces communication load, and thus guarantees low-delay collaboration between a terminal and a cloud. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0050] FIG. 1 is a schematic diagram of an end-edge-cloud collaborative system architecture based on a hybrid scheduling strategy. Figure 1 FIG. 1 is a schematic diagram of an end-edge-cloud collaborative system architecture based on a hybrid scheduling strategy.

[0051] FIG. 1 is a schematic diagram of an end-edge-cloud collaborative system architecture based on a hybrid scheduling strategy. Figure 2 FIG. 1 is a schematic diagram of an end-edge-cloud collaborative system architecture based on a hybrid scheduling strategy. DETAILED DESCRIPTION

[0052] In order to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work should fall within the protection scope of the present application.

[0053] The end-edge-cloud collaborative system based on the hybrid scheduling strategy comprises:

[0054] The model splitting module is configured to analyze the trained deep learning model, perform static or dynamic structure division according to a network structure or a computation graph, and send the split sub-models to the model compression and deployment module for execution on the terminal, edge or cloud according to the computation resources and network conditions.

[0055] The model splitting module performs static splitting or dynamic splitting on the model structure of the deep learning model according to the hierarchical computation intensity, parameter size and dependency relationship.

[0056] The static splitting refers to splitting according to a fixed boundary, and the dynamic splitting refers to real-time re-splitting according to the real-time state of the device.

[0057] After splitting, each slice is sent to the model compression and deployment module for lightweight processing and deployment to the corresponding nodes of the terminal, edge and cloud.

[0058] The state-aware module is configured to collect real-time resource state parameters, including the processor load, memory occupation, network bandwidth, network delay and battery power of the terminal device, edge node and cloud resource.

[0059] The inference scheduling module is configured to perform intelligent task allocation and intermediate result routing among the terminal, edge node and cloud according to the hybrid scheduling strategy combining the static rule scheduling strategy and the dynamic AI scheduling strategy, so as to ensure the system stability and performance upper limit.

[0060] The static rule scheduling strategy is an initial offloading strategy, and the dynamic AI scheduling strategy is an offloading decision optimized in real time during task running.

[0061] The initial offloading strategy is generated based on predefined rules of platform capability, network topology and model structure.

[0062] The offloading decision is generated in real time during task running based on the scheduling engine of reinforcement learning or graph neural network.

[0063] The static rule scheduling strategy is based on preset rules, model structure and deployment environment, and performs initial offloading and resource allocation, including one or more of the following:

[0064] Determine the deployment depth of the edge level according to the hierarchical characteristics of the model;

[0065] Divide the delay-sensitive path and the non-sensitive path according to the real-time requirements of the task;

[0066] Determine whether to migrate part of the inference to the cloud based on the network bandwidth threshold;

[0067] The dynamic AI scheduling strategy uses state awareness and AI inference (such as reinforcement learning RL or graph neural network GNN) to dynamically optimize decision-making, specifically including:

[0068] The utilization rate of the central processing unit CPU, graphics processing unit GPU or neural network processing unit NPU of the current device;

[0069] Network delay and jitter statistical information;

[0070] The calculation and communication overhead of each segment of the model;

[0071] Historical inference results and task success rate.

[0072] The inference scheduling module executes the scheduling strategy decision, and the entire scheduling process is divided into three stages: initial deployment stage, inference running stage and feedback adjustment stage;

[0073] Step S1. Initial deployment stage (static scheduling)

[0074] The inference scheduling module generates an initial model offloading configuration according to the model structure, network topology and platform capability;

[0075] Among them, the model structure includes hierarchical calculation and dependency analysis, the network topology includes terminal-edge-cloud connection relationship, and the platform capability includes device computing power, memory and power threshold;

[0076] The static rule scheduling strategy divides the model into several sub-model segments (such as front / middle / last segments), deploys the front sub-model including input preprocessing and convolution to the terminal, deploys the middle sub-model including feature extraction and fusion layer to the edge, and deploys the last sub-model including fully connected layer and classifier to the cloud.

[0077] Example rule: If the terminal neural network processing unit NPU available computing power is lower than the top 2, limit the number of processed model layers to no more than the top 20%; if the edge network delay is less than 20 milliseconds, enable intermediate inference transfer.

[0078] Step S2. Inference running stage (dynamic scheduling)

[0079] During actual model runtime, the state-aware module continuously monitors the current resource state and calls the inference scheduling module to dynamically adjust the strategy. The process is as follows:

[0080] Step S2.1 The state perception module collects data.

[0081] Real-time acquisition of CPU, GPU, and NPU load, network bandwidth, network latency, cache usage, and battery level of terminal devices, edge nodes, and cloud resources; simultaneously constructing the current scheduling graph state vector.

[0082] Step S2.2 The AI ​​scheduling engine performs inference.

[0083] Based on reinforcement learning (such as Deep Q-Network, Proximal Policy Optimization, PPO) or graph neural networks (such as Graph Attention Network, GAT), the input is a state vector; and the output is an inference path optimization suggestion, for example:

[0084] Switch mid-range inference from the edge to the cloud (when the edge is congested);

[0085] Dynamically switch intermediate layer unloading points (e.g., from layer 8 to layer 10);

[0086] Increase concurrent replicas to improve throughput.

[0087] Step S2.3 Intermediate Result Routing and Offloading Switching

[0088] The inference scheduling module controls the transmission path of intermediate features based on inference path optimization suggestions; the inference scheduling module supports smooth migration within multiple frames to avoid frequent retransmissions and failures.

[0089] Step S3. Feedback and Adjustment Phase (Adaptive Learning)

[0090] The results of the operation (such as latency, accuracy, and energy consumption) are fed back to the inference scheduling module to continuously train the AI ​​model and update the scheduling strategy to adapt to new scenarios and new models.

[0091] In the event of extreme network failures or node offline situations, the system will automatically revert to the static rule scheduling strategy. Once the connection is restored, the dynamic AI scheduling strategy will take the lead in making decisions, with the initial unloading strategy serving as a fallback mechanism.

[0092] The model compression and deployment module is used to perform differentiated compression processing on the models deployed at each level based on the target platform's capabilities, and deploy them to terminals, edge, or cloud nodes;

[0093] The differentiated compression processing methods include, but are not limited to, pruning, quantization and distillation. The processed model retains custom key precision to reduce resource consumption and model size, and adapt to heterogeneous device capabilities.

[0094] The edge optimization engine, deployed on edge servers, is used to deploy dedicated operators on edge devices to perform model graph structure fusion optimization, edge operator replacement, and underlying instruction set mapping to achieve hardware acceleration and improve inference efficiency.

[0095] For edge hardware platforms, the edge optimization engine calls dedicated AI acceleration operator libraries (such as the NPU optimization library for neural network processors, TensorRT for tensor inference optimizer, and OpenVINO for open visual inference toolkit) and uses graph compilation optimization tools (such as the TVM machine learning compiler) for automated optimization, compilation and deployment.

[0096] The communication synchronization module is used to achieve low-latency transmission and synchronization of intermediate features and control information between multi-layer heterogeneous devices.

[0097] The communication synchronization module supports end-side fault transfer to the edge or local fault tolerance when the edge connection is disconnected. It adopts feature compression coding, adaptive bit rate and caching mechanism to reduce communication burden. Each sub-model segment is executed asynchronously in the order of dependency and coordinated through lightweight intermediate feature transmission to achieve low-latency transmission and synchronization of intermediate features between different devices, ensuring controllable inference latency.

[0098] Example 1: Manufacturing Scenario

[0099] This edge-cloud collaborative system, based on a hybrid scheduling strategy, achieves automated deployment through an AI model management platform, enabling edge-cloud collaborative inference in scenarios such as industrial quality inspection, smart security, and autonomous driving. Taking image recognition tasks as an example, the camera front end is only responsible for image preprocessing and feature extraction, edge nodes complete target recognition and alarm generation, and the central cloud is used for model training and iterative optimization, with overall latency controlled at the millisecond level.

[0100] In summary, this edge-cloud collaborative system based on a hybrid scheduling strategy supports the elastic deployment and dynamic inference of complex models in multi-layered heterogeneous environments. It can perceive the system status in real time and optimize the scheduling path, thereby improving the system's robustness. By combining model compression and edge operator optimization, it significantly improves inference efficiency, reduces communication load, and ensures low-latency collaboration between the terminal and the cloud.

[0101] The embodiments described above are merely one specific implementation of the present invention. Ordinary changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included within the protection scope of the present invention.

Claims

1. An edge-cloud collaborative system based on a hybrid scheduling strategy, characterized in that: include: The model segmentation module is used to analyze the trained deep learning model, perform static or dynamic structural division based on the network structure or computation graph, and send the segmented sub-models to the model compression and deployment module for execution on the terminal, edge or cloud according to computing resources and network conditions. The status awareness module is used to collect current resource status parameters in real time, including processor load, memory usage, network bandwidth, network latency, and battery level of terminal devices, edge nodes, and cloud resources. The inference scheduling module is used to perform intelligent task allocation and intermediate result routing between terminals, edge nodes and the cloud based on a hybrid scheduling strategy that combines static rule scheduling strategy and dynamic AI scheduling strategy, so as to ensure system stability and performance limits. The model compression and deployment module is used to perform differentiated compression processing on the models deployed at each level based on the target platform's capabilities, and deploy them to terminals, edge, or cloud nodes; The edge optimization engine, deployed on edge servers, is used to deploy dedicated operators on edge devices to perform model graph structure fusion optimization, edge operator replacement, and underlying instruction set mapping to achieve hardware acceleration and improve inference efficiency. The communication synchronization module is used to achieve low-latency transmission and synchronization of intermediate features and control information between multi-layer heterogeneous devices.

2. The edge-cloud collaborative system based on a hybrid scheduling strategy according to claim 1, characterized in that: The model segmentation module performs static or dynamic segmentation of the deep learning model structure based on the hierarchical computational density, parameter scale, and dependencies. Static segmentation refers to segmentation according to fixed boundaries, while dynamic segmentation refers to re-segmentation based on the real-time status of the equipment. After being segmented, each slice is processed by the model compression and deployment module to perform lightweight processing and is deployed to the corresponding nodes on the terminal, edge and cloud.

3. The edge-cloud collaborative system based on a hybrid scheduling strategy according to claim 1, characterized in that: The static rule scheduling strategy is the initial unloading strategy, while the dynamic AI scheduling strategy is an unloading decision that is optimized in real time during task execution. The initial unloading strategy is generated based on predefined rules of platform capabilities, network topology, and model structure. The unloading decision is generated in real time during task execution, based on a scheduling engine using reinforcement learning or graph neural networks.

4. The edge-cloud collaborative system based on a hybrid scheduling strategy according to claim 3, characterized in that: The static rule scheduling strategy, based on preset rules, model structure, and deployment environment, performs initial unloading and resource allocation, including one or more of the following: The deployment depth of the edge layer is determined based on the hierarchical characteristics of the model; Divide latency-sensitive paths into latency-insensitive paths based on the real-time requirements of the task; Whether to migrate part of the inference to the cloud is determined based on network bandwidth thresholds; The dynamic AI scheduling strategy uses state awareness and AI reasoning to dynamically optimize decisions, specifically including: Current device CPU, GPU, or NPU utilization; Network latency and jitter statistics; The computational and communication overhead of each segment of the model; Historical reasoning results and task success rate.

5. The edge-cloud collaborative system based on a hybrid scheduling strategy according to claim 4, characterized in that: The inference scheduling module executes scheduling strategy decisions. The entire scheduling process is divided into three stages: the initial deployment stage, the inference operation stage, and the feedback adjustment stage. Step S1. Initial Deployment Phase The inference scheduling module generates the initial model unloading configuration based on the model structure, network topology, and platform capabilities. The model structure includes hierarchical computation and dependency analysis, the network topology includes the terminal-edge-cloud connection relationship, and the platform capabilities include the device's computing power, memory, and power thresholds. The static rule scheduling strategy divides the model into several sub-model segments, deploys the front-end sub-model, including input preprocessing and multi-level convolution, on the terminal, deploys the middle-end sub-model, including feature extraction and fusion layers, on the edge, and deploys the back-end model, including fully connected layers and classifiers, on the cloud. Step S2. Inference Execution Phase During actual model runtime, the state-aware module continuously monitors the current resource state and calls the inference scheduling module to dynamically adjust the strategy. The process is as follows: Step S2.1 The state perception module collects data. Real-time acquisition of CPU, GPU, and NPU load, network bandwidth, network latency, cache usage, and battery level of terminal devices, edge nodes, and cloud resources; simultaneously constructing the current scheduling graph state input vector; Step S2.2 The AI ​​scheduling engine performs inference. Based on reinforcement learning or graph neural networks, the input state vector is used, and inference path optimization suggestions are output. Step S2.3 Intermediate Result Routing and Unloading Switching The inference scheduling module controls the transmission path of intermediate features based on inference path optimization suggestions; the inference scheduling module supports smooth migration within multiple frames to avoid frequent retransmissions and failures. Step S3. Feedback and Adjustment Phase The results of the operation, including latency, accuracy and energy consumption, are fed back to the inference scheduling module to continuously train the AI ​​model and update the scheduling strategy to adapt to new scenarios and new models. In the event of extreme network failures or node offline situations, the system will automatically revert to the static rule scheduling strategy. Once the connection is restored, the dynamic AI scheduling strategy will take the lead in making decisions, with the initial unloading strategy serving as a fallback mechanism.

6. The edge-cloud collaborative system based on a hybrid scheduling strategy according to claim 1, characterized in that: The differentiated compression processing methods include, but are not limited to, pruning, quantization and distillation. The processed model retains custom key precision to reduce resource consumption and model size, and adapt to heterogeneous device capabilities.

7. The edge-cloud collaborative system based on a hybrid scheduling strategy according to claim 1, characterized in that: For edge hardware platforms, the edge optimization engine calls a dedicated AI acceleration operator library and uses graph compilation optimization tools for automated optimization and compilation deployment.

8. The edge-cloud collaborative system based on a hybrid scheduling strategy according to claim 1, characterized in that: The communication synchronization module supports end-side fault transfer to the edge or local fault tolerance when the edge connection is disconnected. It adopts feature compression coding, adaptive bit rate and caching mechanism to reduce communication burden. Each sub-model segment is executed asynchronously in the order of dependency and coordinated through lightweight intermediate feature transmission to achieve low-latency transmission and synchronization of intermediate features between different devices.

Images