A cloud-edge-terminal collaborative inference method and system for a heterogeneous edge cluster

Abstract

This invention provides a cloud-edge-device collaborative inference method and system for heterogeneous edge clusters, relating to the fields of artificial intelligence inference systems and distributed computing technology. The method includes: real-time acquisition of node resources, network status, and video stream information of the heterogeneous edge cluster; dynamic selection of the optimal feature calculation node for each video stream to complete the video stream feature extraction task; adaptive routing of the video stream features to the most suitable rule judgment node after feature calculation, forming a continuous pipeline operation from video stream inflow to feature calculation and then to inference decision-making; the rule judgment node performs inference tasks based on the received video stream features, generates structured detection results and alarm events, and efficiently reports the result-level data to the cloud, completing the closed-loop scheduling and execution of the entire cloud-edge-device collaborative inference pipeline; this invention achieves effective utilization of heterogeneous resources under conditions of significant network jitter and uneven computing power distribution, improving system stability and end-to-end service quality.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence inference systems and distributed computing technology, specifically to a cloud-edge-device collaborative inference method and system for heterogeneous edge clusters. Background Technology

[0002] In scenarios such as smart construction site safety production and park security supervision, it is often necessary to perform real-time identification and alarm closed-loop of multiple camera videos on site, such as detecting the wearing status of personnel's protective equipment and triggering safety prompts. Such services usually have engineering requirements such as continuous operation, stable end-to-end latency, and multi-channel concurrent scalable processing.

[0003] The deployment environment for the above scenarios faces significant constraints: First, edge network links may experience bandwidth limitations, jitter, packet loss, and short-term disconnections. Uploading video or large-scale intermediate data back to the cloud for processing can easily introduce bandwidth overhead and latency fluctuations, while also increasing privacy and compliance risks. Second, to achieve low-latency processing on-site, the edge often employs heterogeneous clusters composed of multiple nodes or multiple nodes. Different nodes possess accelerated inference capabilities and general processing capabilities, resulting in uneven distribution of computing power and dynamic resource levels that change with load. When multiple video streams are accessed concurrently, without a task distribution and overall scheduling mechanism for heterogeneous resources, an imbalance can easily occur, with some high-performance computing resources idling while others are queued and congested. This leads to decreased throughput and increased end-to-end latency tails, affecting the stability of the alarm closed-loop.

[0004] In existing technologies, common solutions deploy the entire model on a single node or allocate traffic based on simple strategies such as polling or minimum queues. These solutions often treat the inference process as a monolithic black box, making it difficult to map computationally intensive and logic-light stages to different hardware resources according to their characteristics. Furthermore, they lack a global coordination mechanism for multi-board heterogeneous clusters, making it difficult to continuously suppress congestion and long tails under conditions of varying loads, resource fluctuations, and weak networks. While some solutions attempt to decompose the model, they lack unified data contracts and supporting designs for cross-node communication and reliable delivery. The benefits of collaborative execution are easily compromised by intermediate data transmission overhead or link fluctuations.

[0005] Therefore, existing technologies treat inference tasks as a whole and execute them on black-box hardware. They lack fine-grained decomposition and dynamic scheduling mechanisms for heterogeneous computing power, resulting in low resource utilization, large latency fluctuations, and difficulty in achieving stable and efficient collaborative inference under weak edge network conditions. Summary of the Invention

[0006] To address issues such as bandwidth constraints, network jitter, instability of multi-path concurrent inference, and uneven distribution of computing power at the edge in edge-side inference scenarios, this invention proposes a cloud-edge-device collaborative inference method and system for heterogeneous edge clusters. This method effectively utilizes heterogeneous resources and improves system stability and end-to-end service quality under conditions of significant network jitter and uneven distribution of computing power.

[0007] According to some embodiments, the present invention adopts the following technical solution: A cloud-edge-device collaborative inference method for heterogeneous edge clusters uses multiple video streams as the scheduling granularity. It focuses on constructing a dynamically adaptive cloud-edge-device collaborative inference pipeline. By real-time sensing of node resources and network status in the heterogeneous edge cluster, it achieves three-layer collaborative mapping and pipeline orchestration of video streams, feature computation nodes, and rule judgment nodes along the inference task path. Specifically, it includes: The system acquires node resources, network status, and video stream information of heterogeneous edge clusters in real time, uses multiple video streams to be processed as pipeline inputs, divides the nodes in the cluster into functional categories, and clarifies the roles of feature calculation nodes and rule judgment nodes. Based on the real-time node resources and network status of the heterogeneous edge cluster, as well as video stream information, the optimal feature calculation node is dynamically selected for each video stream to complete the video stream feature extraction task. After the feature calculation is completed, the load and location information of the node are judged according to the current cluster rules, and the video stream features are adaptively routed to the most suitable rule judgment node, forming a continuous pipeline operation from video stream to feature calculation and then to inference decision. The rule-based judgment node performs inference tasks based on the features of the received video stream, generates structured detection results and alarm events, and efficiently reports the result-level data to the cloud, completing the closed-loop scheduling and execution of the entire cloud-edge-device collaborative inference pipeline.

[0008] According to some embodiments, the present invention adopts the following technical solution: A cloud-edge-device collaborative inference system for heterogeneous edge clusters includes: The acquisition module is configured to: acquire node resources, network status, and video stream information of the heterogeneous edge cluster in real time; use the multiple video streams to be processed as pipeline input objects; divide the nodes in the cluster into functional categories; and clarify the role positioning of feature calculation nodes and rule judgment nodes. The routing module is configured to dynamically select the optimal feature calculation node for each video stream based on the real-time node resources and network status of the heterogeneous edge cluster and video stream information, and complete the video stream feature extraction task. After the feature calculation is completed, the module determines the load and location information of the node according to the current cluster rules, and adaptively routes the video stream features to the most suitable rule judgment node, forming a continuous pipeline operation from video stream inflow to feature calculation and then to inference decision. The reporting module is configured as follows: the rule judgment node performs inference tasks based on the features of the received video stream, generates structured detection results and alarm events, and efficiently reports the result-level data to the cloud, completing the closed-loop scheduling and execution of the entire cloud-edge-device collaborative inference pipeline.

[0009] According to some embodiments, the present invention adopts the following technical solution: A computer program product includes a computer program that, when executed by a processor, implements the cloud-edge-device collaborative inference method for heterogeneous edge clusters.

[0010] According to some embodiments, the present invention adopts the following technical solution: A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the cloud-edge-device collaborative inference method for heterogeneous edge clusters.

[0011] According to some embodiments, the present invention adopts the following technical solution: An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the cloud-edge-device collaborative inference method for heterogeneous edge clusters.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention divides nodes in a cluster into feature computing nodes and rule judgment nodes based on their functions, forming a collaborative inference pipeline of feature computing, feature transmission, and rule judgment. This allows the cross-node transmission object to be lightweight video stream features, while computationally intensive feature extraction and inference calculations are performed on feature computing nodes with strong computing resources. This enables heavy computational loads to be borne by accelerated computing units, while lightweight logic and rules are borne by general-purpose processors, reducing cross-node transmission overhead and suppressing end-to-end long tails caused by network jitter, thereby improving overall effective utilization. This invention is based on binary feasibility mask and cooperative state representation. It integrates the utilization rate of accelerated computing units of feature computing nodes, the degree of queue backlog, and the network cost of video stream source to achieve one-to-one mapping and dynamic rebinding of each video stream and feature computing nodes. Under multi-channel concurrency, it achieves load balancing and suppresses local hotspots. This invention adopts a result-level reporting mechanism, transmitting only structured detection results and alarm events from the edge to the cloud, without transmitting the original video data, thereby reducing bandwidth consumption and privacy leakage risks. At the same time, reliable delivery under weak network conditions is achieved through local temporary storage, confirmation retransmission, and idempotent deduplication. Attached Figure Description

[0013] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0014] Figure 1 This is a flowchart of the method framework for Example 1.

[0015] Figure 2 This is a flowchart of the method in Example 1. Detailed Implementation

[0016] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0017] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0018] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0019] The inventive concept of this invention is as follows: Using multiple video streams as the scheduling granularity, and focusing on building a dynamic and adaptive cloud-edge-device collaborative inference pipeline, this approach achieves three-layer collaborative mapping and pipeline orchestration of video streams, feature computation nodes, and rule judgment nodes along the inference task path by real-time sensing of node resources and network status in heterogeneous edge clusters. Feature computation nodes utilize accelerated computing units to handle heavy computational loads such as convolution and feature extraction, while rule judgment nodes employ general-purpose processors to handle lighter logic loads such as decoding, post-processing, and rule judgment. The cross-node transmission object is lightweight video stream features. Based on collaborative state representation and binary feasibility masks, the optimal feature computation node is dynamically selected for each video stream, and the video stream features are adaptively routed to the most suitable rule judgment node, forming a continuous pipeline operation from video stream inflow to feature computation and then to inference decision-making. After generating structured detection results and alarm events, the rule judgment node reports the results to the cloud, completing the closed-loop scheduling and execution of the entire cloud-edge-device collaborative inference pipeline.

[0020] To avoid ambiguity, in this scheme, "stage" refers to the logical processing step in the inference chain, "component" refers to the software service unit that carries one or more stages, and "node" refers to the physical or virtual computing carrier on which the component is deployed. Stages and components do not necessarily correspond one-to-one. A component can carry a single stage or multiple adjacent stages. In this embodiment, the "video stream feature" is preferably the prediction tensor output by the detection head of the target detection model.

[0021] Example 1 One embodiment of the present invention provides a cloud-edge-device collaborative inference method for heterogeneous edge clusters. It uses multiple video streams as the scheduling granularity and focuses on constructing a dynamically adaptive cloud-edge-device collaborative inference pipeline. By real-time sensing of node resources and network status in the heterogeneous edge cluster, it achieves three-layer collaborative mapping and pipeline orchestration of video streams, feature calculation nodes, and rule judgment nodes along the inference task path. Specifically, it includes: Step S1: Real-time acquisition of node resources, network status, and video stream information of the heterogeneous edge cluster; use the multiple video streams to be processed as pipeline input objects; perform functional division of nodes in the cluster; and clarify the role positioning of feature calculation nodes and rule judgment nodes. Specifically, at each discrete time step, information from the cluster side, video stream side, and link side is synchronously observed and acquired, and timestamps are aligned. The cluster side includes the utilization and available capacity of the accelerated computing units of the feature computing nodes, the utilization and available capacity of the general processors of the rule judgment nodes, and the input queue length and queue growth rate of each node. The video stream side includes stream identifier, resolution, frame rate, bit rate, service priority, and end-to-end latency budget. The link side includes bandwidth usage, round-trip time (RTT), packet loss rate, and short-term unreachability flag. The observed information is mapped to a set of collaborative objects, including video stream objects, feature computation node objects, and rule judgment node objects. Normalized feature vectors are assembled for different types of objects, which at least include available resources, utilization rate, queue length, and time context, and retain stream identifiers and service priority labels. Among them, the available computing power of feature computation nodes is determined based on the utilization rate and available capacity of their accelerated computing units, the queue pressure is determined based on their input queue length and queue growth rate, and the service priority of video streams is determined based on the service priority information on the video stream side. The remaining processing capacity of rule judgment nodes is determined based on their general processor utilization rate and available capacity, and the queue pressure is determined based on their input queue length and queue growth rate.

[0022] Based on business semantics, multiple types of relationships are generated to form the topological skeleton of the current collaborative state, including at least the candidate relationship between video stream and feature calculation node, the collaborative relationship between feature calculation node and rule judgment node, and the relationship between node and queue occupancy.

[0023] For the video stream-feature calculation node candidate relationship and the feature calculation node-rule judgment node collaborative relationship, further link features such as bandwidth usage, RTT and packet loss rate are added, and the link cost from the video stream source to the candidate feature calculation node and the link cost from the feature calculation node to the candidate rule judgment node are calculated accordingly. Combining the end-to-end latency budget of the video stream, the current cumulative stage latency and the node queue waiting status, the real-time risk of the candidate rule judgment node is estimated.

[0024] For candidate pairs of "video stream - feature computing node", a binary feasibility mask M is generated based on the video stream input load, the current available computing power and queue pressure of the feature computing node, and the cross-node link quality constraints. When the candidate feature computing node meets the available resource threshold, queue length threshold, and link RTT and packet loss rate constraints, M(s,c)=1; otherwise, M(s,c)=0. This mask is used to shield infeasible or high-risk allocation actions in the subsequent scheduling phase, where s is the video stream and c is the candidate feature computing node. The objects, features, relationships, and feasibility masks are uniformly assembled into a cooperative state representation for the current time step. The output is a state data packet containing the cooperative state topology, object feature set, object type and relationship type identifiers, and binary feasibility mask, which serves as the input for step S2.

[0025] Step S2: Based on the real-time node resources and network status of the heterogeneous edge cluster and the video stream information, the optimal feature calculation node is dynamically selected for each video stream to complete the video stream feature extraction task; after the feature calculation is completed, the load and location information of the node are judged according to the current cluster rules, and the video stream features are adaptively routed to the most suitable rule judgment node, forming a continuous pipeline operation from video stream to feature calculation and then to inference decision. Based on the collaborative state representation and binary feasibility mask generated in step S1, entry point allocation is performed for new access flows, and necessary rerouting and dynamic rebinding are performed for existing flows. For candidate feature calculation nodes and candidate rule judgment nodes, candidate scores are generated based on node resource status, queue status, link cost, and service priority or real-time risk, and node allocation and dynamic routing are completed accordingly.

[0026] Specifically, for each video stream s, under the condition of satisfying In the set of candidate feature computing nodes, the scheduler generates candidate scores based on the available computing power, queue pressure, link cost from the video stream source to the candidate node, and service priority of the candidate nodes. The candidate scores can be expressed as:

[0027] in, Represents candidate feature calculation nodes Available computing power Represents candidate feature calculation nodes queue pressure, This represents the distance from the source of video stream s to the candidate feature calculation node. Link cost, This indicates the service priority of video stream s. These are weighting coefficients, and all are greater than 0.

[0028] The scheduler satisfies The candidate feature calculation node with the highest score is selected from the set of candidate feature calculation nodes as the current binding node. When the queue waiting time, link quality, or stage processing latency of an already bound node continuously exceeds a preset migration threshold, dynamic rebinding of the video stream is triggered.

[0029] After feature calculation is completed, the predicted tensor output by the feature calculation node is used as the video stream feature. For each predicted tensor, from the set of candidate rule judgment nodes that satisfy the constraints of remaining general processing capacity, queue length, and link quality, a routing score is generated by comprehensively considering the remaining processing capacity of the candidate rule judgment nodes, queue pressure, link cost from the feature calculation node to the candidate rule judgment node, and real-time risk. The routing score can be expressed as:

[0030] in, Indicates the node for judging candidate rules The remaining processing capacity Indicates the node for judging candidate rules queue pressure, This represents the process from the feature calculation node to the candidate rule judgment node. Link cost, Indicates the node for judging candidate rules Real-time risks, These are weighting coefficients, and all are greater than 0.

[0031] The scheduler selects the candidate rule-judgment node with the highest score as the target routing node and issues a dynamic routing table. When the queue of the target rule-judgment node continues to back up, the link deteriorates, or the health check fails, the routing score is recalculated and rerouting is performed.

[0032] The transmission process follows a unified data contract and is subject to link quality constraints. When RTT increases or packet loss increases, timeout and retry strategies can be adjusted or backoff paths can be triggered to ensure link continuity, forming a continuous pipeline operation from video inflow to feature calculation and then to inference decision-making.

[0033] Step S3: The rule judgment node performs inference tasks based on the features of the received video stream, generates structured detection results and alarm events, and efficiently reports the result-level data to the cloud, completing the closed-loop scheduling and execution of the entire cloud-edge-device collaborative inference pipeline.

[0034] Specifically, the rule judgment node receives the prediction tensor sent by the feature calculation node, performs decoding, nonmaximum suppression and business rule fusion on the prediction tensor, and generates structured detection results and alarm events; the structured detection results include at least the detection box, category, confidence level, timestamp, stream identifier and rule hit information.

[0035] Generate idempotent keys for alarm events and perform deduplication to avoid duplicate alarms caused by retries. At the same time, record the processing latency, queue waiting time and throughput statistics of this stage. The edge-to-cloud architecture employs a result-level reporting mechanism, uploading only structured detection results and alarm events, without transmitting raw video data or intermediate features, in order to reduce bandwidth consumption and minimize the risk of privacy leaks. When the cloud is unreachable or the link is briefly interrupted, the structured results are temporarily stored locally and the delivery status is recorded; after the link is restored, the data is retransmitted in chronological order; the combination of idempotent keys and confirmation mechanisms ensures that the results are eventually reachable and that no duplicate alarms are triggered. The rule-judgment node will feed back processing latency, queue level, and alarm success rate to serve as the basis for dynamic rebinding and routing strategy adjustment in step S2, thus completing the closed-loop scheduling and execution of the entire cloud-edge-device collaborative inference pipeline.

[0036] The above steps constitute a complete end-to-end inference process, referring to the entire chain of processing from the moment the edge camera acquires video and accesses it via the edge, to the rule judgment node generating structured detection results and / or alarm events and reporting the results to the cloud. This entire chain can be abstracted into a phased chain, which includes at least the following stages: edge video acquisition, edge video access and decoding, preprocessing, feature calculation, prediction output, post-processing, rule judgment, and result reporting. The video access and decoding stage and the preprocessing stage are preferably deployed on the edge gateway node or access node; the feature calculation stage and the prediction output stage are deployed on the feature calculation node; the post-processing stage, the rule judgment stage, and the alarm generation stage are deployed on the rule judgment node; and the result reporting stage is executed by the rule judgment node or the edge gateway node to the cloud.

[0037] Provide a specific example, such as Figure 1As shown, the end-to-end inference link is decomposed into deployable, staged component services, and model segmentation is used as the method for forming these componentized services, so that the model forward inference is encapsulated into multiple continuous subgraph stages. Based on this, multiple video streams are used as scheduling objects, and tasks are mapped to heterogeneous resources by combining node-side computing power and network status, forming a cross-device, edge, and cloud collaborative inference pipeline. Furthermore, cluster-level multi-board scheduling and communication reliability mechanisms are used to suppress end-to-end long-tail drag, such as... Figure 2 As shown, the specific steps are as follows: Step 1: Data Acquisition and Collaborative Status Construction The system collects video stream information, edge cluster resource status, and network link status at discrete time steps. Based on this, it constructs a collaborative state representation that includes three types of objects: "video stream, feature calculation node, and rule judgment node" and multiple types of relationships such as "traffic binding, cross-node collaborative link, node occupancy, and queue pressure". This forms an object feature set and a binary feasibility mask for "video stream - feature calculation node", which serves as the input for subsequent model segmentation component orchestration, collaborative inference path selection, and multi-board scheduling.

[0038] At each discrete time step, the video stream, feature computation nodes, and rule judgment nodes are modeled as three types of objects. Based on the candidate relationship between video stream and feature computation nodes, the collaborative relationship between feature computation nodes and rule judgment nodes, and the node-queue occupancy relationship, multiple types of relationships are constructed to generate a binary feasibility mask for video stream and feature computation nodes. .

[0039] Specifically, at discrete time step t, resource levels and queue status of each node are retrieved from the edge monitoring and service observation interface, input characteristics and service priorities of multiple video streams are retrieved from the access layer, and bandwidth usage, RTT, and packet loss rate information are retrieved from the network probing or communication statistics interface; based on this, the current collaborative state is constructed and a feasibility mask is output. .

[0040] Step 1-1: Aligning observation data with time: Three types of information are collected synchronously at each discrete time step and the timestamps are aligned.

[0041] The cluster side includes the utilization and available capacity of the accelerated computing units of the feature computing nodes, the utilization and available capacity of the general processors of the rule judgment nodes, and the length and growth rate of the input queue at each stage; The video stream side includes stream identifier, resolution, frame rate, bitrate and service priority, and end-to-end latency budget; Link-side metrics include bandwidth usage, RTT and packet loss rate, and short-term unreachability indicators.

[0042] These metrics come from the monitoring and communication statistics interfaces of the edge cluster and are used to characterize current schedulability, queuing pressure, and weak network risks.

[0043] Steps 1-2: Object Set and Feature Assembly The observations are mapped to a set of collaborative objects, including video stream objects, feature calculation node objects, and rule judgment node objects. Normalized feature vectors are assembled for different types of objects, including at least available resources, utilization rate, queue length, recent latency statistics, and time context, while retaining necessary stream / node identifiers and business priority labels.

[0044] Steps 1-3: Relationship Building Generate multiple types of relationships based on business semantics to form the topological skeleton of the current collaborative state, including at least the video stream-feature calculation node candidate relationship, the feature calculation node-rule judgment node collaborative relationship, and the node-queue occupancy relationship.

[0045] Steps 1-4: Feasibility Mask Generation For candidate pairs of "video stream -> feature computation node", a binary feasibility mask is generated based on the video stream input load, the remaining acceleration computing power of the candidate feature computation node, the current queue pressure of the candidate node, and the link quality from the video stream source to the candidate node. .

[0046] Specifically, for any video stream s and candidate feature calculation node c, if the remaining acceleration computing power of candidate feature calculation node c is not lower than the minimum computing power threshold required to process video stream s, and its queue length is not higher than a preset queue length threshold, and the round-trip time (RTT) from the video stream source to candidate feature calculation node c is not higher than a preset delay threshold and the packet loss rate is not higher than a preset packet loss rate threshold, then is recorded as follows: Otherwise, remember The above thresholds can be statically set or dynamically adjusted based on business latency budgets, historical statistical results, or manual configuration.

[0047] Steps 1-5: Cooperative state assembly and interface output: The objects, features, relationships, and feasibility masks are uniformly assembled into a data packet for the current time step and output as direct input for subsequent model segmentation to generate componentized services, collaborative inference execution, and multi-board overall scheduling. Preferably, the data packet includes at least the collaborative state topology for the current time step, the set of object features, the object type identifier, the relationship type identifier, and the binary feasibility mask of the video stream to the candidate feature calculation node.

[0048] Step 2: Model Segmentation and Componentized Service Generation The execution model of the monolithic inference model is divided and encapsulated into stage service components, transforming the inference chain from monolithic inference into a programmable componentized pipeline, forming a system architecture of "perception service component - decision service component - gateway service component", providing controllable objects for heterogeneous resource mapping, cross-node collaborative inference and multi-board scheduling.

[0049] In this embodiment, YOLOv8 is selected as the example model for object detection. YOLOv8 includes at least a Backbone, a Neck, and a Detect Head. A "detector head post-cut" approach is adopted, using the output of the Detect Head as the cross-node segmentation boundary, rather than separating the Detect Head from the model and executing it at the rule-making node.

[0050] Specifically, the feature calculation node performs a forward inference process including Backbone, Neck, and Detect Head and outputs a prediction tensor; the rule judgment node receives the prediction tensor and performs decoding, nonmaximum suppression, and business rule fusion to generate structured detection results and alarm events.

[0051] Step 2-1, Stage Link Abstraction: The end-to-end inference process is abstracted into a stage link, which includes at least the stages of video access and decoding, preprocessing, feature calculation, prediction output, postprocessing, rule judgment and result reporting. Input and output objects and stage delay budgets are defined for each stage, so that the dependencies between stages are clear and describable and can be directly used by the scheduler.

[0052] Step 2-2: Determining the segmentation point after the detection head: Generate a set of candidate segmentation points based on the model structure, estimate the intermediate output size and communication cost for each segmentation point, and select the segmentation point located after the YOLO detection head so that the computationally intensive convolution, feature extraction and prediction output stages are executed on the feature calculation node side. The cross-node transmission object is a lightweight prediction tensor, avoiding the communication overhead from offsetting the collaborative benefits due to the transmission of the original video or high-dimensional features.

[0053] Steps 2-3: Stage Component Encapsulation and Interface Solidification: The segmented continuous subgraphs are encapsulated into stage service components, and the service interfaces and input / output formats are solidified. The perception service component performs feature calculation and prediction output, generating a prediction tensor; the decision service component performs decoding, non-maximum suppression, and rule fusion on the prediction tensor, outputting structured detection results and alarm events; the gateway service component handles flow access orchestration, routing, scheduling control, and state synchronization. Each component exposes health check and performance statistics interfaces to support elastic scaling and overload rollback.

[0054] Steps 2-4: Define unified data contract and reliable semantics: Define the data contract for inter-stage interaction, which should include at least frame-level data objects, prediction tensor objects, structured detection result objects, and alarm event objects. Specify the serialization method, version number, timeout, retry, and idempotent semantics. At the same time, set the size limit and necessary compression strategies for cross-node transmission objects to ensure that transmission behavior is predictable and does not generate duplicate alarms under weak network conditions.

[0055] Steps 2-5: Component deployment, orchestration, and decoupling: Components are deployed independently within the cluster orchestration system and support on-demand scaling. Computationally intensive phases are prioritized for feature computing nodes, and rule judgment and alarm logic are prioritized for deployment to rule judgment nodes. Decoupling enables fault isolation, canary upgrades, and stable evolution, providing controllable operational units for subsequent overall scheduling.

[0056] Step 3, Edge Collaborative Reasoning Execution: The collaborative reasoning paradigm based on "feature computation - feature transmission - rule judgment" forms a cross-node pipeline between feature computation nodes and rule judgment nodes, so that the heavy computational load is borne by the feature computation nodes, and the light logic and rules are borne by the rule judgment nodes, reducing the end-to-end long tail and improving the overall effective utilization rate.

[0057] In this embodiment, the feature calculation node is responsible for convolution and feature extraction and outputs the prediction tensor, while the rule judgment node is responsible for decoding, non-maximum suppression, rule fusion and result return; the cross-node transmission object is the prediction tensor, forming a collaborative inference link that can enter a multi-board pipeline.

[0058] Step 3-1, Edge-side Video Access and Preprocessing: The camera terminal acquires video and sends it to the edge side via the access protocol. The access and decoding components perform multi-channel management, decoding, and backpressure control, while the preprocessing component performs input normalization and format conversion, generating frame-level input objects carrying timestamps, sequence numbers, and stream identifiers. The access and decoding components handle the video access and decoding stages, while the preprocessing component handles the preprocessing stage.

[0059] Step 3-2, Feature Calculation and Prediction Output: On the target feature calculation node side determined by the candidate scoring in step S2, the perception service component performs feature calculation and prediction output on the frame-level input, generates a prediction tensor, and records the processing latency, queue waiting time and throughput statistics of this stage, which serve as the basis for the scheduler to identify bottlenecks and make migration decisions.

[0060] Step 3-3, Feature Transmission: The predicted tensor is transmitted from the feature calculation node to the target rule judgment node according to the dynamic routing table determined by the routing score in step S2 and issued by the scheduler. The transmission process follows the data contract defined in step 2 and is subject to link quality constraints. When RTT increases or packet loss increases, the timeout and retry strategy can be adjusted or a backoff path can be triggered to ensure link continuity.

[0061] Steps 3-4: Rule Judgment and Alarm Generation: The decision service component performs decoding, non-maximum suppression, and rule fusion on the prediction tensor at the rule judgment node side, generates structured detection results and alarm events, and returns them to the business side; it generates idempotent keys for alarm events and performs deduplication to avoid repeated alarms caused by retries, and records the processing latency and queue level at this stage to support the scheduling closed loop.

[0062] Steps 3-5: Multi-board pipeline parallelism: In a multi-board environment, multiple feature calculation nodes are allowed to perform feature calculations in parallel, and multiple rule judgment nodes are allowed to perform rule judgments in parallel. The gateway completes the routing, forwarding and call orchestration, so that multiple video streams can be pushed forward in parallel between multiple boards, thereby improving throughput and suppressing end-to-end latency.

[0063] Step 4: Multi-board overall scheduling closed loop: The cluster-level scheduling closed loop runs in discrete time steps, and generates scheduling decisions by integrating information such as resource utilization, queue status, end-to-end and stage latency, bandwidth, RTT and packet loss rate, and stream priority. It dynamically allocates and migrates multiple video streams, and performs overload protection and fallback when overloaded or when the link degrades, so as to maintain stable service quality and improve system carrying capacity.

[0064] Step 4-1, Indicator Aggregation and Status Update: The scheduler periodically aggregates the online collaborative status formed in Step 1, including at least the utilization rate of feature calculation nodes, the utilization rate of rule judgment nodes, queue length, stage processing latency, end-to-end latency distribution, bandwidth usage, RTT and packet loss rate, and video stream priority, to form the scheduling input status and update the candidate set and feasibility mask.

[0065] Step 4-2, Traffic Allocation and Rerouting Strategy: Based on the scheduling input state, perform entry allocation for new access flows and perform necessary rerouting for existing flows.

[0066] Specifically, for a newly accessed video stream s, in the set of candidate feature calculation nodes that satisfy M(s,c)=1, the scheduling score of each candidate feature calculation node is calculated according to the candidate scoring formula described in step S2, and the node with the highest score is selected as the target feature calculation node; for the prediction tensor output by the feature calculation node, the routing score of each candidate rule judgment node is calculated according to the routing scoring formula described in step S2, and the node with the highest score is selected as the target rule judgment node.

[0067] For high-priority video streams, the scoring weight corresponding to the service priority is increased to prioritize low-congestion, low-link-cost paths. For ordinary video streams, allocation is based on candidate scoring results to achieve multi-board load balancing and suppress local hotspots. When node queue pressure continues to rise, link quality deteriorates, or stage processing latency exceeds a preset threshold, scheduling scores and routing scores are recalculated, and existing streams are rebounded or rerouted.

[0068] Step 4-3, Overload Protection and Rate Limiting / Degradation: When the utilization rate of a certain feature computing node exceeds the threshold or the queue continues to back up, overload protection is triggered, causing the node to stop receiving new streams and allocate new streams to other candidate nodes; rate limiting, frame reduction, or resolution reduction are performed on low-priority streams, while maintaining the complete link for high-priority streams, thereby avoiding the spread of overload and causing end-to-end long-tail drag.

[0069] Step 4-4, Migration Rollback and Stability Assurance: When network metrics deteriorate or node health status changes, the scheduler triggers migration or rollback strategies to reduce cross-node collaboration intensity or switch to a more stable intra-machine path; during the migration process, flow identifiers and events are kept idempotent to ensure that results are not lost or duplicated, and the migration effect is evaluated through the observation window, and if necessary, it is rolled back to the previous stable configuration.

[0070] Steps 4-5: Scheduling effect evaluation and adaptive update: Evaluate the end-to-end latency, queue growth rate, resource utilization and alarm success rate after the scheduling policy takes effect; if the key indicators improve, solidify the policy and update the threshold parameters; if the indicators deteriorate, reduce the migration frequency or roll back the policy to form a continuously adaptive scheduling closed loop.

[0071] Step 5: Edge-to-cloud communication mechanism and result-level reporting: A layered communication mechanism is established between the edge and cloud. Service discovery and efficient invocation are used to achieve component collaboration within the edge cluster. Structured detection results and alarm events are reported from the edge to the cloud, thereby reducing bandwidth consumption and privacy leakage risks. A reliable delivery mechanism ensures that results can be reached even under weak network conditions.

[0072] Step 5-1, Edge Access and Control Signaling: The edge sends video data to the edge through the video access protocol and sends control signaling and device status through the message protocol. The edge maintains the connection status and supports reconnection and disconnection detection to ensure that multi-source access is manageable.

[0073] Step 5-2, Internal Collaborative Communication within the Edge Cluster: The edge cluster uses a "cluster service discovery and DNS resolution + gRPC call" approach to complete the phased calls between perception services and decision services. The prediction tensor is used as the cross-node transmission object, enabling the collaborative inference link of "detection head post-segmentation" and "feature calculation - feature transmission - rule judgment" to operate stably.

[0074] Step 5-3, Result-level reporting: Transmit structured detection results and alarm events from the edge to the cloud. The structured results should include at least the detection box, category, confidence level, timestamp, flow identifier, and rule hit information to reduce bandwidth costs and privacy risks.

[0075] Step 5-4, Local Storage and Reliable Delivery: When the cloud is unreachable or the link is interrupted for a short time, the structured results are temporarily stored locally and the delivery status is recorded; after the link is restored, the results are retransmitted in chronological order; combined with acknowledgment retransmission and idempotent deduplication, the results are eventually reachable and no duplicate alarms are triggered.

[0076] This embodiment provides a cloud-edge-device collaborative inference method for edge scenarios characterized by significant network jitter, uneven computing power distribution, and concurrent multi-channel video. It uses model segmentation-driven componentized services as the engineering carrier, transforming the end-to-end inference link from a monolithic deployment to a programmable, staged pipeline. A cluster-level multi-board scheduling closed loop ensures efficient utilization of heterogeneous resources and stable end-to-end service quality. This method abstracts the inference process into a collaborative paradigm of "feature calculation—feature transmission—rule judgment." In one implementation, a segmentation strategy using the YOLO Head is employed, making the cross-node transmission object a lightweight prediction tensor. Result-level reporting is used in edge-cloud communication, uploading only structured detection results and alarm events, not video data. This reduces bandwidth consumption and mitigates the amplifying effect of weak network jitter on tail latency, offering advantages in scalability and feasibility.

[0077] This embodiment provides a cloud-edge-device collaborative inference method to improve latency stability and system carrying capacity in complex edge environments. Through the collaborative design of staged task abstraction and multi-board parallel pipeline, the computationally intensive stage continuously bears the main computational load on the feature computing node side, while the logic-intensive stage stably executes post-processing and alarm rules on the rule judgment node side, avoiding weak computing power nodes from becoming queuing bottlenecks under multi-path concurrency conditions. At the same time, by using stage-level queue level, stage processing latency, and end-to-end latency distribution as observation signals, it can suppress queuing jitter and end-to-end long-tail drag under time-varying load conditions, reducing the operation and maintenance costs caused by frequent manual intervention.

[0078] This embodiment provides a cloud-edge-device collaborative inference method. Through a deployable structure of "YOLO Head followed by model segmentation + componentized service encapsulation," the inference process of a single model is mapped to independently deployable, scalable, and can-scale upgradeable stage components. This solves the problem of fine-grained scheduling and fault isolation in multi-board clusters when using monolithic deployments, thus achieving deployment-friendly results and controllable evolution costs. Model segmentation transforms cross-node transmission objects from video frames or high-dimensional features into prediction tensors, while componentized encapsulation further solidifies input / output formats and timeout, retry, and idempotent semantics, ensuring that collaborative benefits remain predictable and repeatable even in weak network environments.

[0079] This embodiment provides a cloud-edge-device collaborative inference method that achieves heterogeneous resource matching and cluster-level load balancing through a joint strategy of "stage-resource mapping + multi-board overall scheduling". The scheduler generates allocation and migration decisions based on core criteria such as accelerated computing unit utilization, general processor utilization, stage queue length, stage and end-to-end latency quantiles, bandwidth usage, RTT and packet loss rate, and video stream priority. This ensures that high-performance computing resources prioritize heavy computation stages such as feature calculations, while low-performance resources are avoided from handling stages that could amplify queuing, thereby improving overall effective utilization and reducing P95 / P99 tail latency risks.

[0080] This embodiment provides a cloud-edge-device collaborative inference method that introduces feasibility masks and overload protection at the policy front end to ensure action execution and control system risks. By generating feasibility masks for candidate pairs of "video stream - feature computing node", candidate entries with resource out-of-bounds, queue overload, or substandard link quality are blocked in advance. An overload protection strategy is set at the system layer. When the utilization rate or queue pressure of a feature computing node exceeds a threshold, its new stream access is reduced or stopped and a fallback is triggered, so that the overload does not spread and the congestion can be converged, thereby reducing the additional latency, bandwidth, and energy consumption caused by rollback, migration, and retries.

[0081] This embodiment provides a cloud-edge-device collaborative inference method that, through a communication design of "result-level reporting + reliable delivery," enables the edge side to stably output service results even under weak network conditions. Only structured detection results and alarm events are transmitted from the edge to the cloud, avoiding bandwidth saturation and latency jitter caused by video transmission. Simultaneously, combined with local storage, acknowledgment retransmission, and idempotent deduplication mechanisms, it ensures that results are eventually delivered and alarms are not duplicated under conditions of short-term link loss or packet loss, reducing the risk of missed and false alarms caused by link fluctuations.

[0082] This embodiment provides a cloud-edge-device collaborative inference method that employs an engineering architecture that separates the control plane and data plane to enhance system maintainability and evolvability. The data plane is responsible for video access, stage inference, and result generation, while the control plane is responsible for state aggregation, policy decision-making, and configuration distribution. The two are linked through a unified data contract and observability system, enabling the system to complete component scaling, path switching, and policy updates without service interruption. It also supports the reuse of the same framework in multiple scenarios, allowing for rapid migration and deployment by replacing models or rules, demonstrating the reusability value of a general framework.

[0083] This embodiment provides a cloud-edge-device collaborative inference method with horizontal scalability for the growth of multi-channel video. By combining the expansion of perception service replicas on the feature calculation node side with the expansion of decision service replicas on the rule judgment node side, and with the help of gateway-side routing forwarding and global allocation by the scheduler, it can maintain scalable operation when the number of task flows increases, and maintain stable service quality under mixed loads with different input resolutions and priorities. It is suitable for edge deployment scenarios that require long-term stable operation, such as security production monitoring.

[0084] Example 2 One embodiment of the present invention provides a cloud-edge-device collaborative inference system for heterogeneous edge clusters, comprising: The acquisition module is configured to: acquire node resources, network status, and video stream information of the heterogeneous edge cluster in real time; use the multiple video streams to be processed as pipeline input objects; divide the nodes in the cluster into functional categories; and clarify the role positioning of feature calculation nodes and rule judgment nodes. The routing module is configured to dynamically select the optimal feature calculation node for each video stream based on the real-time node resources and network status of the heterogeneous edge cluster and video stream information, and complete the video stream feature extraction task. After the feature calculation is completed, the module determines the load and location information of the node according to the current cluster rules, and adaptively routes the video stream features to the most suitable rule judgment node, forming a continuous pipeline operation from video stream inflow to feature calculation and then to inference decision. The reporting module is configured as follows: the rule judgment node performs inference tasks based on the features of the received video stream, generates structured detection results and alarm events, and efficiently reports the result-level data to the cloud, completing the closed-loop scheduling and execution of the entire cloud-edge-device collaborative inference pipeline.

[0085] Example 3 One embodiment of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the cloud-edge-device collaborative inference method for heterogeneous edge clusters.

[0086] Example 4 In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided for storing computer instructions. When the computer instructions are executed by a processor, they implement the cloud-edge-device collaborative inference method for heterogeneous edge clusters.

[0087] Example 5 One embodiment of the present invention provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the cloud-edge-device collaborative inference method for heterogeneous edge clusters.

[0088] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0090] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A cloud-edge-end collaborative inference method for a heterogeneous edge cluster, characterized in that, Using multiple video streams as the scheduling granularity, and focusing on building a dynamically adaptive cloud-edge-device collaborative inference pipeline, this approach achieves three-layer collaborative mapping and pipeline orchestration of video streams, feature computation nodes, and rule judgment nodes along the inference task path by real-time sensing of node resources and network status in heterogeneous edge clusters. Specifically, this includes: The system acquires node resources, network status, and video stream information of heterogeneous edge clusters in real time, uses multiple video streams to be processed as pipeline inputs, divides the nodes in the cluster into functional categories, and clarifies the roles of feature calculation nodes and rule judgment nodes. Based on the real-time node resources and network status of the heterogeneous edge cluster, as well as video stream information, the optimal feature calculation node is dynamically selected for each video stream to complete the video stream feature extraction task. After the feature calculation is completed, the load and location information of the node are judged according to the current cluster rules, and the video stream features are adaptively routed to the most suitable rule judgment node, forming a continuous pipeline operation from video stream to feature calculation and then to inference decision. The rule-based judgment node performs inference tasks based on the features of the received video stream, generates structured detection results and alarm events, and efficiently reports the result-level data to the cloud, completing the closed-loop scheduling and execution of the entire cloud-edge-device collaborative inference pipeline.

2. The cloud-edge-cooperative inference method for heterogeneous edge cluster of claim 1, wherein, The node resource status of the heterogeneous edge cluster includes at least the utilization rate of the accelerated computing unit of the feature computing node, the utilization rate of the general processor of the rule judgment node, and the queue length of each node. The network link status includes bandwidth, latency, and packet loss rate; The video stream information includes stream identifier, resolution, frame rate, bit rate and service priority, and end-to-end latency budget.

3. The cloud-edge-cooperative inference method for heterogeneous edge cluster of claim 1, wherein, The real-time node resources and network status of the heterogeneous edge cluster, as well as video stream information, are represented by a collaborative state representation that includes video stream objects, feature computing node objects, and rule judgment node objects. A binary feasibility mask for video stream-feature computing nodes is generated. The mask is dynamically generated based on the available computing power threshold, queue length threshold, link round-trip delay, and packet loss rate constraints of the feature computing nodes. It is used to shield infeasible allocation actions at the pipeline entry point.

4. The cloud-edge-cooperative inference method for heterogeneous edge cluster of claim 1, wherein, The feature computation node employs an accelerated computation unit to speed up the recomputation of convolution, feature extraction, and prediction output. The rule judgment node uses a general-purpose processor to handle lightweight rule judgment and alarm logic.

5. The cloud-edge-device collaborative reasoning method for heterogeneous edge clusters as described in claim 1, characterized in that, When dynamically selecting the optimal feature calculation node for a video stream, based on the binary feasibility mask and cooperative state representation, the utilization rate of the accelerated computing unit of the feature calculation node, the degree of queue backlog, and the network cost at the source of the video stream are considered to generate candidate scores and select the feature calculation node with the best score, thereby realizing a one-to-one mapping and dynamic rebinding between each video stream and the feature calculation node.

6. The cloud-edge-cooperative inference method for heterogeneous edge cluster of claim 1, wherein, The adaptive routing of video stream features to the most suitable rule-judgment node includes: from a set of candidate rule-judgment nodes that meet the constraints of remaining general processing capacity, queue length, and link quality, comprehensively considering the remaining processing capacity of the candidate rule-judgment nodes, queue waiting time, link cost from the feature calculation node to the candidate rule-judgment node, and real-time requirements of the service, selecting the rule-judgment node with the best score as the target routing node; the transmission of the video stream features follows a predefined data contract, and the timeout and retry strategy is adjusted or a fallback path is triggered when latency increases or packet loss increases.

7. A cloud-edge-end collaborative inference system for heterogeneous edge clusters, characterized in that, The cloud-edge-device collaborative inference method for heterogeneous edge clusters as described in any one of claims 1-6 includes: The acquisition module is configured to: acquire node resources, network status, and video stream information of the heterogeneous edge cluster in real time; use the multiple video streams to be processed as pipeline input objects; divide the nodes in the cluster into functional categories; and clarify the role positioning of feature calculation nodes and rule judgment nodes. The routing module is configured to dynamically select the optimal feature calculation node for each video stream based on the real-time node resources and network status of the heterogeneous edge cluster and video stream information, and complete the video stream feature extraction task. After the feature calculation is completed, the module determines the load and location information of the node according to the current cluster rules, and adaptively routes the video stream features to the most suitable rule judgment node, forming a continuous pipeline operation from video stream inflow to feature calculation and then to inference decision. The reporting module is configured as follows: the rule judgment node performs inference tasks based on the features of the received video stream, generates structured detection results and alarm events, and efficiently reports the result-level data to the cloud, completing the closed-loop scheduling and execution of the entire cloud-edge-device collaborative inference pipeline.

8. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the cloud-edge-device collaborative reasoning method for heterogeneous edge clusters as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium, comprising: The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement a cloud-edge-device collaborative inference method for heterogeneous edge clusters as described in any one of claims 1-6.

10. An electronic device, comprising: include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement a cloud-edge-device collaborative inference method for heterogeneous edge clusters as described in any one of claims 1-6.

Images