Complexity and Semantic Aware Video Analysis Methods and Systems Based on Edge Cloud Collaboration

Abstract

This invention discloses a complexity- and semantically aware video analysis method and system based on edge cloud collaboration. The method optimizes video preprocessing, efficiently processes consecutive similar video frames, and considers video semantic features and system workload changes to achieve an effective scheduling strategy. By constructing a difficulty estimator and an inter-frame difference detector, the method predicts the processing difficulty and redundancy of the task based on its semantic features. It then generates a state by combining current system metrics (queue length, bandwidth), and makes optimal decisions using a decision engine based on the PPO algorithm, selecting the optimal task resolution and the last processing node. After inference, the processing reward for the current task is calculated, and the DRL model is updated based on the reward, continuously adapting to changes in video content and system load to maximize overall performance and balance inference quality, processing speed, and resource utilization.

Description

Technical Field

[0001] This invention relates to the fields of edge computing, cloud computing, deep reinforcement learning, and computer vision, specifically to a complexity and semantic awareness video analysis method and system based on edge cloud collaboration. Background Technology

[0002] The application of deep neural networks (DNNs) in computer vision has significantly improved the accuracy of inference. With the widespread adoption of smartphones and the extensive deployment of surveillance cameras, video data is growing exponentially, posing a significant challenge to traditional cloud-based video analytics methods. The surge in video content has led to a new level of demand for efficient processing and analysis technologies. While cloud computing offers abundant computing resources, the need to transmit large amounts of video data results in expensive bandwidth costs and significant inference latency. In contrast, edge computing, while providing low-latency processing, faces the problem of limited computing power. This binary division between cloud computing and edge computing presents complex challenges for optimizing video analytics systems.

[0003] To reduce bandwidth costs and data transmission latency in video analytics applications, we can consider using an edge-cloud collaborative video analytics approach. Deep learning models with varying processing capabilities are deployed based on the different computing power of edge and cloud nodes. Edge-cloud collaboration can achieve cooperative cooperation between the edge and cloud based on the offloading strategy algorithm used, thereby improving inference accuracy and reducing processing latency. However, for certain situations, especially the analysis of surveillance video, many challenges arise that significantly impact the efficiency of video analytics systems. These challenges stem from the redundancy of surveillance video content and the varying levels of inference complexity across different video frames. Therefore, it is necessary to address these issues to avoid resource waste and achieve more precise resource scheduling, thereby improving the overall performance of the video analytics system.

[0004] Considering the above factors, several key issues arise regarding the inference analysis of real-world traffic surveillance videos. First, the inference difficulty varies across different video frames; some frames can produce satisfactory results without complex reasoning. Second, traffic surveillance recordings often contain significant content redundancy, with consecutive frames typically containing similar or identical information. Addressing these factors can optimize the current video analysis system's inefficient resource utilization and suboptimal scheduling strategies. Summary of the Invention

[0005] Purpose of the invention: In view of the differences in processing capabilities of edge cloud nodes, the time-varying redundancy and complexity of monitoring videos, and the real-time changes and imbalances in workload in real deployment scenarios, this invention provides a complexity- and semantically-aware video analysis method and system based on edge cloud collaboration.

[0006] Technical Solution: A complexity and semantic awareness video analysis method based on edge cloud collaboration. The method is applied to an edge-cloud collaborative video analysis system composed of edge computing nodes and cloud computing nodes. It utilizes edge cloud collaboration to achieve complexity and semantic awareness video analysis, including implementing an effective scheduling strategy based on video semantic features and system workload changes, using deep reinforcement learning to learn the optimal strategy in real time, and achieving a balance between inference quality, processing speed and resource utilization.

[0007] The specific implementation process of the method is as follows:

[0008] (1) The system receives the video frame f captured by the camera at time t. t The difficulty estimator and inter-frame difference detector deployed on edge nodes extract semantic features of video frames to predict the inference difficulty of video frames. t and inter-frame difference v t,t′ And the number of similar frames a in the most recently processed task. t ;

[0009] (2) Obtain the real-time load information of the system, including the local waiting queue length l t Cloud waiting queue length c t and available edge cloud bandwidth b t ;

[0010] (3) The decision engine makes decisions based on the obtained task status and system status, and then applies the decision actions to the video frames, including the selection of inference nodes, resolution preprocessing of video frames, and discarding of duplicate and similar frames.

[0011] (4) Based on the decision-making action, there are three ways to process video frames:

[0012] Video frames are processed at local edge nodes when they are considered easy to process or when the system load is too high.

[0013] Video frames are considered similar and duplicate frames and are discarded to avoid wasting system resources on repetitive operations.

[0014] Video frames are considered to be difficult to process. Based on the difficulty of inference and system load, resolution preprocessing is performed at the edge nodes and then the processing is offloaded to the cloud nodes.

[0015] (5) After the system processes the video task based on the decision action, it calculates the inference accuracy, processing time and processing cost, and calculates the processing reward R(t) of the current task based on the task reward function.

[0016] (6) The system collects a set of video task processing rewards and trains the decision engine according to the learning strategy of the reinforcement learning decision engine so that the decision engine can adapt to the current video frame and the system in real time.

[0017] Furthermore, the specific construction methods for some metrics in the difficulty estimator, inter-frame difference detector, and reward function deployed on the edge nodes in step (1) are as follows:

[0018] (11) Difficulty estimator It combines a pre-trained DNN for semantic feature extraction and an MLP for difficulty prediction. To train the difficulty estimator... We compare the edge model (ε) and the cloud model. We will use the inference results to construct a dataset:

[0019]

[0020] Where o′ t is the true value of the inference difficulty of the training data, and ф is a comparison function that quantifies the difference between the inference results of the edge model and the cloud model.

[0021] For video frame f captured in time slot t t Its reasoning difficulty is estimated to be:

[0022]

[0023] (12) Inter-frame difference detector: It uses low-level image features to identify the similarity between consecutive video frames. The detector consists of a low-level feature extractor. Sum and difference comparators It consists of two parts, where frame f t and frame f t′ The inter-frame difference is calculated as follows:

[0024]

[0025] (13) Bandwidth cost: The system incurs bandwidth costs when transmitting video data to the cloud. These costs can be reduced by dynamically adjusting the resolution based on video task characteristics and system status. The system will select a resolution for each frame. (Including 480P, 720P, 1080P). Its cloud transmission frame f t Bandwidth cost C b (f t )for:

[0026] C b (f t )=Bz v

[0027] Where zv Where v is the frame size at resolution v, and B is the transmission cost per unit of data.

[0028] (14) Computational Costs: For cloud nodes, the rental cost of cloud servers typically needs to be considered. This includes the processing of frames f on cloud nodes. t The computational cost C c (f t )for:

[0029]

[0030] Where U is the computing cost per unit time for a cloud node, and d v Let v be the inference time for a video frame with resolution v.

[0031] (15) Inference Accuracy: Inference accuracy is used to calculate the reward for the corresponding task. We calculate the inference accuracy of the task by comparing the processing results of the edge model and the cloud model. Specifically, for frame f t Let R e (f t ) and R c (f t ) are frames f t The inference accuracy of the edge model and cloud model recognition results is calculated as follows:

[0032]

[0033] Where |·| represents the number of detected objects, and ∩ represents the intersection of correctly detected objects. The cloud model inference result is considered as the ground truth because the cloud model has higher complexity and better performance. The inference accuracy is measured by the proportion of objects that match the edge model detection results with the cloud model results.

[0034] Furthermore, the specific implementation of the deep reinforcement learning algorithm for constructing the decision engine agent in step (3) of this method is as follows:

[0035] (31) System state: The system state S(t) at time slot t contains the semantic features of the video task and the system's metric information.

[0036]

[0037] Among them o t To predict reasoning complexity, To normalize the inter-frame difference, a t This represents the number of similar frames in recent history. and To normalize the lengths of local and cloud task queues, b tThis represents available edge cloud bandwidth. The state representation captures key features of video content and system operating conditions, enabling the system to adapt to varying content complexity and dynamically changing system conditions. It can address issues related to similar redundant tasks and differences in recognition difficulty.

[0038] (32) Action Selection: Based on the collected state information, it is input into the Actor network in the decision engine to obtain the probability distribution of actions. Action Set This includes: local inference, which processes video tasks at edge nodes; cloud inference, which processes data at cloud nodes and selects the transmission resolution of frames based on decisions: 480P, 720P, and 1080P; and discarding, which skips the current frame if the decision engine identifies it as similar or duplicate.

[0039] (33) Reward Function: After completing the inference task in time slot t, the system will calculate the processing reward R(t) for the current task. We define a comprehensive reward function to maximize system performance while considering multiple key factors. For a video frame in time slot t, the reward R(t) is calculated as follows:

[0040] R(t)=α(t)-ω1*d(t)-ω2*(C b (f t )+C c (f t ))

[0041] Where α(t) is the inference accuracy, d(t) is the total latency, which includes node inference time, waiting queue time, and bandwidth transmission time, and C b (f t C represents the bandwidth transmission cost. c (f t Let ω1 and ω2 be the penalty weights for balancing latency and cost, respectively. This reward function encapsulates the trade-offs between inference accuracy, total latency, and resource utilization cost. By adjusting the weights ω1 and ω2, the behavior of the system can be controlled to consider different priorities of these three performance parameters.

[0042] (34) After processing the inference task in time slot t, the system will transfer task f t System status, task f t Actions, tasks t Rewards and tasks t The probability distribution of actions is stored in the experience buffer as a transformation process.

[0043] (35) The system continuously receives new inference tasks and repeats steps (1) to (4). After collecting data that meets the training requirements in the experience buffer, the system uses the data set to train the Critic network in the decision engine and optimizes the policy objective of the Actor network and the loss objective of the Critic network through the Adam optimizer.

[0044] (36) The system employs a DRL method based on the Proximal Policy Optimization (PPO) algorithm, consisting of two main components: an Actor network and a Critic network. The Actor network is responsible for processing the current state S(t) and making the policy engine's decision selection. It receives the state vector S(t) as input, processes it through multiple fully connected layers using the ReLU activation function, and generates a probability distribution in the action space using the Softmax activation function in the output layer. This output corresponds to the probability of selecting each action: local execution, cloud execution (resolution options are 480P, 720P, 1080P), and frame dropping. On the other hand, the Critic network estimates the value function, predicting the expected cumulative reward of the current state. It shares a similar structure with the Actor network, receiving the same state vector S(t) as input, processing it through hidden layers with ReLU activation, and outputting a single value representing the estimated state value. Both networks are trained simultaneously using the PPO algorithm; the Actor network is used to maximize the expected reward, while the Critic network is used to minimize the difference between the predicted and actual rewards.

[0045] The present invention also provides a complexity and semantic awareness video analysis system based on edge cloud collaboration, which is used to implement the above-mentioned complexity and semantic awareness monitoring video analysis method in the edge cloud continuum.

[0046] The system includes:

[0047] Video component: Used for video capture;

[0048] Difficulty estimator: Predicts the inference difficulty of each video frame based on semantic features. It uses a DNN network to extract semantic features and an MLP network as a regressor to estimate inference complexity, thereby optimizing task inference latency and improving the utilization of system resources.

[0049] Inter-frame difference detector: Analyzes low-level image features and calculates the differences between consecutive frames. It consists of a low-level feature extractor and a difference comparator. By identifying similar or repeated frames, it reduces redundant processing and wastes system resources.

[0050] Decision engine: Used to decide how to process video tasks arriving at edge nodes. Decision choices include the selection of edge cloud nodes, the selection of video frame resolution, and the judgment and processing of duplicate frames. The decision engine is also updated in real time through reinforcement learning and dynamically learns the optimal strategy suitable for the current environment.

[0051] Preprocessing module: Based on the decision engine's decision, a resolution reduction strategy is adopted to reduce the frame size, thereby minimizing transmission latency and inference time when offloading to cloud nodes;

[0052] Edge model: edge nodes are responsible for decision-making and reasoning for tasks that are offloaded to local processing.

[0053] Cloud model: Used for high-quality reasoning on challenging tasks.

[0054] Beneficial effects: Compared with the prior art, the substantial progress and significant effects of the present invention are as follows:

[0055] (1) This invention proposes a complexity- and semantic-aware video analysis method for edge cloud continuums. For each task, the system selects a strategy based on the semantic features of the task and the state features of the system to maximize performance. Moreover, it can learn autonomously based on past strategy selections and result analysis, and update the strategy scheduling algorithm in real time.

[0056] (2) This invention can predict the repetition and complexity of video frames by extracting semantic features of video tasks. Thus, in a real deployment environment, it can achieve the same quality inference tasks with minimal computing resources, improve the real-time performance of the system, and reduce the inference cost of the system. Attached Figure Description

[0057] Figure 1 It is a video analytics system framework based on edge-cloud collaboration;

[0058] Figure 2 It is the decision-making and update framework for the Actor and Critic networks in the system's deep reinforcement learning.

[0059] Figure 3 This is a comparison of different metrics between the present invention and the baseline method;

[0060] Figure 4 This is a comparison of the indicators of the present invention with baseline methods with different penalty weights. Detailed Implementation

[0061] To illustrate the technical solutions disclosed in this invention in detail, further explanation will be provided below with reference to the accompanying drawings and specific embodiments.

[0062] This invention provides a complexity- and semantically aware video analysis method based on edge cloud collaboration. This method optimizes video preprocessing, efficiently processes consecutive similar video frames, and considers video semantic features and system workload variations to achieve effective scheduling strategies. This method can dynamically balance the trade-off between accuracy and efficiency, adapting to constantly changing video content and environmental conditions.

[0063] Combination Figure 1 The video analytics system block diagram, based on the complexity of edge cloud collaboration and semantic awareness, shows that the video analytics system consists of the following video analytics components:

[0064] (1) Difficulty Estimator: This component predicts the inference difficulty of each video frame based on semantic features. It uses a DNN network to extract semantic features and an MLP network as a regressor to estimate the inference complexity. This optimizes task inference latency and improves the utilization of system resources.

[0065] (2) Inter-frame difference detector: This component analyzes low-level image features to calculate the differences between consecutive frames. It consists of a low-level feature extractor and a difference comparator. By identifying similar or repeating frames, the system can reduce redundant processing and waste system resources.

[0066] (3) Preprocessing module: Based on the decision of the decision engine, a resolution reduction strategy is adopted to reduce the frame size, thereby minimizing transmission delay and inference time when offloading to cloud nodes.

[0067] (4) Decision Engine: The decision engine deployed at the edge nodes determines how to process video tasks arriving at the edge nodes. Decision choices include the selection of edge cloud nodes, the selection of video frame resolution, and the handling of duplicate frames. The decision engine can be updated in real time through reinforcement learning, dynamically learning the optimal strategy suitable for the current environment.

[0068] (5) Edge Model: Edge nodes are responsible for decision-making and reasoning for tasks offloaded to the local machine. Due to the computational limitations of edge nodes, they utilize lightweight models with lower precision to optimize performance.

[0069] (6) Cloud Model: Cloud nodes run more complex and accurate models for high-quality inference on challenging tasks.

[0070] For model building in component (1), this invention constructs a set of differences between video frames and edge cloud inference results to form a training database, thereby completing the training of component (1) and estimating the inference complexity of video frames.

[0071] For the decision engine in component (4), a deep reinforcement learning method based on the proximal policy optimization (PPO) algorithm is used to learn the optimal policy. The network structures of the Actor network and the Critic network are as follows: Figure 2 As shown.

[0072] The Actor network is responsible for mapping the task state information and system state information captured by the system to a set of decision actions. It receives a state vector S(t) as input, which includes the inference complexity predicted by component (1), the inter-frame difference calculated by component (2), the number of similar frames in the recent history, the current local node waiting queue length, the current cloud node waiting queue length, and the bandwidth between the edge node and the cloud node. The inference complexity, inter-frame difference, and the number of similar frames in the recent history represent task state information, reflecting the metrics of the current task that the system is concerned with, and serving as influencing factors for the decision engine's decisions. The current local node waiting queue length, the current cloud node waiting queue length, and the bandwidth between the edge node and the cloud node reflect the system's load and session status. The Actor network receives the above state information and ultimately outputs the logarithmic probability distribution of the decision actions. The decision-making actions are divided into five discrete actions. The system selects the final decision action from the logarithmic probability distribution of the output according to a specific strategy. These discrete actions include selecting the original resolution for inference at the edge nodes, discarding video frames that are duplicates, selecting the minimum resolution of 480P for transmission to the cloud node for inference, selecting a resolution of 720P for transmission to the cloud node for processing, and selecting the original resolution of 1080P for transmission to the cloud node for processing. The Critic network is used to estimate the value of the decision action, calculating the difference between the current reward and the expected reward. It receives the same task state information and system state information, processes them through hidden layers with ReLU activation, and outputs a single predicted value representing the estimated state value to minimize the difference between the predicted and actual rewards.

[0073] During training, a reward experience pool is built for deep reinforcement learning, storing current and past processing results, rewards, and action selections. When the experience pool reaches a threshold size for method updates, the system's decision engine algorithm is triggered to update. This ensures that the decision engine can always learn the most recent optimal scheduling strategy, thereby achieving the best system performance.

[0074] For the DNN models in components (5) and (6), the present invention deploys a small model (fasterrcnn_mobilenet_v3_large_fpn) in the edge nodes and a large model (fasterrcnn_resnet50_fpn) in the cloud nodes.

[0075] This invention was compared with several baseline methods:

[0076] (1) RDE: Set the inference difficulty estimator of the edge node to unavailable, and the decision engine will no longer select a strategy based on the inference difficulty.

[0077] (2)RDD: Set the inter-frame difference detector of the edge node to unavailable, and the decision engine will no longer select a strategy based on the inter-frame difference.

[0078] (3) Edge-DRL: The decision engine no longer assigns tasks to cloud nodes for inference, but only considers edge inference and discarding.

[0079] (4) Cloud-DRL: The decision engine no longer assigns tasks to edge nodes for inference, but only considers cloud inference. Select the lowest resolution.

[0080] (5) RDrop: The decision engine does not make drop-off decisions, but only considers edge reasoning and cloud reasoning.

[0081] (6) SQueue: Consider the most brute-force greedy approach to make task decisions. Inference requests passed in during time slot t are directed to the edge node or cloud node with the shortest waiting queue. Choose the lowest resolution.

[0082] like Figure 3 As shown, the differences in overall performance, inference accuracy, inference cost, and total inference latency between the present invention and different baseline methods under the same weights are compared.

[0083] In terms of overall performance, this invention achieves better performance compared to other baseline methods. RDE and RDD remove the difficulty estimator and inter-frame difference detector from edge nodes, which leads to reduced resource utilization efficiency and performance degradation due to the lack of semantic and low-level image features for determining inference difficulty and frame similarity. RDrop performs worse than this invention because it cannot handle similar or repeated frames, resulting in excessive system load and performance degradation. While Cloud-DRL achieves similar accuracy to this invention, it increases total latency and total cost due to the lack of real-time, low-cost edge-cloud inference actions. Edge-DRL minimizes total latency and cost by using only edge inference, but its accuracy is lower due to the computational limitations of edge nodes. Finally, the traditional greedy method SQueue performs poorly because it does not integrate system state, task state, and past experience to optimize task offloading.

[0084] exist Figure 3 (b) Figure 3 (c) and Figure 3In section (d), three key metrics were compared: average inference accuracy, average total inference latency, and average total inference cost. This invention achieved the highest average inference accuracy under default weights. Furthermore, compared to all other methods except Edge-DRL, this invention ensured lower average total inference latency and cost. The above performance comparison confirms that this invention, by comprehensively considering the real-time state of the system and the semantic characteristics of the task, can select the optimal scheduling strategy and processing decisions, and accurately and effectively process duplicate video frames.

[0085] like Figure 4 As shown, three control groups were designed based on different penalty weights: high inference quality, optimal, and low inference cost. The inference accuracy, total latency, total cost, and overall performance of the baseline methods were compared. The high-quality group achieved the highest inference accuracy but incurred greater total inference latency and cost, which is the price of higher accuracy. Conversely, the low-cost group provided real-time performance and the lowest processing cost, but delegated more tasks to edge nodes or discarded tasks as duplicate frames, resulting in reduced accuracy.

Claims

1. A complexity- and semantically-aware video analysis method based on edge cloud collaboration, characterized in that, The method is applied to an edge-cloud collaborative video analysis system composed of edge computing nodes and cloud computing nodes. It utilizes edge-cloud collaboration to achieve complex and semantically perceptive video analysis, including implementing effective scheduling strategies based on video semantic features and system workload changes, using deep reinforcement learning to learn the optimal strategy in real time, and achieving a balance between inference quality, processing speed and resource utilization. The specific implementation process of the method is as follows: (1) The system receives the video frame f captured by the camera at time t. t The difficulty estimator and inter-frame difference detector deployed on edge nodes extract semantic features of video frames to predict the inference difficulty of video frames. t and inter-frame difference v t,t' And the number of similar frames a in the most recently processed task. t ; (2) Obtain the real-time load information of the system, including the local waiting queue length l t Cloud waiting queue length c t and available edge cloud bandwidth b t ; (3) The decision engine makes decisions based on the obtained task status and system status, and then applies the decision actions to the video frames, including the selection of inference nodes, resolution preprocessing of video frames, and discarding of duplicate and similar frames. (4) Based on the decision-making action, there are three ways to process video frames: When video frames are considered easy to process or when the system load is too high, the video frames are processed at the local edge node. Video frames are considered similar and duplicate frames and are discarded to avoid wasting system resources on repetitive operations. Video frames are considered to be difficult to process. Based on the difficulty of inference and system load, resolution preprocessing is performed at the edge nodes and then the processing is offloaded to the cloud nodes. (5) After the system processes the video task based on the decision action, it calculates the inference accuracy, processing time and processing cost, and calculates the processing reward R(t) of the current task based on the task reward function. (6) The system collects a set of video task processing rewards and trains the decision engine according to the learning strategy of the reinforcement learning decision engine so that the decision engine can adapt to the current video frame and the system in real time.

2. The complexity and semantic awareness video analysis method based on edge cloud collaboration as described in claim 1, characterized in that, The specific construction methods for some metrics in the difficulty estimator, inter-frame difference detector, and reward function deployed on the edge nodes in step (1) are as follows: (11) Difficulty estimator It combines a pre-trained DNN for semantic feature extraction and an MLP for difficulty prediction; to train the difficulty estimator By comparing the edge model ε and the cloud model We will use the inference results to construct a dataset: o' t is the true value of the inference difficulty of the training data, and ф is a comparison function that quantifies the difference between the inference results of the edge model and the cloud model; For video frame f captured in time slot t t Its reasoning difficulty is estimated to be: (12) Inter-frame difference detector: It uses low-level image features to identify the similarity between consecutive video frames; The detector consists of a low-level feature extractor Sum and difference comparators It consists of two parts, where frame f t and frame f t' The inter-frame difference is calculated as follows: (13) Bandwidth cost: The system will incur bandwidth cost when transmitting video data to the cloud. By combining the characteristics of video tasks and system status, the resolution can be dynamically adjusted to reduce bandwidth cost. The system will select a resolution for each frame. Its cloud transmission frame f t Bandwidth cost C b (f t )for: C b (f t )=Bz v Where z v Where v is the frame size at resolution v, and B is the transmission cost per unit of data. (14) Computational cost: For cloud nodes, the rental cost of cloud servers needs to be considered when processing frames f on the cloud node. t The computational cost C c (f t )for: Where U is the computing cost per unit time for a cloud node, and d v The inference time for a video frame with resolution v; (15) Inference accuracy: Inference accuracy is used to calculate the reward for the corresponding task. The inference accuracy of the task is calculated by comparing the processing results of the edge model and the cloud model. For frame f t Let R e (f t ) and R c (f t ) are frames f t The inference accuracy of the edge model and cloud model recognition results is calculated as follows: Where |·| represents the number of detected objects, and ∩ represents the intersection of correctly detected objects; We consider the cloud model inference results as the truth value because the cloud model has higher complexity and better performance. We measure the proportion of objects that match the edge model detection results with the cloud model results as the inference accuracy.

3. The complexity and semantic awareness video analysis method based on edge cloud collaboration as described in claim 1, characterized in that, The specific implementation of the deep reinforcement learning algorithm for constructing the decision engine agent in step (3) is as follows: (31) System state: The system state S(t) at time slot t contains the semantic features of the video task and the system's metric information. Among them o t To predict reasoning complexity, To normalize the inter-frame difference, a t This represents the number of similar frames in recent history. and To normalize the lengths of local and cloud task queues, b t The available edge cloud bandwidth; the state representation captures key features of video content and system operating conditions, enabling the system to adapt to different content complexities and dynamically changing system conditions, and to solve problems of similar redundant tasks and differences in recognition difficulty; (32) Action selection: Based on the collected state information, it is input into the Actor network in the decision engine to obtain the probability distribution of actions; action set Includes: Local inference, processing video tasks at edge nodes; Cloud inference: Processing is performed on cloud nodes, and the transmission resolution of frames will be selected based on the decision. Discarding: If the decision engine identifies a frame as similar or duplicate, skip the current frame; (33) Reward function: After completing the inference task in time slot t, the system will calculate the processing reward R(t) of the current task. By defining a comprehensive reward function, the goal is to maximize system performance while considering multiple key factors. For a video frame in time slot t, the reward R(t) is calculated as follows: R(t)=α(t)-ω1*d(t)-ω2*(C b (f t )+C c (f t )) Where α(t) is the inference accuracy, d(t) is the total latency, which includes node inference time, waiting queue time, and bandwidth transmission time, and C b (f t C represents the bandwidth transmission cost. c (f t ) represents the task reasoning cost, and ω1 and ω2 are the penalty weights for balancing the importance of delay and cost, respectively; This reward function encapsulates the trade-offs between inference accuracy, total latency, and resource utilization cost, and can control the behavior of the system by adjusting the weights ω1 and ω2 to take into account the different priorities of the above three performances. (34) After processing the inference task in time slot t, the system will transfer task f t System status, task f t Actions, tasks t Rewards and tasks t The probability distribution of actions is stored in the experience buffer as a single transformation process; (35) The system continuously receives new inference tasks and repeats steps (1) to (4). After collecting data that meets the training requirements in the experience buffer, the system uses the data to train the Critic network in the decision engine and optimizes the policy objective of the Actor network and the loss objective of the Critic network through the Adam optimizer. (36) The system adopts the DRL method based on the near-end strategy optimization PPO algorithm, which consists of two components: the Actor network and the Critic network; The Actor network is responsible for processing the current state S(t) and making decisions for the policy engine. It receives the state vector S(t) as input, processes it through multiple fully connected layers using the ReLU activation function, and generates a probability distribution in the action space using the Softmax activation function at the output layer. This output corresponds to the probability of selecting each action: local execution, cloud execution, and frame dropping. The Critic network estimates the value function, predicting the expected cumulative reward of the current state. It shares a similar structure with the Actor network, receiving the same state vector S(t) as input, processing it through hidden layers with ReLU activation, and outputting a single value representing the estimated state value. Both networks are trained using the PPO algorithm simultaneously. The Actor network is used to maximize the expected return, while the Critic network is used to minimize the difference between the prediction and the actual return.

4. A complexity and semantic-aware video analysis system based on edge cloud collaboration, characterized in that, This system is used to perform the complexity and semantic awareness video analysis method based on edge cloud collaboration as described in any one of claims 1-3; the system includes: Video component: Used for video capture; Difficulty estimator: Predicts the inference difficulty of each video frame based on semantic features. It uses a DNN network to extract semantic features and an MLP network as a regressor to estimate inference complexity, thereby optimizing task inference latency and improving the utilization of system resources. Inter-frame difference detector: Analyzes low-level image features and calculates the differences between consecutive frames. It consists of a low-level feature extractor and a difference comparator. By identifying similar or repeated frames, it reduces redundant processing and wastes system resources. Decision engine: Used to decide how to process video tasks arriving at edge nodes. Decision choices include the selection of edge cloud nodes, the selection of video frame resolution, and the judgment and processing of duplicate frames. The decision engine is also updated in real time through reinforcement learning and dynamically learns the optimal strategy suitable for the current environment. Preprocessing module: Based on the decision engine's decision, a resolution reduction strategy is adopted to reduce the frame size, thereby minimizing transmission latency and inference time when offloading to cloud nodes; Edge model: edge nodes are responsible for decision-making and reasoning for tasks that are offloaded to local processing. Cloud model: Used for high-quality reasoning on challenging tasks.

Images