Complexity and semantic perception video analysis method and system based on edge cloud collaboration

Abstract

The invention discloses a complexity and semantic perception video analysis method and system based on edge cloud collaboration. According to the method, video preprocessing is optimized, continuous similar video frames are efficiently processed, video semantic features and system workload changes are considered, and an effective scheduling strategy is achieved. According to the method, a difficulty estimator and an inter-frame difference detector are constructed, the processing difficulty and redundancy degree of a task are predicted according to semantic features of a video task, a state is generated in combination with current system measurement (queue length and bandwidth), and an optimal decision is made through a decision engine based on a PPO algorithm, and the optimal task resolution and the final processing node are selected. And after reasoning, the processing reward of the current task is calculated, and the DRL model is updated according to the reward to continuously adapt to the changes of video contents and system loads, so that the overall performance is improved to the greatest extent, and the reasoning quality, the processing speed and the resource utilization rate are balanced.

Description

Technical Field

[0001] The present invention relates to the fields of edge computing, cloud computing, deep reinforcement learning and computer vision technology, and in particular to a complexity and semantic perception video analysis method and system based on edge cloud collaboration. Background Art

[0002] The application of deep neural networks (DNN) in computer vision has significantly improved the accuracy of reasoning. With the popularity of smartphones and the widespread deployment of surveillance cameras, video data has grown exponentially, which has brought major challenges to traditional cloud computing video analysis methods. The surge in video content has led to a new level of demand for efficient processing and analysis technologies. Although cloud computing can provide abundant computing resources, it has expensive bandwidth costs and large inference delays due to the need to transmit large amounts of video data. In contrast, edge computing can provide low-latency processing, but it faces the problem of limited computing power. This binary division between cloud computing and edge computing poses complex challenges for optimizing video analysis systems.

[0003] In order to reduce the bandwidth cost and data transmission latency of video analysis applications, we can consider using edge-cloud collaborative video analysis methods. According to the different computing capabilities of edge-cloud nodes and cloud nodes, deep learning models with different processing capabilities are deployed. Edge-cloud collaboration can achieve the collaborative cooperation of edge and cloud according to the offloading strategy algorithm used to achieve the purpose of improving reasoning accuracy and reducing processing latency. However, for some cases, especially the analysis of surveillance videos, many challenges are raised, which greatly affect the efficiency of video analysis systems. These challenges come from the content redundancy of surveillance videos and the different levels of reasoning complexity across different video frames. Therefore, it is necessary to consider solving the above problems to avoid waste of resources and achieve more accurate scheduling of resources to improve the overall performance of video analysis systems.

[0004] Considering the above factors, several key issues are raised for the inference analysis of real-world traffic surveillance videos. First, different video frames have different inference difficulties, and some frames can produce good inference results without complex inference. Second, there is often a lot of content redundancy in traffic surveillance videos, where consecutive frames often contain similar or identical information. Solving these factors can optimize the inefficient resource utilization and suboptimal scheduling strategies of current video analysis systems. Summary of the invention

[0005] Purpose of the invention: In view of the differences in edge-cloud node processing capabilities in the above-mentioned real deployment scenarios, the time-varying redundancy and complexity of surveillance videos, and the real-time changes and imbalances in workloads, the present invention provides a complexity and semantic-aware video analysis method and system based on edge-cloud collaboration.

[0006] Technical solution: A complexity and semantics-aware video analysis method based on edge-cloud collaboration, which is applied to an edge-cloud collaborative video analysis system composed of edge computing nodes and cloud computing nodes, and uses edge-cloud collaboration to implement complexity and semantics-aware video analysis, including implementing effective scheduling strategies based on video semantic features and system workload changes, and using deep reinforcement learning to achieve real-time learning of optimal strategies, achieving a balance between reasoning quality, processing speed, and resource utilization;

[0007] The specific implementation process of the method is as follows:

[0008] (1) The system receives the video frame f captured by the camera at time t t The difficulty estimator and inter-frame difference detector deployed on the edge node extract the semantic features of the video frame and predict the video frame reasoning difficulty o t and the inter-frame difference v t,t′ , and the number of similar frames in the most recent processing task a t ;

[0009] (2) Obtain the real-time load of the system, including the length of the local waiting queue l t , Cloud waiting queue length c t and the available edge cloud bandwidth b t ;

[0010] (3) The decision engine makes decisions based on the obtained task status and system status, and then applies decision actions to video frames, including the selection of inference nodes, resolution preprocessing of video frames, and discarding of repeated similar frames;

[0011] (4) Depending on the decision action, there are three ways to process the video frame:

[0012] When a video frame is considered easy to process or the system load is too large, the video frame is processed at the local edge node;

[0013] Video frames are considered similar and repeated frames and are discarded to avoid wasting system resources by repeated operations;

[0014] Video frames are considered difficult to process, and are offloaded to cloud nodes for processing after resolution preprocessing at the edge nodes based on the inference difficulty and system load;

[0015] (5) After the system processes the video task according to the decision action, it calculates the reasoning accuracy, processing time and processing cost, and calculates the processing reward R(t) of the current task according to the task reward function;

[0016] (6) The system collects processing rewards for a set of video tasks and trains the decision engine according to its learning strategy, so that the decision engine can adapt to the current video frame and system in real time.

[0017] Furthermore, the specific construction method of the difficulty estimator, inter-frame difference detector and some indicators in the reward function deployed in the edge node in step (1) is as follows:

[0018] (11) Difficulty Estimator It combines a pre-trained DNN for semantic feature extraction and an MLP for difficulty prediction. To train the difficulty estimator We compare the edge model (ε) and the cloud model The inference results are used to build a dataset:

[0019]

[0020] where o′ t is the true value of the inference difficulty of the training data, and ф is the comparison function that quantifies the difference between the inference results of the edge model and the cloud model.

[0021] For the video frame f captured at time slot t t , its inference difficulty is estimated to be:

[0022]

[0023] (12) Inter-frame difference detector: It uses low-level image features to identify similarities between consecutive video frames. The detector consists of a low-level feature extractor. Sum and Difference Comparator The frame f t and frame f t′ The inter-frame difference is calculated as:

[0024]

[0025] (13) Bandwidth cost: The system will incur bandwidth costs when transmitting video data to the cloud. The system can dynamically adjust the resolution to reduce bandwidth costs by combining the characteristics of the video task and the system status. The system will select a resolution for each frame. (including 480P, 720P, 1080P). Its cloud transmission frame t The bandwidth cost C b (f t )for:

[0026] C b (f t )=Bz v

[0027] where zv is the frame size at resolution v, and B is the transmission cost per unit data.

[0028] (14) Computational cost: For cloud nodes, the cost of renting cloud servers usually needs to be considered. t The computational cost C c (f t )for:

[0029]

[0030] Where U is the computing cost per unit time of the cloud node, d v is the inference time of a video frame with resolution v.

[0031] (15) Reasoning accuracy: The reasoning accuracy is used to calculate the reward of the corresponding task. We calculate the reasoning accuracy of the task by comparing the processing results of the edge model and the cloud model. Specifically, for frame f t , let R e (f t ) and R c (f t ) are frame f t The recognition results of the edge model and cloud model are given by , and its inference accuracy is calculated as:

[0032]

[0033] Where |·| represents the number of detected objects, and ∩ represents the intersection of correctly detected objects. The cloud model inference result is considered as the true value because the cloud model has higher complexity and better performance. The ratio of objects that match the edge model detection results with the cloud model results is measured as the inference accuracy.

[0034] Furthermore, the deep reinforcement learning algorithm for constructing the decision engine agent in step (3) of the method is specifically implemented as follows:

[0035] (31) System state: The system state S(t) at time slot t contains the semantic features of the video task and the measurement information of the system.

[0036]

[0037] Among them t To predict the inference complexity, is the normalized frame difference, a t is the number of similar frames in recent history, and is the normalized length of local and cloud task queues, b tThe state representation captures the key features of the video content and system operating conditions, enabling the system to adapt to different content complexities and dynamically changing system conditions, and can solve the problems of similar redundant tasks and differences in recognition difficulty.

[0038] (32) Action selection: Based on the collected state information, it is input into the Actor network in the decision engine to obtain the probability distribution of the action. Action set It includes: local reasoning, processing video tasks at the edge node; cloud reasoning: processing at the cloud node, and the frame transmission resolution will be selected according to the decision: 480P, 720P and 1080P; discard processing: if the decision engine identifies it as similar or repeated, skip the current frame.

[0039] (33) Reward function: After completing the inference task at time slot t, the system will calculate the processing reward R(t) for the current task. We define a comprehensive reward function that aims to maximize system performance while considering multiple key factors. For a video frame at time slot t, the reward R(t) is calculated as follows:

[0040] R(t)=α(t)-ω1*d(t)-ω2*(C b (f t )+C c (f t ))

[0041] Where α(t) is the inference accuracy, d(t) is the total delay, which includes the node inference time, the waiting queue time and the bandwidth transmission time, C b (f t ) is the bandwidth transmission cost, C c (f t ) is the task reasoning cost, ω1 and ω2 are penalty weights that weigh the importance of latency and cost. This reward function encapsulates the trade-off between reasoning accuracy, total latency, and resource utilization cost. The behavior of the system can be controlled by adjusting the weights ω1 and ω2 to consider the different priorities of the above three performances.

[0042] (34) After the system completes the reasoning task in time slot t, it will t The system status and task f t Actions and tasks t Rewards and tasks t The action probability distribution of is stored into the experience buffer as a one-time conversion process.

[0043] (35) The system continuously receives new reasoning tasks and repeats steps (1) to (4). After collecting data that meets the requirements of a set of training in the experience buffer, it uses this set of data to train the Critic network in the decision engine and optimizes the policy target of the Actor network and the loss target of the Critic network through the Adam optimizer.

[0044] (36) The system adopts a DRL method based on the proximal policy optimization (PPO) algorithm and consists of two main components: the Actor network and the Critic network. The Actor network is responsible for processing the current state S(t) and completing the decision selection of the policy engine. It receives the state vector S(t) as input, processes it through multiple fully connected layers using the ReLU activation function, and produces a probability distribution over the action space using the Softmax activation function at the output layer. The output corresponds to the probability of selecting each action: local execution, cloud execution (resolution options are 480P, 720P, 1080P), and frame loss. On the other hand, the Critic network estimates the value function and predicts the expected cumulative reward for the current state. It shares a similar structure with the Actor network, receives the same state vector S(t) as input, processes it through a hidden layer with ReLU activation, and outputs a single value representing the estimated state value. The two networks are trained simultaneously using the PPO algorithm, with the Actor network used to maximize the expected return and the Critic network used to minimize the difference between the predicted and actual returns.

[0045] The present invention also provides a complexity and semantics-aware video analysis system based on edge-cloud collaboration, which is used to implement the complexity and semantics-aware surveillance video analysis method in the above-mentioned edge-cloud continuum.

[0046] The system includes:

[0047] Video component: used for video acquisition;

[0048] Difficulty Estimator: predicts the inference difficulty of each video frame based on semantic features, uses a DNN network to extract semantic features, and uses an MLP network as a regressor to estimate the inference complexity, which is used to optimize task inference latency and improve system resource utilization;

[0049] Inter-frame difference detector: Analyzes the underlying image features and calculates the differences between consecutive frames. It consists of a low-level feature extractor and a difference comparator. By identifying similar or repeated frames, it reduces redundant processing and reduces the waste of system resources.

[0050] Decision engine: used to decide how to process the video tasks arriving at the edge node. The decision selection includes the selection of edge cloud nodes, the selection of video frame resolution, and the judgment and processing of repeated frames. The decision engine is also updated in real time through reinforcement learning to dynamically learn the optimal strategy suitable for the current environment.

[0051] Preprocessing module: Based on the decision of the decision engine, a resolution reduction strategy is adopted to reduce the frame size, thereby minimizing the transmission delay and the inference time when offloading to the cloud node;

[0052] Edge model: Edge nodes are responsible for making decisions and reasoning about tasks that are offloaded to local nodes.

[0053] Cloud Model: Used for high-quality reasoning on difficult tasks.

[0054] Beneficial effects: Compared with the prior art, the substantial progress and significant effects of the present invention are as follows:

[0055] (1) The present invention proposes a complexity and semantics-aware surveillance video analysis method in the edge-cloud continuum. For each task, the system selects a strategy based on the semantic characteristics of the task and the state characteristics of the system to maximize performance. In addition, the system can perform autonomous learning based on previous strategy selection and result analysis, and update the strategy scheduling algorithm in real time.

[0056] (2) The present invention can predict the repetitiveness and complexity of video frames by extracting the semantic features of video tasks. Thus, in a real deployment environment, the same quality of reasoning tasks can be achieved with minimal computing resources, and the real-time performance of the system can be improved, while reducing the reasoning cost of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a video analysis system framework based on edge-cloud collaboration;

[0058] Figure 2 It is the decision-making and update framework of the Actor network and Critic network of deep reinforcement learning in the system;

[0059] Figure 3 It is a comparison of different indicators between the present invention and the baseline method;

[0060] Figure 4 It is a comparison of the indicators of the present invention and the baseline methods with different penalty weights. DETAILED DESCRIPTION

[0061] In order to explain the technical solution disclosed in the present invention in detail, further elaboration is made below in conjunction with the accompanying drawings and specific embodiments of the specification.

[0062] The present invention provides a complexity and semantics-aware video analysis method based on edge-cloud collaboration. The method can optimize video preprocessing, efficiently process consecutive similar video frames, and consider video semantic features and system workload changes to implement effective scheduling strategies. The method can dynamically balance the trade-off between accuracy and efficiency and adapt to changing video content and environmental conditions.

[0063] Combination Figure 1 The video analysis system block diagram shows the complexity and semantic awareness of edge-cloud collaboration. The video analysis system consists of the following video analysis components:

[0064] (1) Difficulty Estimator: This component predicts the inference difficulty of each video frame based on semantic features. It uses a DNN network to extract semantic features and an MLP network as a regressor to estimate the inference complexity. This optimizes task inference latency and improves system resource utilization.

[0065] (2) Inter-frame difference detector: This component analyzes the underlying image features to calculate the differences between consecutive frames. It consists of a low-level feature extractor and a difference comparator. By identifying similar or repeated frames, the system can reduce redundant processing and reduce the waste of system resources.

[0066] (3) Preprocessing module: Based on the decision of the decision engine, a resolution reduction strategy is adopted to reduce the frame size, thereby minimizing the transmission delay and the inference time when offloading to the cloud node.

[0067] (4) Decision engine: The decision engine deployed on the edge node decides how to handle the video task arriving at the edge node. The decision selection includes the selection of edge cloud nodes, the selection of video frame resolution, and the judgment and processing of duplicate frames. The decision engine can be updated in real time through reinforcement learning and dynamically learn the optimal strategy suitable for the current environment.

[0068] (5) Edge model: Edge nodes are responsible for making decisions and reasoning about tasks that are offloaded to local nodes. Due to the limited computing power of edge nodes, they use lightweight models with lower accuracy to optimize performance.

[0069] (6) Cloud Model: Cloud nodes run more complex and accurate models for high-quality reasoning on difficult tasks.

[0070] Regarding the model construction in component (1), the present invention constructs a difference set between the video frame and the edge-cloud reasoning results to form a training database, completes the training of component (1), and thus can estimate the reasoning complexity of the video frame.

[0071] For the decision engine in component (4), a deep reinforcement learning method based on the proximal policy optimization (PPO) algorithm is used to learn the optimal strategy. The network structure of the Actor network and the Critic network is as follows: Figure 2 shown.

[0072] The Actor network is responsible for mapping the task state information and system state information captured by the system into a set of decision actions. It receives the state vector S(t) as input, which includes the inference complexity predicted by component (1), the inter-frame difference calculated by component (2), the number of similar frames in the recent history, the current local node waiting queue length, the current cloud node waiting queue length, and the bandwidth between the edge node and the cloud node. The inference complexity, inter-frame difference, and the number of similar frames in the recent history are task state information, which reflect the indicators of the current task that the system is concerned about and serve as influencing factors for the decision engine. The current local node waiting queue length, the current cloud node waiting queue length, and the bandwidth between the edge node and the cloud node reflect the system load and session status. The Actor network receives the above state information and ultimately outputs the logarithmic probability distribution of the decision action. The decision action is divided into five discrete actions. The system will select the final decision action from the output logarithmic probability distribution according to a specific strategy. The discrete actions include selecting the original resolution for inference at the edge node, judging the video frame as a duplicate frame for discarding, selecting the minimum resolution of 480P to transmit to the cloud node for inference, selecting the resolution of 720P to transmit to the cloud node for processing, and selecting the original resolution of 1080P to transmit to the cloud node for processing. The role of the Critic network is to estimate the value of the decision action and calculate the difference between the current reward and the expected reward. It receives the same task state information and system state information, processes it through a hidden layer with ReLU activation, and outputs a single predicted value representing the estimated state value to minimize the difference between the predicted and actual rewards.

[0073] During the training process, a reward experience pool will be built for deep reinforcement learning, which stores current and past processing results, rewards, and action selections. When the experience pool reaches the threshold size for method update, it will trigger the system's decision engine algorithm update. This ensures that the decision engine can always learn the most recent and optimal scheduling strategy, thereby achieving the best system performance.

[0074] For the DNN models in components (5) and (6), the present invention deploys a small model (fasterrcnn_mobilenet_v3_large_fpn) in the edge node and a large model (fasterrcnn_resnet50_fpn) in the cloud node.

[0075] The present invention is compared with several baseline methods:

[0076] (1) RDE: The reasoning difficulty estimator of the edge node is set to be unavailable, and the decision engine no longer selects strategies based on the reasoning difficulty.

[0077] (2) RDD: The inter-frame difference detector of the edge node is set to be unavailable, and the decision engine no longer selects strategies based on the inter-frame difference.

[0078] (3) Edge-DRL: The decision engine no longer assigns tasks to cloud nodes for reasoning, but only considers edge reasoning and discarding.

[0079] (4) Cloud-DRL: The decision engine no longer assigns tasks to edge nodes for reasoning, but only considers cloud reasoning. The lowest resolution is selected.

[0080] (5) RDrop: The decision engine does not make discard decisions and only considers edge reasoning and cloud reasoning.

[0081] (6) SQueue: Considering the most violent greedy processing method to make task decisions, the inference request incoming at time slot t is directed to be offloaded to the edge node or cloud node with the shortest waiting queue. The lowest resolution is selected.

[0082] like Figure 3 As shown, the differences in overall performance, inference accuracy, inference cost and total inference delay between the present invention and different baseline methods under the same weight are compared.

[0083] In terms of overall performance, the present invention can achieve better performance compared with other baseline methods. RDE and RDD remove the difficulty estimator and inter-frame difference detector from the edge node, which leads to reduced resource utilization efficiency and performance degradation due to the lack of semantic and low-level image features for determining reasoning difficulty and frame similarity. Compared with the present invention, RDrop has lower performance because it cannot process similar or repeated frames, resulting in excessive system load and performance degradation. Although Cloud-DRL achieves similar accuracy as the present invention, the lack of real-time, low-cost edge-cloud reasoning actions leads to increased total latency and total cost. Edge-DRL minimizes total latency and cost by using only edge reasoning, but its accuracy is lower due to the computational limitations of edge nodes. Finally, the traditional greedy method SQueue performs poorly because it does not integrate system status, task status, and past experience to optimize task offloading.

[0084] exist Figure 3 (b) Figure 3 (c) and Figure 3In (d), three key indicators are compared: average inference accuracy, average total inference latency, and average total inference cost. The present invention achieves the highest average inference accuracy under the default weights. In addition, compared with all other methods except Edge-DRL, the present invention ensures lower average total inference latency and cost. The above performance comparison confirms that the present invention can select the optimal scheduling strategy and processing decision by comprehensively considering the real-time state of the system and the semantic characteristics of the task, and accurately and effectively processes repeated video frames.

[0085] like Figure 4 As shown in the figure, three control groups are designed according to different penalty weights: high inference quality, optimal, and low inference cost, and the inference accuracy, total latency, total cost, and overall performance of the baseline methods are compared. The high-quality group achieves the highest inference accuracy, but incurs greater total inference latency and cost, which is the price of higher accuracy. In contrast, the low-cost group provides real-time performance and the lowest processing cost, but delegates more tasks to edge nodes or discards tasks as duplicate frames, resulting in reduced accuracy.

Claims

1. A complexity and semantics-aware video analysis method based on edge-cloud collaboration, characterized in that: The method is applied to an edge-cloud collaborative video analysis system composed of edge computing nodes and cloud computing nodes, and uses edge-cloud collaboration to achieve complexity and semantic-aware video analysis, including implementing effective scheduling strategies based on video semantic features and system workload changes, using deep reinforcement learning to achieve real-time learning of optimal strategies, and achieving a balance between reasoning quality, processing speed, and resource utilization; The specific implementation process of the method is as follows: (1) The system receives the video frame f captured by the camera at time t t The difficulty estimator and inter-frame difference detector deployed on the edge node extract the semantic features of the video frame and predict the video frame reasoning difficulty o t and the inter-frame difference v t,t′ , and the number of similar frames in the most recent processing task a t ; (2) Obtain the real-time load of the system, including the length of the local waiting queue l t , length of the cloud waiting queue c t and available edge cloud bandwidth b t ; (3) The decision engine makes decisions based on the obtained task status and system status, and then applies decision actions to video frames, including the selection of inference nodes, resolution preprocessing of video frames, and discarding of repeated similar frames; (4) According to the decision action, there are three ways to process the video frame: When a video frame is considered easy to process or the system load is too large, the video frame is processed at the local edge node; Video frames are considered similar and repeated frames and are discarded to avoid wasting system resources by repeated operations; Video frames are considered difficult to process, and are offloaded to cloud nodes for processing after resolution preprocessing at the edge nodes based on the inference difficulty and system load; (5) After the system processes the video task according to the decision action, it calculates the reasoning accuracy, processing time and processing cost, and calculates the processing reward R(t) of the current task according to the task reward function; (6) The system collects processing rewards for a set of video tasks and trains the decision engine according to its learning strategy, so that the decision engine can adapt to the current video frame and system in real time.

2. The complexity and semantics-aware video analysis method based on edge-cloud collaboration according to claim 1, characterized in that: The specific construction method of the difficulty estimator, inter-frame difference detector and some indicators in the reward function deployed on the edge node in step (1) is as follows: (11) Difficulty Estimator It combines a pre-trained DNN for semantic feature extraction and an MLP for difficulty prediction; By comparing the edge model ε and the cloud model The inference results are used to build a dataset: where o′ t is the true value of the inference difficulty of the training data, ф is the comparison function that quantifies the difference between the inference results of the edge model and the cloud model; For the video frame f captured at time slot t t , its inference difficulty is estimated to be: (12) Inter-frame difference detector: It uses low-level image features to identify similarities between consecutive video frames; The detector consists of a low-level feature extractor Sum and Difference Comparator The frame f t and frame f t′ The inter-frame difference is calculated as: (13) Bandwidth cost: The system generates bandwidth cost when transmitting video data to the cloud. The resolution can be dynamically adjusted to reduce bandwidth cost based on the characteristics of the video task and the system status. The system will select a resolution for each frame Its cloud transmission frame f t The bandwidth cost C b (f t )for: C b (f t )=Bz v where z v is the frame size at resolution v, B is the transmission cost per unit data; (14) Computational cost: For cloud nodes, the cost of renting cloud servers needs to be considered. t The computational cost C c (f t )for: Where U is the computing cost per unit time of the cloud node, d v is the inference time of a video frame with resolution v; (15) Reasoning accuracy: The reasoning accuracy is used to calculate the reward of the corresponding task. The reasoning accuracy of the task is calculated by comparing the processing results of the edge model and the cloud model; For frame f t , let R e (f t ) and R c (f t ) are frame f t The recognition results of the edge model and cloud model are given by , and its inference accuracy is calculated as: Where |·| represents the number of objects detected, and ∩ represents the intersection of correctly detected objects; The cloud model inference result is considered as the true value because the cloud model has higher complexity and better performance. The ratio of objects whose edge model detection results match the cloud model results is measured as the inference accuracy.

3. The complexity and semantics-aware video analysis method based on edge-cloud collaboration according to claim 1, characterized in that: The specific implementation of the deep reinforcement learning algorithm for building the decision engine agent in step (3) is as follows: (31) System state: The system state S(t) at time slot t contains the semantic features of the video task and the measurement information of the system. Among them t To predict the inference complexity, is the normalized frame difference, a t is the number of similar frames in recent history, and is the normalized length of local and cloud task queues, b t The state representation captures the key features of the video content and system operating conditions, enabling the system to adapt to different content complexities and dynamically changing system conditions, and to solve problems of similar redundant tasks and differences in recognition difficulty. (32) Action selection: Based on the collected state information, it is input into the Actor network in the decision engine to obtain the probability distribution of the action; the action set A includes: Local inference, processing video tasks at edge nodes; Cloud inference: Processing is performed on cloud nodes, and the frame transmission resolution is selected based on the decision; Discard processing: If the decision engine identifies it as similar or repeated, skip the current frame; (33) Reward function: After completing the reasoning task at time slot t, the system will calculate the processing reward R(t) of the current task by defining a comprehensive reward function, which aims to maximize the system performance while considering multiple key factors; For a video frame at time slot t, the reward R(t) is calculated as follows: R(t)=α(t)-ω1*d(t)-ω2*(C b (f t )+C c (f t )) Where α(t) is the inference accuracy, d(t) is the total delay, which includes the node inference time, the waiting queue time and the bandwidth transmission time, C b (f t ) is the bandwidth transmission cost, C c (f t ) is the task reasoning cost, ω1 and ω2 are penalty weights that weigh the importance of delay and cost respectively; This reward function encapsulates the trade-offs between inference accuracy, total latency, and resource utilization costs, and can control the behavior of the system to take into account the different priorities of the above three performances by adjusting the weights ω1 and ω2; (34) After the system completes the reasoning task in time slot t, it will t The system status and task f t Actions and tasks t Rewards and tasks t The action probability distribution is stored in the experience buffer as a conversion process; (35) The system continuously receives new reasoning tasks and repeats steps (1) to (4). After collecting data that meets the requirements of a set of training in the experience buffer, the set of data is used to train the Critic network in the decision engine, and the Adam optimizer is used to optimize the policy target of the Actor network and the loss target of the Critic network; (36) The system adopts the DRL method based on the proximal policy optimization PPO algorithm, which consists of two components: the Actor network and the Critic network; The Actor network is responsible for processing the current state S(t) and completing the decision selection of the policy engine. It receives the state vector S(t) as input, processes it through multiple fully connected layers using the ReLU activation function, and uses the Softmax activation function at the output layer to produce a probability distribution over the action space; this output corresponds to the probability of selecting each action: local execution, cloud execution, and frame drop; The Critic network estimates the value function and predicts the expected cumulative reward for the current state. It shares a similar structure with the Actor network, receiving the same state vector S(t) as input, processing it through a hidden layer with ReLU activation, and outputting a single value representing the estimated state value; The two networks are trained simultaneously using the PPO algorithm. The Actor network is used to maximize the expected return, while the Critic network is used to minimize the difference between the predicted and actual returns.

4. A complexity and semantics-aware video analysis system based on edge-cloud collaboration, characterized in that: The system is used to execute the complexity and semantic-aware video analysis method based on edge-cloud collaboration as described in any one of claims 1-3.

5. The complexity and semantics-aware video analysis system based on edge-cloud collaboration according to claim 4, characterized in that: The system includes: Video component: used for video acquisition; Difficulty Estimator: predicts the inference difficulty of each video frame based on semantic features, uses a DNN network to extract semantic features, and uses an MLP network as a regressor to estimate the inference complexity, which is used to optimize task inference latency and improve system resource utilization; Inter-frame difference detector: Analyzes the underlying image features and calculates the differences between consecutive frames. It consists of a low-level feature extractor and a difference comparator. By identifying similar or repeated frames, it reduces redundant processing and reduces the waste of system resources. Decision engine: used to decide how to process the video tasks arriving at the edge node. The decision selection includes the selection of edge cloud nodes, the selection of video frame resolution, and the judgment and processing of repeated frames. The decision engine is also updated in real time through reinforcement learning to dynamically learn the optimal strategy suitable for the current environment. Preprocessing module: Based on the decision of the decision engine, a resolution reduction strategy is adopted to reduce the frame size, thereby minimizing the transmission delay and the inference time when offloading to the cloud node; Edge model: Edge nodes are responsible for making decisions and reasoning about tasks that are offloaded to local nodes. Cloud Model: Used for high-quality reasoning on difficult tasks.

Images