Cloud edge collaborative video real-time analysis method and device based on multi-agent reinforcement learning

Abstract

The invention discloses a cloud edge collaborative video real-time analysis method and device based on multi-agent reinforcement learning. The method comprises the following steps: 1) training and deploying a video analysis model; 2) performing system interaction through the randomly initialized RL model; 3) obtaining cloud side end state information, obtaining a local state of each side end device and a system global state through a reward function, and putting the local state and the system global state as interactive experience into an experience pool; 4) using an iterative two-stage training framework to train the RL model at a preset frequency; and 5) obtaining an optimal strategy model which has a certain generalization ability and is suitable for the current environment through multiple times of interactive training, and realizing real-time video analysis. Aiming at the problems of training deviation and instability caused by delayed feedback in a cloud edge video analysis scene, the invention provides a centralized training and distributed deployment multi-agent RL online training framework and method, so that edge equipment can automatically adapt to continuously changing conditions.

Description

Technical Field

[0001] The present invention relates to the fields of reinforcement learning, video analysis, and edge computing, and relates to an offloading decision method and device based on multi-agent reinforcement learning and a cloud-edge collaborative framework. The method aims to achieve real-time and high-precision video analysis tasks in a distributed cloud-edge environment. Background Art

[0002] Deep learning has been widely used in the field of streaming video, covering aspects such as video surveillance, behavior recognition, autonomous driving, and virtual reality. In practice, deep learning models are often deployed on various terminal devices, such as vehicles, surveillance cameras, mobile phones, and smart furniture. In some key application scenarios, such as autonomous driving, real-time and high-precision analysis of videos is essential. However, the computing power of terminal devices is usually limited, which may lead to a decline in service quality or even serious accidents. To solve this problem, a common idea is to offload video analysis tasks to devices with powerful computing power. The traditional method is to offload video tasks to cloud servers through cloud computing, but this will bring huge transmission costs. In order to reduce network transmission consumption, edge computing has been proposed to push video analysis streams to nearby edge devices. However, edge devices have limited computing power and cannot process multiple task requests in real time. Therefore, cloud-edge collaborative computing has become a popular paradigm, which fully utilizes the advantages of edge computing and cloud computing and significantly reduces the overall process latency.

[0003] Video analysis based on cloud-edge collaboration needs to select appropriate offloading strategies according to system load, network environment, etc. Many studies have explored the coordination of video analysis tasks between cloud servers and a large number of edge devices. They usually design a centralized decision maker to collect loading information from the edge network and the cloud, and then determine a suitable offloading strategy based on the characteristics of the application. For example, DeepDice, FastVA, and Osmoticgate attempt to model a complex environment for cloud-edge collaborative streaming video analysis systems and study new heuristic offloading mechanisms to balance the workload of the system. However, existing edge cloud offloading research is mainly in simple, static, and closed environments, while real-world scenarios are dynamic and uncertain. Centralized heuristic decision-making relies heavily on theoretical modeling of the environment. It cannot quickly capture the dynamic changes in the number, performance, and workload of terminal devices and edge devices, and it is difficult to accurately model the complex interactive environment between distributed devices in the edge cloud, resulting in potential system misconfiguration. In addition, its decision-making requires monitoring and collecting global information of the system. In the face of distributed systems with a large number of edge devices, the search space of centralized decision-making will be very large and the latency will be high. In addition, collecting global information will also introduce additional latency, reducing their effectiveness in responding to sudden traffic peaks. In addition, designing offloading strategies for all components is an NP-hard problem, and it is difficult to ensure its real-time requirements while ensuring the video processing accuracy of each device.

[0004] In recent years, distributed multi-agent reinforcement learning methods have shown potential for adaptive scheduling policies in dynamic cloud-edge environments. However, in cloud-edge video analysis scenarios, training RL agents to generate accurate offloading policies takes time, and there is no available dataset to support offline RL training. In addition, the unknown system environment inevitably makes offline methods impractical. Second, RL algorithms usually rely on the availability of immediate observations of states, actions, and rewards to learn policies. Unfortunately, this assumption is rarely met in edge cloud video analysis. In cloud-edge video analysis scenarios, due to the dynamic nature of the workload, feedback from the real environment (e.g., inference accuracy and latency) may be delayed due to processing queue blocking. This delay in receiving environmental feedback introduces bias in calculating the agent's reward, leading to training bias and instability, or even failure in extreme cases. Summary of the invention

[0005] In order to overcome the shortcomings of traditional centralized decision-making solutions in dynamic distributed systems, the present invention proposes a cloud-edge collaborative video real-time analysis method and device based on multi-agent reinforcement learning. The present invention can dynamically adapt to different distributed environments, realize flexible offloading decisions, balance accuracy and latency, ensure that latency remains within an acceptable range, and maximize analysis accuracy.

[0006] The present invention designs a cloud-edge collaborative video real-time analysis method based on multi-agent reinforcement learning. This method allows each edge device to be equipped with an intelligent agent to independently formulate an unloading strategy based on its own state, thereby significantly reducing the delay and complexity caused by traditional centralized strategy formulation. At the same time, the agents are centrally trained on the server through reinforcement learning, and a reasonable reward function is designed to guide the agents to learn the optimal strategy to adapt to the current environment. After a lot of training, the agents can gradually balance the trade-off between accuracy and latency and achieve collaborative work.

[0007] During the training process, a two-stage online learning framework is designed to address the training bias and instability problems caused by delayed feedback. The framework combines the on-policy and off-policy MAPPO training algorithms to effectively utilize samples through continuous correction. The first stage adopts the on-policy training method to accelerate the learning of the agent in the online environment, and introduces a distributed safety regulator to ensure the stable operation of the system in the early stage of training. In the second stage, the experience correction module is introduced, and the auxiliary off-policy experience replay training algorithm is used to mitigate the training bias and forgetting problems, ensuring that the agent can smoothly adapt to the changing workload.

[0008] The technical solution adopted by the present invention to solve its technical problem is:

[0009] The first aspect of the present invention relates to a cloud-edge collaborative video real-time analysis method based on multi-agent reinforcement learning, the method comprising the following steps:

[0010] S1 trains and deploys video analysis models;

[0011] S2 interacts with the system through a randomly initialized RL model;

[0012] S3 obtains cloud-edge status information, obtains the local status of each edge device and the global status of the system through the reward function, and puts it into the experience pool as interaction experience;

[0013] S4 Reference Figure 2 , using an iterative two-stage training framework to train the RL model at a preset frequency;

[0014] Through multiple interactive trainings, S5 obtains the optimal strategy model with certain generalization ability and suitable for the current environment, realizing real-time video analysis.

[0015] Further, the step S1 includes the following process:

[0016] Step S11, use Nvidia Jetson NX as the edge device, use RTX3090 as the cloud server, design the RTP protocol for video streaming transmission based on the TCP protocol, and install Rabbitmq and InfluxDB time series database on the cloud, and use the publish-subscribe mode of Rabbitmq to realize data transmission and storage between multiple devices.

[0017] Step S12, after the edge device obtains the video stream, it encodes it according to the configuration command and compresses it to the target bit rate and resolution to improve the inference efficiency. The encoding process uses a multi-core architecture to implement multi-process parallel processing. The video analyzed locally is converted to bgr24 format through FFmpeg and pushed into the processing queue; the video to be analyzed in the cloud is compressed into h264 format, and the resolution and bit rate are adjusted before sending it to the transmission queue. When multiple edge devices transmit videos to the cloud, because the cloud can only receive them one by one, delays, packet loss and retransmission, and even congestion problems will occur. For this purpose, a socket pool is designed, and there is a working process in the pool that is responsible for listening to connections and receiving data packets. The concurrent process continuously monitors the socket pool and reassembles the data packets. All data packets are placed in the processing queue and further processed by the cloud server.

[0018] Step S13, use the UA-DETRAC traffic video dataset (30fps, 960×540 resolution) to train the analysis model. Select 120 seconds of videos of different difficulty levels to construct a training dataset, and fine-tune the YOLOv8 (S, M, L) model pre-trained on COCO. Adopt the Adam optimization strategy, with a learning rate of 0.01 and a batch of 64 for the first 300 epochs, and adjusted to 0.001 for the next 300 epochs, and finally select the best model.

[0019] Step S14, implement the inference engine of edge devices and cloud. Encapsulate the YOLOv8 target detection code into an inference module and embed it into the system. The edge device loads multiple configuration models, and the intelligent agent selects the most suitable model for video analysis; the cloud deploys the large model with the best performance. Use TensorRT's layer fusion, precision calibration, and dynamic memory management to optimize the inference speed. Before inference, decode the received video blocks into batch frames and input the model for analysis.

[0020] Further, the step S2 includes the following process:

[0021] Step S21: deploy edge monitors on each edge, continuously monitor the transmission queue size, analyze the queue size and network bandwidth, and integrate the actions of the previous step as local status The data is uploaded to the cloud through RabbitMQ messaging middleware. The cloud collects the status of each edge device, and combined with the cloud analysis queue, informs the edge of the corresponding queue congestion, and sends security warning information to the corresponding edge device. This information includes the size of each queue of the current edge device and the size of the cloud analysis queue.

[0022] Step S22, obtaining the latest uninstallation strategy based on the current state through the locally deployed agent.

[0023] First, the edge decides whether to start the security regulator based on the information returned by the cloud. If the current edge device and cloud video analysis queue sizes are both smaller than the preset threshold, the security regulator will not be started; if either the edge or cloud video analysis queue size exceeds the preset threshold, the edge will select different levels of security policies based on the queue congestion to prevent the system from over-congestion and crashing.

[0024] Initialize a two-dimensional array of all zeros, which is called the invalid action matrix M in this invention. A The actions are discretized according to a certain granularity, and all actions within the security policy limit are set to 1, thereby converting the security policy into a series of invalid action masks.

[0025] The agent (Actor network) deployed on the edge observes the local state Generate the next action. The Actor network handles the edge devices Local state. The network is based on a backbone network and four different linear layers, generating four sets of logits for different actions. Before generating the categorical distribution, these action masks adjust the probability of unreasonable actions to a minimum (i.e., adjust the logit corresponding to unreasonable actions to a negative maximum), effectively preventing the agent from performing unexpected actions. This method of using action masks still allows the generation of valid policy gradients, allowing RL to effectively utilize the experience of switching to safe policies during training.

[0026] Categorical distributions (discrete probability distributions of different actions) are then generated based on this, and actions are sampled from these distributions and mapped to system configurations accordingly. These actions include setting the resolution of the generated video frames, determining the bitrate used to encode and decode video streams, selecting models with different inference delays and analysis accuracy, and determining the offloading ratio of the generated video frames. This action sampling method effectively improves the exploration ability of the Actor network, thereby preventing the learning process from being confined to the local optimum.

[0027] Further, the step S3 includes the following process:

[0028] Step S31, in order to realize the monitoring of system status, the present invention uses RabbitMQ as the middleware to implement the message forwarding module. And deploy the monitor in the cloud, it will start multiple threads to listen to different information channels, including the analysis results of each video block, the transmission and reasoning queue size of each edge device, the cloud video reasoning queue size, bandwidth and the strategy (action) of each edge. The analysis results of each video block on the edge device and server, including the overall delay and reasoning accuracy, will be transmitted to the cloud monitor through RabbitMQ immediately after the analysis is completed. At the same time, each edge device and server will also transmit the operating status to the cloud monitor once a second through Rabbitmq. The monitor will store the collected status information in the InfluxDB database. According to the step identifier to which the analysis result belongs, the monitor will divide it into timely feedback and delayed feedback, and store them in InfluxDB's D respectively. order and D unorder In two data tables, read D every 10 seconds order The database obtains the local state of each edge device and the global state of the system, calculates the reward value of different actions through the reward function, and puts the above information into the experience pool for RL model training.

[0029] Step S32, reward function design. In video analysis applications, optimizing for higher accuracy usually increases latency. Increasing encoding configurations (such as frame rate, resolution, or bit rate) can usually improve analysis accuracy, but this will result in higher processing latency. This creates a conflicting multi-objective optimization problem in which the system needs to balance the competing goals of maximizing accuracy and minimizing latency. Therefore, the absolute value of the difference between the delay and the preset target delay is designed to control the processing delay within an acceptable range, and the conflict is balanced by a linear combination of accuracy and this absolute value. In addition, when the action generated by the agent leads to an abnormal system state, a safety policy is triggered to restore the system to a normal state. However, this behavior should be punished, otherwise the agent may learn to trigger the safety mechanism frequently in order to obtain a potentially higher reward in the next step. Therefore, it is necessary to prevent over-reliance on the safety regulator through a segmented reward function.

[0030] Further, the step S4 includes the following process:

[0031] Step S41: Based on the above model and reward function, a two-stage online learning framework is designed to conduct centralized training on the model. A total of N episodes are trained, each episode is divided into T steps, and each step lasts 10 seconds. When the number of steps in each round reaches T, the first stage of training is started.

[0032] A multi-agent reinforcement learning based on PPO (MAPPO) is designed to train the model. For the model structure, based on the Actor-Critic network framework, a Critic network and an Actor network are designed and stored in the cloud for centralized training. The input layer of the cloud-based Critic network combines all local states and the cloud queue size state (global state), and the output layer converts the collected information into a predicted value based on the global state. Critic minimizes the error L between the predicted value and the target value of the state value function. C (V, φ) is used for updating. Based on the estimation error of the state value by the Critic, the present invention uses GAE technology to calculate the advantage function value, and uses the advantage value to guide the Actor update. The Actor uses the policy gradient method to update its parameters by maximizing the loss function L(π, θ), thereby achieving the update goal of maximizing the cumulative reward. Through the systematic training of the Actor and the Critic, the multi-agent PPO architecture learns and cooperates in an online environment, thus completing the training of an episode.

[0033] Step S42, after N iters After episodes, the training enters the second stage. First, a reward modifier is designed to correct the reward deviation and update the experience pool, which is caused by delayed feedback caused by the blockage of the analysis queue. The present invention creates two databases D order and D unorder To store and collect timely feedback and delayed feedback data in real time. unorder The database query range is the first T·N iters The delayed feedback of the step is merged into the experience pool to correct the reward deviation. Then a batch of experience B is randomly taken from the experience pool. sam Used in the off-policy auxiliary training phase. This phase will use B sam To train the backbone network in the actor network and an additional linear layer (value head) connected after the backbone network, and minimize L aux (π,θ V ), this objective function allows the actor to learn the value function (value head) from a large amount of experience while optimizing its backbone network. sam The Critic network will also be trained again. Through the second stage of off-policy training, the invention can use the corrected experience to correct the Actor and Critic networks and enhance their generalization capabilities.

[0034] Further, the step S5 includes the following process:

[0035] Step S51, repeat the training process based on steps S2, S3, and S4. After each training is completed, the cloud forwards the latest Actor network to each edge device through Rabbitmq.

[0036] In step S52, the edge device continues to interact with the environment based on the latest agent, and makes decisions based on the locally observed state through the agent deployed at the edge to maximize system performance.

[0037] The second aspect of the present invention relates to a cloud-edge collaborative video real-time analysis device based on multi-agent reinforcement learning, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the cloud-edge collaborative video real-time analysis method based on multi-agent reinforcement learning of the present invention.

[0038] The third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a cloud-edge collaborative video real-time analysis method based on multi-agent reinforcement learning of the present invention.

[0039] In response to the training deviation and instability problems caused by delayed feedback in cloud-edge video analysis scenarios, the present invention proposes a centralized training, distributed deployment multi-agent RL online training framework and method, which enables edge devices to autonomously adapt to changing conditions. A two-stage learning paradigm and an online safety supervision mechanism are integrated during training to effectively respond to rapid environmental changes. In order to mitigate training deviation, the present invention develops a new algorithm for assisting off-policy training after empirical correction, which enables the system to stably adapt to dynamic and even extreme system load environments and achieve high-precision real-time video analysis.

[0040] The advantages of the present invention are:

[0041] In response to the training bias and instability problems caused by delayed feedback in cloud-edge video analysis scenarios, the present invention proposes a centralized training and distributed deployment multi-agent RL online training framework, which enables edge devices to autonomously adapt to changing conditions. Unlike traditional RL methods, the present invention integrates a two-stage learning paradigm and an online safety supervision mechanism to effectively respond to rapid environmental changes. In order to mitigate training bias, the present invention develops a new algorithm for assisting off-policy training after empirical correction, so that the system can stably adapt to dynamic and even extreme system load environments, and achieve high-precision real-time video analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The figure is the overall workflow diagram of the method of the present invention.

[0043] Figure 2 This is a flow chart of the two-stage online learning framework for model training of the present invention.

[0044] Figure 3 It is the main flow chart of the method of the present invention.

[0045] Figure 4 Schematic diagram of the device of the present invention. DETAILED DESCRIPTION

[0046] In order to make the technical solution and design idea of ​​the present invention clearer, the present invention is further described in detail below with reference to the accompanying drawings.

[0047] Example 1

[0048] This embodiment relates to a cloud-edge collaborative video real-time analysis method based on multi-agent reinforcement learning. This method allows each edge device to be equipped with an intelligent agent to independently formulate an unloading strategy based on its own state, thereby significantly reducing the delay and complexity caused by traditional centralized strategy formulation. At the same time, through reinforcement learning, each agent is centrally trained on the server, and a reasonable reward function is designed to guide the agent to learn the optimal strategy to adapt to the current environment. After a lot of training, the agent can gradually balance the trade-off between accuracy and latency to achieve collaborative work.

[0049] Reference Figure 1 , a cloud-edge collaborative video real-time analysis method based on multi-agent reinforcement learning, comprising the following steps:

[0050] S1 trains and deploys video analysis models;

[0051] S2 interacts with the system through a randomly initialized RL model;

[0052] S3 obtains cloud-edge status information, obtains the local status of each edge device and the global status of the system through the reward function, and puts it into the experience pool as interaction experience;

[0053] S4 uses an iterative two-stage training framework to train the RL model at a preset frequency;

[0054] Through multiple interactive trainings, S5 obtains the optimal strategy model with certain generalization ability and suitable for the current environment, realizing real-time video analysis.

[0055] Further, the step S1 includes the following process:

[0056] Step S11, use Nvidia Jetson NX as the edge device, use RTX3090 as the cloud server, design the RTP protocol for video streaming transmission based on the TCP protocol, and install Rabbitmq and InfluxDB time series database on the cloud, and use the publish-subscribe mode of Rabbitmq to realize data transmission and storage between multiple devices.

[0057] Step S12, after the edge device obtains the video stream, it encodes it according to the configuration command and compresses it to the target bit rate and resolution to improve the inference efficiency. The encoding process uses a multi-core architecture to implement multi-process parallel processing. The video analyzed locally is converted to bgr24 format through FFmpeg and pushed into the processing queue; the video to be analyzed in the cloud is compressed into h264 format, and the resolution and bit rate are adjusted before sending it to the transmission queue. When multiple edge devices transmit videos to the cloud, because the cloud can only receive them one by one, delays, packet loss and retransmission, and even congestion problems will occur. For this purpose, a socket pool is designed, and there is a working process in the pool that is responsible for listening to connections and receiving data packets. The concurrent process continuously monitors the socket pool and reassembles the data packets. All data packets are placed in the processing queue and further processed by the cloud server.

[0058] Step S13, use the UA-DETRAC traffic video dataset (30fps, 960×540 resolution) to train the analysis model. Select 120 seconds of videos of different difficulty levels to construct a training dataset, and fine-tune the YOLOv8 (S, M, L) model pre-trained on COCO. Adopt the Adam optimization strategy, with a learning rate of 0.01 and a batch of 64 for the first 300 epochs, and adjusted to 0.001 for the next 300 epochs, and finally select the best model.

[0059] Step S14, implement the inference engine of edge devices and cloud. Encapsulate the YOLOv8 target detection code into an inference module and embed it into the system. The edge device loads multiple configuration models, and the intelligent agent selects the most suitable model for video analysis; the cloud deploys the large model with the best performance. Use TensorRT's layer fusion, precision calibration, and dynamic memory management to optimize the inference speed. Before inference, decode the received video blocks into batch frames and input the model for analysis.

[0060] Further, the step S2 includes the following process:

[0061] Step S21: deploy edge monitors on each edge, continuously monitor the transmission queue size, analyze the queue size and network bandwidth, and integrate the actions of the previous step as local status It is expressed as:

[0062]

[0063] The data is uploaded to the cloud through RabbitMQ messaging middleware. The cloud collects the status of each edge device, and combined with the cloud analysis queue, informs the edge of the corresponding queue congestion, and sends security warning information to the corresponding edge device. This information includes the size of each queue of the current edge device and the size of the cloud analysis queue.

[0064] Step S22, obtaining the latest uninstallation strategy based on the current state through the locally deployed agent.

[0065] First, the edge decides whether to start the security regulator based on the information returned by the cloud. If the current edge device and cloud video analysis queue sizes are both smaller than the preset threshold, the security regulator will not be started; if the edge or cloud video analysis queue size exceeds the preset threshold, the edge will select different levels of security policies based on the queue congestion to prevent the system from over-congestion and crash. Specific security mechanisms include:

[0066] (1) Edge overload (P1): When the size of the current edge device analysis queue exceeds the threshold, but the cloud analysis queue does not exceed the threshold, 80%-100% of the local video analysis tasks are offloaded to the cloud, and the video resolution and bit rate are adjusted to medium and small configurations (the specific configuration is determined by the agent deployed locally, and here it only limits its decision scope).

[0067] (2) Cloud overload (P2): When the cloud analysis queue exceeds the preset threshold and the current edge analysis queue does not exceed the preset threshold, the video analysis task offload rate is adjusted to 0%-20%, that is, most of the video analysis tasks are processed locally, and the local video analysis model, resolution, and bit rate are adjusted to medium or small configurations.

[0068] (3) System overload (P3): When the size of the current edge device and cloud video analysis queue is greater than the preset threshold, the entire system is overloaded. At this time, the local video analysis model, resolution, and bit rate are adjusted to a small configuration, and the offload rate is formulated according to the ratio of the edge and cloud analysis queues. The specific calculation formula is shown in (2):

[0069]

[0070] K is the number of edge devices, is the analysis queue size of the current edge, QS t Analyze the size of the queue for the cloud.

[0071] Initialize a two-dimensional array of all zeros, which is called the invalid action matrix M in this invention. A :

[0072]

[0073] Among them, R, B, M, and Q represent the sets of four actions respectively. The actions are discretized according to a certain granularity, and the actions within the limit of the security policy are set to 1, thereby converting the security policy into a series of invalid action masks.

[0074] The agent (Actor network) deployed on the edge observes the local state As the input of the model, the next action is generated. The network is based on a backbone network and four different linear layers, generating four sets of logits for different actions. Before generating the categorical distribution, these action masks adjust the probability of unreasonable actions to a minimum (i.e., the logit corresponding to unreasonable actions is adjusted to a negative maximum), effectively preventing the agent from performing unexpected actions. This method of using action masks still allows the generation of valid policy gradients, allowing RL to effectively utilize the experience of switching to safe policies during training.

[0075] Categorical distributions (discrete probability distributions of different actions) are then generated based on this, and actions are sampled from these distributions and mapped to system configurations accordingly. These actions include setting the resolution of the generated video frames, determining the bitrate used to encode and decode video streams, selecting models with different inference delays and analysis accuracy, and determining the offloading ratio of the generated video frames. This action sampling method effectively improves the exploration ability of the Actor network, thereby preventing the learning process from being confined to the local optimum.

[0076] Further, the step S3 includes the following process:

[0077] Step S31, in order to realize the monitoring of system status, the present invention uses RabbitMQ as the middleware to implement the message forwarding module. And deploy the monitor in the cloud, it will start multiple threads to listen to different information channels, including the analysis results of each video block, the transmission and reasoning queue size of each edge device, the cloud video reasoning queue size, bandwidth and the strategy (action) of each edge. The analysis results of each video block on the edge device and server, including the overall delay and reasoning accuracy, will be transmitted to the cloud monitor through RabbitMQ immediately after the analysis is completed. At the same time, each edge device and server will also transmit the operating status to the cloud monitor once a second through Rabbitmq. The monitor will store the collected status information in the InfluxDB database. According to the step identifier to which the analysis result belongs, the monitor will divide it into timely feedback and delayed feedback, and store them in InfluxDB's D respectively. order and D unorderIn two data tables, read D every 10 seconds order The database obtains the local state of each edge device and the global state of the system, calculates the reward value of different actions through the reward function, and puts the above information into the experience pool for RL model training.

[0078] Step S32, reward function design. In video analysis applications, optimizing for higher accuracy usually increases latency. Increasing encoding configurations (such as frame rate, resolution, or bit rate) can usually improve analysis accuracy, but this will result in higher processing latency. This creates a conflicting multi-objective optimization problem in which the system needs to balance the competing goals of maximizing accuracy and minimizing latency. The reward function is expressed as:

[0079]

[0080] Delay by Design and preset target delay The absolute value of the difference is used to control the processing delay within an acceptable range, and The conflict is balanced by a linear combination of the precision and this absolute value. In addition, when the action generated by the agent leads to an abnormal system state, the safety policy is triggered to restore the system to a normal state. However, this behavior should be punished, otherwise the agent may learn to trigger the safety mechanism frequently in order to obtain a potentially higher reward in the next step. Therefore, it is necessary to prevent over-reliance on the safety regulator through a piecewise reward function.

[0081] Further, the step S4 includes the following process:

[0082] Step S41: Based on the above model and reward function, a two-stage online learning framework is designed to conduct centralized training on the model. A total of N episodes are trained, each episode is divided into T steps, and each step lasts 10 seconds. When the number of steps in each round reaches T, the first stage of training is started.

[0083] A multi-agent reinforcement learning based on PPO (MAPPO) is designed to train the model. For the model structure, based on the Actor-Critic network framework, a Critic network and an Actor network are designed and stored in the cloud for centralized training. The input layer of the cloud-based Critic network combines all local states and the cloud queue size state (global state), and the output layer converts the collected information into a predicted value based on the global state. Critic minimizes the error L between the predicted value and the target value of the state value function. C (V,φ) is used to update, which can be expressed as:

[0084]

[0085] Among them, It is a discounted reward, which indicates the total possible future benefit under a certain state, and attenuates the future reward by a discount factor (γ).

[0086] Actor uses the policy gradient method to update its parameters by maximizing the loss function L(π,θ), expressed as:

[0087]

[0088] in, It is based on the estimated error of the Critic on the state value and is calculated by the generalized advantage estimate (GAE). This advantage value can be used to guide the Actor update. ∈ is a hyperparameter that controls the clip value to prevent the new strategy from deviating too much from the old strategy. S is the policy entropy used to increase exploration, and σ is the policy entropy coefficient. This achieves the update goal of maximizing the cumulative reward. Through the systematic training of the Actor and the Critic, the multi-agent PPO architecture learns to cooperate in an online environment, thus completing the training of an episode.

[0089] Step S42, after N iters After episodes, the training enters the second stage. First, a reward modifier is designed to correct the reward deviation and update the experience pool, which is caused by delayed feedback caused by the blockage of the analysis queue. The present invention creates two databases D order and D unorder To store and collect timely feedback and delayed feedback data in real time. unorder The database query range is the first T·N iters The delayed feedback of the step is merged into the experience pool to correct the reward deviation. Then a batch of experience B is randomly taken from the experience pool. sam Used in the off-policy auxiliary training phase. This phase will use B sam To train the backbone network in the actor network and an additional linear layer (value head) connected after the backbone network, and minimize L aux (π,θ V ):

[0090]

[0091] This objective function allows the actor to learn the value function (value head) from a large amount of experience while optimizing its backbone network. Another KLloss() function is also used to prevent the policy from deviating too much from the original policy. These two objectives are balanced by a coefficient α. B samFormula (5) is also used to train the Critic network again. Through the off-policy training in the second stage, the invention can use the corrected experience to correct the Actor and Critic networks and enhance their generalization capabilities.

[0092] Further, the step S5 includes the following process:

[0093] Step S51: After each training is completed, the cloud will forward the latest Actor network to each edge device through Rabbitmq.

[0094] In step S52, the edge device continues to interact with the environment based on the latest agent, and makes decisions based on the locally observed state through the agent deployed at the edge to maximize system performance.

[0095] Example 2

[0096] like Figure 4 This embodiment relates to a cloud-edge collaborative video real-time analysis device based on multi-agent reinforcement learning, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the cloud-edge collaborative video real-time analysis method based on multi-agent reinforcement learning of Example 1.

[0097] Example 3

[0098] This embodiment relates to a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a cloud-edge collaborative video real-time analysis method based on multi-agent reinforcement learning of Example 1 is implemented.

[0099] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art can still make other modifications or changes in different forms according to the aforementioned embodiments or drawings. It is impossible to list all implementation methods or technical solutions here, and all modifications, replacements, etc. within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cloud-edge collaborative video real-time analysis method based on multi-agent reinforcement learning, characterized in that: include: S1 trains and deploys video analysis models; S2 interacts with the system through a randomly initialized RL model; S3 obtains cloud-edge status information, obtains the local status of each edge device and the global status of the system through the reward function, and puts it into the experience pool as interaction experience; S4 uses an iterative two-stage training framework to train the RL model at a preset frequency; Through multiple interactive trainings, S5 obtains the optimal strategy model with certain generalization ability and suitable for the current environment, realizing real-time video analysis.

2. The method according to claim 1, characterized in that Step S2 includes: Each edge device of S21 obtains the local status and security warning information sent from the cloud; S22 obtains the latest unloading strategy based on the current state through the local agent.

3. The method according to claim 2, characterized in that Step S22 includes: S221 uses a heuristic algorithm to allow the security regulator at each edge to adopt different security policies according to the congestion of the edge-cloud queue, and converts the security policies into a series of invalid action masks. S222 uses the state information as the input of the edge agent model to generate logits of four different actions. S223 combines logits and action masks to minimize the probability of unreasonable actions, and then generates discrete probability distributions of different actions. S224 samples actions according to the discrete probability distribution of the different actions, and maps them to the system configuration accordingly, to obtain a strategy that can ultimately be used for system execution.

4. The method according to claim 3, characterized in that Step S221 includes: S2211 does not start the safety regulator when the sizes of the edge and cloud video analysis queues are both smaller than the preset thresholds; When the size of either the edge or cloud video analysis queue exceeds the preset threshold, S2212 selects different levels of security policies based on the queue congestion to prevent the system from crashing due to excessive congestion.

5. The method according to claim 1, characterized in that Step S4 includes: After each round of system interaction, S41 obtains the trajectory data of this round from the experience pool for one-stage training based on PPO; S42 in preset N iters After rounds of system interaction and training, the experience pool data is corrected and the second stage of training is carried out.

6. The method according to claim 5, characterized in that Step S41 includes: S411 obtains the trajectory data of this round from the experience pool and uses the Actor-Critic framework to train the policy model; S412 Critic is updated by minimizing the error between the predicted value and the target value of the state value function; S413 uses GAE technology to calculate the advantage function value based on the Critic's estimation error of the state value, and uses this value to guide the Actor update; S414 Actor uses the policy gradient method to update the policy parameters by maximizing the cumulative reward.

7. The method according to claim 5, characterized in that Step S42 includes: S421 queries a database storing delay feedback information, compares the query delay feedback with N iters The trajectory data of each round is merged, and the experience pool is corrected and updated; S422 randomly takes a batch of corrected trajectory data B from the experience pool sam Used for off-policy-based auxiliary training, so that the corrected experience can be used to correct the Actor and Critic networks and enhance their generalization capabilities; S423: This stage is based on the backbone network in the Actor network and an additional linear layer is connected for training, allowing the Actor to learn the value function from a large amount of experience while optimizing its backbone network; S424 Critic uses B sam Perform the training described in step S422 again.

8. A cloud-edge collaborative video real-time analysis device based on multi-agent reinforcement learning, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the cloud-edge collaborative video real-time analysis method based on multi-agent reinforcement learning as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a cloud-edge collaborative video real-time analysis method based on multi-agent reinforcement learning as described in any one of claims 1-7 is implemented.

Images