Distributed edge collaborative video analysis method based on multi-agent reinforcement learning

Abstract

The application discloses a kind of distributed edge collaborative video analysis methods based on multi-agent reinforcement learning, through mutual cooperation of edge nodes, jointly learn the optimal strategy of video frame pre-processing, model selection and request scheduling, to minimize the overall cost of system.The application models each edge node as an agent, which is an autonomous entity and makes distributed control decisions by observing its local state.The application uses an attention mechanism to distinguish the importance of information collected from different edge nodes, and its performance is verified by deploying a video analysis test platform with multiple edge nodes, and extensive experiments are conducted using real-world datasets and experimental settings.The experimental results show that compared with existing baseline methods, the application can significantly improve the overall reward by 33.6%-86.4%.

Description

Technical Field

[0001] This invention relates to the fields of edge computing, deep reinforcement learning, and computer vision, specifically to a distributed edge collaborative video analysis method based on multi-agent reinforcement learning. Background Technology

[0002] Video analytics has been widely applied in many computer vision-based applications, such as video surveillance, augmented reality, and autonomous driving. Currently, most state-of-the-art video analytics algorithms are implemented using deep neural networks (DNNs). While DNN-based video analytics offers higher accuracy, deploying DNN models for video analysis faces several challenges in real-world scenarios. First, DNN models used for video analytics typically consist of hundreds of layers, leading to significant inference latency. Furthermore, the large volume of video content itself can result in substantial bandwidth costs and unacceptable transmission latency when transmitting the raw video content.

[0003] To reduce bandwidth costs and data transmission latency in video analytics applications, DNN models can be deployed on edge nodes close to users. Edge nodes can receive video data from users with minimal latency. However, the processing power of a single edge node is limited. When a large number of inference requests arrive, the edge node may become overloaded, exceeding its processing capacity, which will significantly increase the inference latency of video frames. To ensure system performance, a carefully designed video analytics mechanism with edge computing is necessary. The insufficient computing power of a single edge node can easily lead to overload. Furthermore, these edge nodes are located in different geographical areas, and their workloads are time-varying and unbalanced; some edge nodes may have light workloads while others may be overloaded. Therefore, it is necessary to consider collaboration among multiple edge nodes in different locations to improve the overall performance of the video analytics system.

[0004] Considering the factors mentioned above, deploying edge systems in practice presents the following challenges. An edge node not only needs to consider its own video frame preprocessing and model selection decisions, but also the decisions of other edge nodes to maximize overall performance. Therefore, the decision-making process in a video analytics pipeline becomes more complex. Furthermore, each edge node is an autonomous entity that needs to collaborate with other edge nodes while making its own decisions for received inference requests. Therefore, a distributed decision-making mechanism is required to support the collaborative work of edge nodes. Summary of the Invention

[0005] Purpose of the invention: To address the problems of insufficient processing capacity of edge nodes, time-varying and unbalanced workloads, and how edge nodes can collaborate to learn and optimize system performance in real-world deployment scenarios, this invention provides a distributed edge collaborative video analysis method based on multi-agent reinforcement learning.

[0006] Technical Solution: A distributed edge collaborative video analysis method based on multi-agent reinforcement learning. This method is based on a distributed edge computing video analysis system, deploying DNN models on edge nodes located in different geographical locations. Each edge node receives video analysis inference requests from its corresponding region. Collaborative learning is used to minimize the overall cost of the system, including the following steps:

[0007] (1) Inference requests from different regions send video frames to the corresponding edge nodes i (i∈N) via the HTTP protocol;

[0008] (2) When edge node i (i∈N) receives an inference request, the decision engine determines the inference request based on the current system state o. i (t) Make a decision, and then control decision a i (t) is applied to video frames, and the decisions include the selection of inference nodes, the selection of DNN models, and the selection of resolution;

[0009] (3) Edge node i will reduce the video frame to the specified resolution based on the resolution decision in order to reduce the transmission delay and inference delay between edge nodes;

[0010] (4) Edge node i determines whether the inference of the video frame is performed locally or forwarded to another edge node based on the inference node decision;

[0011] (5) If the video frame of the current time slot is inferred at the local edge node, the preprocessed video frame is placed in the local inference queue and waits for the selected DNN model deployed on the local edge node to perform inference.

[0012] (6) If the local edge node is overloaded, the video frame will be assigned to another edge node. The preprocessed video frame will be placed in the scheduling queue and wait to be forwarded to another edge node for inference via gRPC.

[0013] Furthermore, in step (2), for a discrete-time system, where time is denoted as t = 0, 1, 2, ..., in each time slot t, each edge node i observes its local state in time slot t, and the expression is as follows:

[0014] o i (t)=(λ i (t),l i (t),q ij (t),bij (t))

[0015] Where λ i (t) is the average inference request arrival rate of several time slot edge nodes i before time slot t, l i (t) is the length of the local inference queue of edge node i at time slot t, q ij (t) is the length of the scheduling queue from edge node i to edge node j, b ij (t) is the bandwidth between edge node i and edge node j at time slot t;

[0016] Assuming that each edge node can only observe its local state, the global state of the environment consists of the local states observed by each edge node. The global state of the environment at time slot t is represented as:

[0017] s(t)=(o1(t),o2(t),...,o N (t))

[0018] Where N is the number of edge nodes.

[0019] Furthermore, in the method, the decision engine described in step (2) is based on a multi-agent reinforcement learning algorithm, and includes the following process:

[0020] Considering discrete-time systems, the local state o of each edge node is... i (t) is input into the Actor network μ i,θ In the process, the logarithmic probability distribution of each action category is obtained, and then the action 'a' of each edge node is obtained by sampling according to the probability distribution. i (t), the expression for the class log probability distribution of each action is as follows:

[0021] p i (t)=μ i,θ (o i (t)),a i (t)~p i (t)

[0022] Where, p i (t) represents the class log probability distribution of edge node i at time slot t, μ i,θ (o i (t) represents an Actor network with parameter θ for edge node i, whose input is (o i (t)), which is used to generate the class log probability distribution p for each action. i (t);

[0023] The system will perform video analysis on inference requests based on control actions. The actions of each edge node include deciding whether to perform local inference or forward to another edge node, selecting the DNN model for inference, and the resolution for video frame preprocessing.

[0024] The control action for edge node i at time slot t is defined as follows:

[0025] a i (t)=(e,m,v),e∈E,m∈M,v∈V

[0026] Where e is the selected edge node for inference, E is the set of edge nodes, m is the selected DNN model for inference, M is the set of available DNN models, v is the selected resolution for video frame preprocessing, and V is the set of available video resolutions.

[0027] If the selected edge node for inference is the same as the edge node that received the inference request, inference is performed locally; otherwise, the request will be forwarded to the corresponding other edge node for inference.

[0028] Furthermore, at the end of time slot t, each edge node i will calculate its reward r during time slot t. i (t) and the shared reward r(t) of the edge system during time slot t;

[0029] In the method, if an inference request is successfully completed, its reward is calculated as a linear combination of accuracy and latency; conversely, if a request is dropped, the reward for that inference request is defined as a weighted penalty value.

[0030] The reward for the j-th request made on edge node i during time period t is defined as:

[0031]

[0032] Where g i,j (t) represents the waiting time in the queue for the j-th request at edge node i during time slot t, where T is the video frame dropping threshold, and α i,j (t) represents the j-th request made on edge node i during time slot t, d i,j (t) is the total delay of the j-th request made on edge node i during time slot t, where F is the penalty constant and ω is the penalty weight for the overall delay;

[0033] The reward r of edge node i during time slot t i (t) represents the sum of rewards obtained by edge node i for inference during time slot t, expressed as:

[0034]

[0035] Among them, P i (t) is the number of inference requests made by edge node i during time slot t;

[0036] To optimize the overall performance of collaborative work among edge nodes, this method designs the reward function as a shared reward, which is the sum of the rewards of all edge nodes. The shared reward is represented as r(t), and its expression is as follows:

[0037]

[0038] Furthermore, the data collection and processing methods for the training data of the decision engine's corresponding model are as follows:

[0039] Each edge node i obtains its new local state o from the video analytics system at the start of time slot t. i The local state (t) and the global state s(t); the local state, global state, action, shared reward, new local state and new global state will be stored in the experience buffer as a transformation process;

[0040] After each batch of data is collected, the estimated advantage is calculated using GAE, and the cumulative reward is calculated in the trajectory τ. The network is then trained using that batch of data, and the policy objective of the Actor network and the loss objective of the Critic network are optimized using the Adam optimizer.

[0041] Furthermore, the update process for the model corresponding to the decision engine is as follows:

[0042] The Actor network of an edge node is represented as μ. θ For each edge node, the Actor network optimizes the control policy to increase rewards and updates it by maximizing the following objective, expressed as follows:

[0043]

[0044] Where B is the batch size of the sample. It is the probability ratio of importance sampling, which allows the sample to be used to update the parameter θold. It is calculated by GAE and used to evaluate the state and the quality of action pairs. ∈ indicates relatively good; otherwise, it is relatively bad; ∈ is a hyperparameter that controls the clipping intensity and is used to prevent the new policy function from changing too much relative to the old policy function; S is the policy entropy, which is used to increase exploration, and σ is the coefficient of the policy entropy;

[0045] The Critic network of an edge node is represented as V. φ The Critic network is trained according to the loss objective defined in the following equation:

[0046]

[0047] in It's a discount reward, s i It is the global state from this transition, V φ It is the Critic network function with parameter φ, V φold It is a Critic network with parameter φold, and ε is a hyperparameter controlling the shear strength.

[0048] Beneficial effects: Compared with the prior art, the distributed edge collaborative video analysis method based on multi-agent reinforcement learning described in this invention, upon implementation, has the following significant effects and substantial features:

[0049] (1) This invention proposes a multi-agent reinforcement learning method with an attention mechanism to learn the optimal policy. Each edge node can cooperate with other nodes to jointly process inference requests from different regions. Moreover, they can collaboratively perform video frame preprocessing, model selection, and request scheduling to optimize the system performance.

[0050] (2) This invention can effectively improve inference accuracy and reduce overall latency, and can cope with the time-varying and unbalanced nature of engineering loads at different edge nodes, as well as dynamic network bandwidth. Compared with a large number of baseline methods, this method can significantly improve the overall performance of the system and obtain higher rewards. Attached Figure Description

[0051] Figure 1 It is a video analytics system framework based on edge node collaboration;

[0052] Figure 2 It is the network structure of Actor network and Critic network in multi-agent reinforcement learning;

[0053] Figure 3 This is a comparison of the overall reward of the present invention with that of the baseline method;

[0054] Figure 4 This is a comparison of the average accuracy, overall latency, and percentage of dropped video frames between the present invention and the baseline method. Detailed Implementation

[0055] To illustrate the technical solution of this invention in detail, further explanation is provided below with reference to the accompanying drawings.

[0056] This invention provides a video analysis and processing method based on multi-agent reinforcement learning for edge collaboration. Edge nodes are located in different geographical locations and can cooperate with other edge nodes to process video analysis inference requests from their respective regions. When an edge node receives an inference request, it can process the request itself or forward the request to other edge nodes for processing.

[0057] The first step is to implement the method described in this invention, which includes the following steps:

[0058] (1) Inference requests from different regions send video frames to the corresponding edge nodes i (i∈N) via the HTTP protocol;

[0059] (2) When edge node i (i∈N) receives an inference request, the decision engine determines the inference request based on the current system state o. i (t) Make a decision, and then control decision a i (t) is applied to video frames, and the decisions include the selection of inference nodes, the selection of DNN models, and the selection of resolution;

[0060] (3) Edge node i will reduce the video frame to the specified resolution based on the resolution decision in order to reduce the transmission delay and inference delay between edge nodes;

[0061] (4) Edge node i determines whether the inference of the video frame is performed locally or forwarded to another edge node based on the inference node decision;

[0062] (5) If the video frame of the current time slot is inferred at the local edge node, the preprocessed video frame is placed in the local inference queue and waits for the selected DNN model deployed on the local edge node to perform inference.

[0063] (6) If the local edge node is overloaded, the video frame will be assigned to another edge node. The preprocessed video frame will be placed in the scheduling queue and wait to be forwarded to another edge node for inference via gRPC.

[0064] Based on the above method, the method includes a multi-agent reinforcement learning algorithm for constructing a decision engine, thereby constructing a model corresponding to the decision engine. The construction and training of the model are implemented as follows:

[0065] 1) For the edge computing video analytics system, it is considered as a discrete-time system, where time is denoted as t = 0, 1, 2, ... In each time slot t, each edge node i observes its local state in time slot t.

[0066] o i (t)=(λ i (t),l i (t),qij (t),b ij (t))

[0067] Where λ i (t) is the average inference request arrival rate of several time slot edge nodes i before time slot t, l i (t) is the length of the local inference queue of edge node i at time slot t, q ij (t) is the length of the scheduling queue from edge node i to edge node j, b ij (t) is the bandwidth between edge node i and edge node j at time slot t.

[0068] Assume that each edge node can only observe its local state. The global state of the environment consists of the local states observed by each edge node. The global state of the environment at time slot t is represented as:

[0069] s(t)=(o1(t),o2(t),...,o N (t))

[0070] Where N is the number of edge nodes.

[0071] 2) Local state of each edge node i (t) will be input into its Actor network μ i,θ The class log probability distribution of each action is obtained, and then the action 'a' of each edge node is obtained by sampling based on the probability distribution. i (t),

[0072] p i (t)=μ i,θ (o i (t)),a i (t)~p i (t)

[0073] Where, p i (t) represents the class log probability distribution of edge node i at time slot t. Then, the system will perform video analysis on the inference request based on the control action.

[0074] The action of each edge node includes the selected edge node (i.e., local inference or forwarding to another edge node), the selected DNN model for inference, and the resolution for video frame preprocessing. This invention represents the control action of edge node i at time slot t as follows:

[0075] a i (t)=(e,m,v),e∈E,m∈M,v∈V

[0076] Where e is the selected edge node for inference, E is the set of edge nodes, m is the selected DNN model for inference, M is the set of available DNN models, v is the selected resolution for video frame preprocessing, and V is the set of available video resolutions. If the selected edge node for inference is the same as the edge node that received the inference request, inference is performed locally. Otherwise, the request is forwarded to the appropriate other edge node for inference.

[0077] 3) At the end of time slot t, each edge node i will calculate its reward r during time slot t. i (t) and the shared reward r(t) of the edge system during time slot t.

[0078] If an inference request is successfully completed, its reward is calculated as a linear combination of accuracy and latency. Conversely, if a request is dropped, its reward is defined as a larger, weighted penalty value. Specifically, the reward for the j-th request performed on edge node i during time period t is defined as follows:

[0079]

[0080] Where g i,j (t) represents the waiting time in the queue for the j-th request at edge node i during time slot t, where T is the video frame dropping threshold, and α i,j (t) represents the j-th request made on edge node i during time slot t, d i,j (t) represents the total delay of the j-th request made on edge node i during time slot t, where F is the penalty constant and ω is the penalty weight for the overall delay.

[0081] The reward r of edge node i during time slot t i (t) represents the sum of rewards obtained by edge node i for inference during time slot t.

[0082]

[0083] Among them, P i (t) is the number of inference requests made by edge node i during time slot t.

[0084] To optimize the overall performance of collaborative work among edge nodes, this method designs the reward function as a shared reward, which is the sum of the rewards of all edge nodes. The shared reward is denoted as r(t).

[0085]

[0086] 4) Each edge node i obtains its new local state o from the video analytics system at the start of time slot t. i(t) and global state s(t). Local state, global state, action, shared reward, new local state, and new global state will be stored in the experience buffer as a transition process, i.e., (o i (t),s(t),a i (t),r(t),o i (t+1),s(t+1)).

[0087] 5) Repeat steps (1) to (4) continuously. After collecting a sufficient set of data (a certain number of transformation processes), use GAE to calculate the estimated advantage in this trajectory τ and calculate the cumulative reward in the trajectory. Then use this batch of data to train the network and optimize the policy objective of the Actor network and the loss objective of the Critic network through the Adam optimizer.

[0088] The Actor network of an edge node is represented as μ. θ For each edge node, the Actor network optimizes its control policy to increase rewards and updates it by maximizing the following objectives:

[0089]

[0090] Where B is the batch size of the sample. It is the probability ratio of importance sampling, which allows the sample to be used to update the parameter θold. It is calculated by GAE and used to evaluate the state and the quality of action pairs. (Relatively good; otherwise relatively bad). ∈ is a hyperparameter that controls the pruning intensity, used to prevent the new policy function from changing too much relative to the old policy function. S is the policy entropy, used to increase exploration, and σ is the coefficient of the policy entropy.

[0091] The Critic network of an edge node is represented as V. φ The Critic network is trained based on the loss objective defined in the equation below.

[0092]

[0093] in It's a discount reward, s i It is the global state from this transition, V φ It is the Critic network function with parameter φ, V φold This is the Critic network with parameter φold, where ε is a hyperparameter controlling the shear strength to prevent the new Critic network function from changing too much compared to the old function.

[0094] (6) Deploy the trained reinforcement learning model to the corresponding edge nodes to obtain a decision agent, and make an optimized decision for the incoming inference request.

[0095] For specific implementation and application, combined with Figure 1 The following is a block diagram of a video analytics system, where each edge node consists of the following video analytics modules:

[0096] (a) Decision Engine Module: The decision engine at each edge node determines how to process each video frame arriving at the edge node. Video frame decisions include inference node selection, DNN model selection, and resolution selection. Decision agents at different edge nodes can collaboratively learn to obtain the optimal strategy.

[0097] (b) Preprocessing module: Based on the decision of the decision agent, video frames are reduced to a lower resolution to reduce transmission and inference latency between edge nodes. Lower video resolution results in shorter transmission and inference latency.

[0098] (c) Local Inference Module: Based on the decision of the decision agent, if the video frame needs to be inferred on the local edge node (i.e. the edge node that receives the inference request), the preprocessed video frame is put into the local queue and waits for inference to be performed by the selected DNN model deployed on the local edge node.

[0099] (d) Scheduling Module: Based on the decision of the decision agent, requests will be scheduled to another edge node if the local edge node may be overloaded. Preprocessed video frames will be placed in the scheduling queue, waiting to be forwarded to another edge node for inference by the selected DNN model deployed on that remote edge node.

[0100] For the decision agent in module (a), an optimal policy is learned using a multi-agent deep reinforcement learning approach. The network structures of the Actor network and the Critic network are as follows: Figure 2 As shown.

[0101] The input to the Actor network is the local state of the edge nodes, including the average inference request arrival rate over the past few time slots, the current local inference queue length, the current queue length scheduled to other edge nodes, and the bandwidth between edge nodes and other edge nodes. The average inference request arrival rate over the past few time slots on the edge node assesses the incoming workload of the edge node, and the current length of the local inference queue describes the pending workload of the edge node. The current length of the scheduling queue and the bandwidth between edge nodes and other edge nodes assess the scheduling latency. The output of the Actor network is a classification log probability distribution of three discrete actions. Discrete control actions can be generated by sampling from the classification distribution, including the selected edge node for inference, the selected DNN model for inference, and the selected resolution for video frame preprocessing. The input to the Critic network is the global state of the video analytics system, which consists of the local states of all agents, reflecting the current system information of the video analytics system. The output of the Critic network is the predicted value used to compute the GAE.

[0102] When training using the Actor-Critic framework, each edge node's Critic network needs to communicate with other nodes to learn about their local state. As the number of edge nodes increases, the input dimensionality of the Critic network grows, and irrelevant information can lead to significant value prediction errors, thus affecting the Actor network's ability to learn optimal strategies. To avoid indiscriminately focusing on all information, this invention employs a multi-head attention mechanism to extract valuable information. For example... Figure 2 As shown, after passing through the embedding layer, the local state of the edge nodes is fed into the multi-head attention network to extract more valuable system information, thereby enabling the Critic network to make better predictions.

[0103] During training, each decision agent is given additional information that cannot be observed locally, enabling them to collaboratively learn the optimal policy. After training, each decision agent only needs its local state to make decisions, thus achieving distributed control of edge nodes and reducing communication costs.

[0104] For the preprocessing resolution in module (b), the original video resolution is 1080P, which can be reduced to 720P, 480P, 360P, or 240P through video preprocessing.

[0105] For the DNN models in modules (c) and (d), this invention deploys four DNN-based object detection models on each edge node, including two small models (fasterrcnn-mobilenet-320, fasterrcnnmobilenet) and two large models (retinanet-resnet-50, maskrcnn-resnet-50).

[0106] This invention was compared with several baseline methods:

[0107] (1)IPPO: Edge nodes learn their policies independently through Proximal Policy Optimization (PPO), which is a single-agent reinforcement learning method without inter-agent communication.

[0108] (2) Local-PPO: Each edge node only processes inference requests locally and does not assign them to other edge nodes. PPO is used to select the model and resolution.

[0109] (3) Shortest-Queue: Inference requests arriving within a time slot will be forwarded to the edge node with the shortest waiting queue length. This embodiment also considers three methods for selecting the model and resolution:

[0110] • Random: Select the random model and resolution.

[0111] •Min: Select the smallest model and the lowest resolution.

[0112] • Max: Select the largest model and the highest resolution.

[0113] (4) Random: Inference requests arriving at an edge node within a time slot will be randomly scheduled to the edge node. Model and resolution selection strategies include Random, Min, and Max.

[0114] (5) Local: Inference requests arriving at edge nodes will only be processed locally, taking into account the model and resolution selected by Random, Min and Max methods.

[0115] like Figure 3As shown, the overall rewards of the present invention and baseline methods are compared under different weights. The present invention achieves higher rewards compared to other baseline methods under different weights. IPPO yields lower rewards and is unstable during training because each agent learns its policy independently. Baseline methods that process video frames locally without scheduling (i.e., Local-PPO, Local-Random, Local-Min, and Local-Max) perform poorly when the workload of edge nodes is high because edge nodes cannot collaborate with other nodes to utilize idle resources on other nodes to process inference requests. Baseline methods using the most complex models and highest resolutions (i.e., Shortest-Queue-Max, Random-Max, and Local-Max) perform poorly due to larger inference and transmission latency when the penalty weights are large (i.e., ω = 5, 15). Choosing the simplest model and lowest resolution (i.e., Shortest-Queue-Min and Random-Min) reduces overall latency and increases rewards. In contrast, the present invention can achieve higher rewards because it can dynamically select the most suitable resolution, model, and edge nodes for video frames.

[0116] like Figure 4 As shown, the performance metrics of average accuracy, average total latency, and average video packet loss rate for different methods are illustrated. Figure 4 As shown in (a), the method described in this invention, like IPPO, has high accuracy, such as... Figure 4 (b) and Figure 4 As shown in (c), the present invention achieves lower overall latency and video frame loss rate, which verifies that the present invention can more effectively schedule inference requests among edge nodes, thereby achieving higher performance. The average accuracy of Local-PPO is close to that of the present invention; however, in Local-PPO, each edge node can only process inference requests independently and cannot schedule them. When the workload of edge nodes is large, it leads to longer queuing delays, resulting in more video frames being dropped. Therefore, as... Figure 4 (b) and Figure 4 As shown in (c), Local-PPO results in greater overall latency and video frame drop rate. Local-Random, Local-Min, and Local-Max also have similar issues. The baseline methods Shortest-Queue-Min and Random-Min always choose the simplest model and the lowest resolution. Figure 4 (a) and Figure 4As observed in (b), these methods have the lowest inference accuracy and the lowest overall latency. However, they cannot dynamically select the most suitable model and resolution, resulting in lower performance. Even so, as... Figure 4 As shown in (c), these methods have a higher video packet loss rate than the method of this invention because they cannot effectively schedule inference requests between edge nodes, resulting in poor performance. Since Shortest-Queue-Max, Random-Max, and Local-Max always select the largest model and the highest resolution, it is possible to... Figure 4 (a) and Figure 4 As observed in (b), these methods achieve the highest accuracy, but this also results in the greatest overall latency. Furthermore, as shown in 4(c), these methods exhibit a high percentage of dropped video frames, indicating insufficient system processing power. Even though Short-Queue-Max and Random-Max can schedule inference requests across different edge nodes, these methods still lead to significant queuing latency and video packet loss rates.

[0117] Combination Figure 3 and Figure 4 The experimental results can verify the superiority of the present invention in edge collaborative video analysis scenarios.

Claims

1. A distributed edge collaborative video analysis method based on multi-agent reinforcement learning, characterized in that: This method is based on a distributed edge computing video analytics system, which deploys DNN models on edge nodes located in different geographical locations. Each edge node receives video analytics inference requests from its corresponding region. Through collaborative learning, the overall cost of the system is minimized, and the method includes the following steps: (1) Reasoning requests from different regions are processed through http The protocol sends video frames to the corresponding edge nodes. ; (2) When edge nodes When a reasoning request is received, the decision engine determines the decision based on the current system state. Make a decision, and then control the decision. Applied to video frames, the decisions include the selection of inference nodes, the selection of DNN models, and the selection of resolution; (3) Edge nodes The video frames will be reduced to a specified resolution based on the resolution decision, in order to reduce transmission latency and inference latency between edge nodes; (4) Edge nodes The inference node determines whether the inference of a video frame is performed locally or forwarded to another edge node. (5) If the video frame of the current time slot is inferred at the local edge node, the preprocessed video frame is placed in the local inference queue and waits for the selected DNN model deployed on the local edge node to perform inference. (6) If the local edge node is overloaded, the video frame will be reassigned to another edge node, and the preprocessed video frame will be placed in the scheduling queue to wait for processing. gRPC Forward to another edge node for inference; The update and training methods for the corresponding model of the decision engine are as follows: Represent the Actor network of an edge node as follows: For each edge node, the Actor network optimizes the control policy to increase rewards and updates it by maximizing the following objective, expressed as follows: in It is the sample size. It is the probability ratio of importance sampling, which allows the sample to be in the range of parameters. Used to update parameters , It is the discounted reward of edge node i, calculated by GAE, used to evaluate the quality of the state-action pair. A value greater than 0 indicates a relatively good result; otherwise, it indicates a relatively poor result. It is a hyperparameter that controls the clipping intensity and is used to prevent the new strategy function from changing too much relative to the old strategy function; σ is the strategy entropy, used to increase exploration; Representing the Critic network of an edge node as follows: The Critic network is trained according to the loss objective defined in the following equation: in It's a discount reward. Represents edge nodes In the time slot t Transitioning to the global state in time slot t+1 transition, The parameter is The Critic network function, The parameter is The Critic network below, It is a hyperparameter that controls shear strength.

2. The distributed edge collaborative video analysis method based on multi-agent reinforcement learning according to claim 1, characterized in that: The decision engine described in step (2) is based on a multi-agent reinforcement learning algorithm and includes the following process: Considering discrete-time systems, the local state of each edge node is... Input into the Actor network In the process, the logarithmic probability distribution of each action category is obtained, and then the action of each edge node is obtained by sampling according to the probability distribution. The expression for the category log probability distribution of each action is as follows: in, Represents edge nodes In the time slot The log-log probability distribution of the categories at that location, Represents edge nodes The parameters are The Actor network, whose input is It is used to generate the class log probability distribution for each action. ; The distributed edge computing video analytics system will perform video analysis on inference requests based on control actions. The actions of each edge node include deciding whether to perform local inference or forward to another edge node, selecting the DNN model for inference, and the resolution for video frame preprocessing. For edge nodes In the time slot The control actions at this location are defined as follows: in These are the edge nodes selected for inference. It is a set of edge nodes. It is the DNN model selected for inference. It is a collection of available DNN models. It is the resolution selected for video frame preprocessing. It is the set of available video resolutions; If the selected edge node for inference is the same as the edge node that received the inference request, inference is performed locally; otherwise, the request will be forwarded to the corresponding other edge node for inference.

3. The distributed edge collaborative video analysis method based on multi-agent reinforcement learning according to claim 2, characterized in that: The method is in time slots At the end, each edge node It will be calculated in the time slot. Rewards during the period and time slots The video analytics system shared rewards during this period ; In the method, if an inference request is successfully completed, its reward is calculated as a linear combination of accuracy and latency; conversely, if a request is dropped, the reward for that inference request is defined as a weighted penalty value. Defined in time period During the period at the edge node The first time on The reward for each request is: in For time slots Edge nodes during the period Upper The waiting time of each request in the queue The threshold for dropped frames in the video. For time slots During the period at the edge node The first time on This request, Time slot During the period at the edge node The first time on Total latency for each request Let be the penalty constant. The penalty weight for the overall delay; edge nodes In the time slot Rewards during the period In time slot Edge nodes during the period The sum of rewards obtained from reasoning is expressed as: in, It is an edge node In the time slot Number of inference requests made during the period; To optimize the overall performance of collaborative work among edge nodes, this method designs the reward function as a shared reward, which is the sum of the rewards of all edge nodes. This shared reward is represented as... Its expression is as follows: 。 4. The distributed edge collaborative video analysis method based on multi-agent reinforcement learning according to claim 3, characterized in that: The training data collection and processing method for the decision engine's corresponding model is as follows: Each edge node In the time slot At the beginning, obtain its new local state from the video analytics system. and global state The local state, global state, action, shared reward, new local state, and new global state will be stored in the experience buffer as a transformation process. After each batch of data is collected, the estimated advantage is calculated using GAE, and the cumulative reward is calculated in the trajectory τ. The network is then trained using that batch of data, and the policy objective of the Actor network and the loss objective of the Critic network are optimized using the Adam optimizer.

5. The distributed edge collaborative video analysis method based on multi-agent reinforcement learning according to claim 2, characterized in that: In step (2), for a discrete-time system, where time is denoted as... In each time slot Each edge node In the time slot Observe their local state; the expression takes the following form: in In the time slot Previous time slot edge nodes The average inference request arrival rate, It is an edge node In the time slot The length of the local inference queue at that location. It is an edge node To edge nodes The length of the scheduling queue, It is a time slot Edge nodes and edge nodes Bandwidth between; Assuming each edge node can only observe its local state, the global state of the environment consists of the local states observed by each edge node, and the time slots are... The global state of the environment is represented as follows: in, This represents the number of edge nodes.

Images