DDQN-Based UAV Cooperative Video Processing Method in Edge Environment

Abstract

A method for collaborative video processing of unmanned aerial vehicles based on DDQN in an edge environment, and the steps of running in the mobile edge server UES are as follows: SA1. The mobile edge server UES collects system information; SA2. Input the system state into the model trained by the DDQN algorithm; SA3. Determine whether the mobile edge server UES moves. When it moves, the mobile edge server UES moves to a specified position and establishes a communication link with the selected intelligent camera EC, otherwise enter step SA6; SA4. Establish a process for video analysis according to the allocated port number; SA5. Receive the data stream transmitted by the selected intelligent camera EC, return the result to the selected intelligent camera EC after processing, and then determine whether there is an intelligent camera EC with exhausted power in the system. When there is, the work ends. When there is no, enter step SA6; SA6. The mobile edge server UES hovers and waits, and enter step SA1. This application can share the energy consumption overhead of task processing for intelligent cameras EC with less power and extend their working life.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicles, and particularly relates to a method for collaborative video processing of unmanned aerial vehicles based on DDQN in an edge environment. Background Art

[0002] Mobile Edge Computing (MEC) is an emerging paradigm that utilizes the computing resources of edge devices to process tasks for mobile terminal users. This provides end-users with a user experience of low latency and high quality of service. In recent years, a mobile edge computing architecture based on Unmanned Aerial Vehicles (UAVs) has been proposed, and UAVs equipped with computing, storage, and communication resources are called Mobile Edge Servers UES (UES). In specific scenarios such as emergency rescue and wilderness search and rescue, UAVs have the advantage of not being restricted by terrain. UES inherits the characteristics of UAVs and can provide more flexible, easier, and faster computing services than fixed MEC infrastructures in complex scenarios.

[0003] In the era of the Internet of Everything, various video acquisition terminals and video services are widely used, and video processing technologies in mobile environments have been extensively studied. For mobile video data processing, it is necessary to design an efficient task scheduling strategy for specific computing resources to ensure the real-time performance and reliability of video services. However, in complex scenarios such as disasters, it is difficult to obtain real-time and effective information using video surveillance systems. The existing intelligent cameras (ECs) are very suitable for assisting rescue operations in disaster scenarios. However, when a disaster occurs, the ECs may lose power supply. Backup batteries can be installed for the ECs, but these batteries cannot support the ECs for long-term video analysis. Once the battery runs out, the life of the EC ends. For this reason, we can offload the video analysis tasks of the ECs to the UES to reduce the energy consumption overhead of EC task processing and extend the working duration of the ECs. When there are multiple ECs with insufficient power supply in a video surveillance system, multiple issues need to be considered when using UES to assist these ECs in processing tasks. 1. How to select appropriate offloading users, and the long-term energy consumption of all users should be considered; 4. Although UAVs have good mobility, their load-bearing capacity and on-board energy are limited, and problems such as function loss and low timeliness of task execution may occur. Therefore, it is crucial to plan the flight trajectory of the UAVs. Summary of the Invention

[0004] Aiming at computationally intensive tasks with sensitive response time in a mobile edge environment, to solve the problem of insufficient power supply of the terminal but high energy consumption during task execution, for this purpose, the present invention proposes a method for collaborative video processing of unmanned aerial vehicles based on DDQN in an edge environment, and the specific scheme is as follows:

[0005] Unmanned Aerial Vehicle (UAV) Cooperative Video Processing Method Based on DDQN in Edge Environment, including a Mobile Edge Server (UES), and the steps running in the Mobile Edge Server (UES) are as follows:

[0006] SA1. The Mobile Edge Server (UES) collects system information;

[0007] SA2. Input the system state into the model trained by the DDQN algorithm;

[0008] SA3. Determine whether the Mobile Edge Server (UES) moves. When it moves, the Mobile Edge Server (UES) moves to the specified position and establishes a communication link with the selected Intelligent Camera (EC). Otherwise, go to step SA6;

[0009] SA4. The Mobile Edge Server (UES) establishes a video analysis process according to the allocated port number;

[0010] SA5. The Mobile Edge Server (UES) receives the data stream transmitted by the selected Intelligent Camera (EC), returns the result to the selected Intelligent Camera (EC) after processing, and then determines whether there is an Intelligent Camera (EC) with exhausted battery in the system. When there is, the work ends. When there isn't, go to step SA6;

[0011] SA6. The Mobile Edge Server (UES) hovers and waits, and goes to step SA1.

[0012] Specifically, a priority queue is set for all the Intelligent Cameras (EC). When the Mobile Edge Server (UES) finishes processing the task of one Intelligent Camera (EC), the priorities of all the Intelligent Cameras (EC) are re - sorted;

[0013] The priority p of the i - th Intelligent Camera (EC) i is:

[0014]

[0015] where e i represents the remaining battery ratio of the i - th Intelligent Camera (EC), represents the total remaining battery of all the Intelligent Cameras (EC).

[0016] Specifically, it also includes an Intelligent Camera (EC), and the steps running in the Intelligent Camera (EC) are as follows:

[0017] SB1. The Intelligent Camera (EC) generates a video analysis task and sends a scheduling request to the Mobile Edge Server (UES);

[0018] SB2. Determine whether the mobile edge server UES responds to the request. If it responds to the request, establish a video stream transmission process according to the assigned IP address and port number. The mobile edge server UES feeds back the execution time and energy consumption data of the task processing to the intelligent camera EC; otherwise, execute the computing task locally and send the completion information process according to the IP address and port of the mobile edge server UES.

[0019] SB3. Determine whether there is remaining power in the intelligent camera EC. When there is no remaining power, end the work; when there is remaining power, return to step SB1.

[0020] Specifically, the DDQN algorithm is as follows:

[0021] SC1. Find the optimal trajectory among several UES movement trajectories. Each UES trajectory corresponds to a state sequence, and the state sequence consists of the action space, state space, and reward function of the mobile edge server UES.

[0022] SC2. Design a UES trajectory planning algorithm for the mobile edge server according to the DDQN algorithm to enable the mobile edge server UES to assist the appropriate intelligent camera EC in processing tasks, and calculate the sum of the reduced energy consumption of all intelligent cameras EC.

[0023] SC3. After the DDQN model training is completed, deploy it to the mobile edge server UES. When it receives a scheduling request from an intelligent camera EC, it will generate a new state and input it into the trained model. The model will output an action to guide the mobile edge server UES to perform the next action.

[0024] Specifically, in step SC1, the dynamic space is denoted as:

[0025] A t ={1, 2, …, N};

[0026] where N is the position mapping of N intelligent cameras EC to N offloading points.

[0027] Specifically, in step SC1, the state space S t includes two parts: the remaining power ratio of each intelligent camera EC and the scheduling request, denoted as:

[0028] S t =[e 1 , e 2 , …, e N ; α 1 , α 2 , …, α N

[0029] e N ​is the remaining power ratio of the Nth intelligent camera EC, α N is the scheduling request of the Nth intelligent camera EC. When α N = 1 indicates that EC N requests task scheduling.

[0030] Specifically, in step SC1, the reward function is the reduced energy overhead of the intelligent camera EC with the assistance of the mobile edge server UES, and the calculation is as follows:

[0031]

[0032] where

[0033]

[0034]

[0035]

[0036] where: and respectively represent the energy consumption of local task processing and local standby of the intelligent camera EC in state S t ; i The standby energy consumption of the intelligent camera EC and respectively represent the task transmission energy consumption of the intelligent camera EC in state S t and the standby energy consumption of the intelligent camera EC when the mobile edge server UES assists in processing tasks i ; i The standby power and transmission power of the intelligent camera EC are represented respectively; respectively represent the standby power and transmission power of the intelligent camera EC i ; respectively represent the local computing time, the transmission time of task offloading, and the time for the mobile edge server UES to process the tasks of the intelligent camera EC i ; i The time for the mobile edge server UES to process the tasks of the intelligent camera EC

[0037] Specifically, in step SC2, the sum of the reduced energy consumption of all intelligent cameras EC is:

[0038]

[0039] Satisfies the following two conditions:

[0040]

[0041] E UES > 0; (2)

[0042] Where T is the termination state, condition (1) is the limit on the number of intelligent cameras EC that the mobile edge server UES can assist each time; condition (2) is the limit on the on-board energy of the mobile edge server UES. When the mobile edge server UES loses power, the system also terminates operation.

[0043] The beneficial effects of the present invention are as follows: The architecture proposed in this paper is based on a two-layer architecture composed of mobile edge computing resources and edge terminal computing resources. The task offloading strategy is studied based on the characteristics of unmanned aerial vehicles to achieve efficient management and utilization of the computing and energy resources of edge terminals, mobile edge servers, and the bandwidth resources between the two-layer architectures. This mobile edge server scheduling scheme can share the energy consumption overhead of task processing for intelligent cameras EC with less power, extending their working life. Brief Description of the Drawings

[0044] Figure 1 It is the state change process of the mobile edge server UES during the algorithm training process;

[0045] Figure 2 It is the state change process of the intelligent camera EC during the algorithm training process.

[0046] Figure 3 It is the detailed process of algorithm model training and updating. Detailed Embodiment

[0047] As Figures 1-3 shown, the method for collaborative video processing of unmanned aerial vehicles based on DDQN in the edge environment includes local edge devices for communication and a mobile edge server UES carried by the unmanned aerial vehicle. The edge device is an intelligent camera EC. The intelligent camera EC regularly generates a video analysis task of a fixed size. After the task is generated, the intelligent camera EC offloads the task to the mobile edge server UES for processing or processes it locally. Assuming that the remaining power of each intelligent camera EC is limited, offloading the task can reduce the energy consumption overhead of task processing and extend the operating life of the intelligent camera EC. And each intelligent camera EC is an independent individual, and their hardware architectures, deployed task types, and the size of the tasks generated intermittently are different.

[0048] Specifically, to achieve the above object, as Figure 1 shown, the steps of running in the mobile edge server UES are as follows:

[0049] SA1. The mobile edge server UES collects system information;

[0050] SA2. Input the system state into the model trained by the DDQN algorithm;

[0051] SA3. Determine whether the mobile edge server UES moves. When it moves, the mobile edge server UES moves to the specified position and establishes a communication link with the selected intelligent camera EC. Otherwise, proceed to step SA6;

[0052] SA4. The mobile edge server UES establishes a video analysis process according to the allocated port number;

[0053] SA5. The mobile edge server UES receives the data stream transmitted by the selected intelligent camera EC, returns the result to the selected intelligent camera EC after processing, and then determines whether there is an intelligent camera EC with exhausted power in the system. When there is, the work ends. When there is none, proceed to step SA6;

[0054] SA6. The mobile edge server UES hovers and waits, and proceeds to step SA1.

[0055] When the latency sum of the mobile edge server UES moving over the system and assisting the intelligent camera EC in processing tasks exceeds the specified upper limit, the mobile edge server UES does not respond to the system's task requests. If multiple intelligent cameras EC request scheduling simultaneously, the intelligent camera EC with a lower priority also does not receive a response from the mobile edge server UES. The priority scheme is as follows:

[0056] Set a priority queue for all intelligent cameras EC. When the mobile edge server UES finishes processing the task of one intelligent camera EC, re - sort the priorities of all intelligent cameras EC. So at a certain moment, when multiple ECs request task scheduling simultaneously, the one with a higher priority is responded to first. This avoids some intelligent cameras EC monopolizing the UES resources. The priority p of the i - th intelligent camera EC i is:

[0057]

[0058] where e i represents the remaining power ratio of the i - th intelligent camera EC, and represents the sum of the remaining power of all intelligent cameras EC.

[0059] As Figure 2 shown, the running steps of a single intelligent camera EC based on the assistance of the mobile edge server UES are as follows:

[0060] SB1. The intelligent camera EC generates a video analysis task and sends a scheduling request to the mobile edge server UES;

[0061] SB2. Determine whether the mobile edge server UES responds to the request. If it responds to the request, establish a video stream transmission process according to the assigned IP address and port number. The mobile edge server UES feeds back the execution time and energy consumption data of task processing to the intelligent camera EC; otherwise, execute the computing task locally and send a completion information process according to the IP address and port of the mobile edge server UES.

[0062] SB3. Determine whether there is remaining power in the intelligent camera EC. When there is no remaining power, end the work; when there is remaining power, return to step SB1.

[0063] The goal pursued by this system is to save the maximum energy consumption before the system stops running.

[0064] The above DDQN algorithm is specifically as follows:

[0065] SC1. Find the optimal trajectory among several UES movement trajectories. Each UES trajectory corresponds to a state sequence, and the state sequence is composed of the action space, state space, and reward function of the mobile edge server UES.

[0066] Map the positions of N intelligent cameras EC in the system to N offloading points. Each time the mobile edge server UES chooses to move to any offloading point, the dynamic space is denoted as:

[0067] A t ={1, 2, …, N}

[0068] When any intelligent camera EC in the system generates a video analysis task, it sends a task scheduling request to the mobile edge server UES. If the task request is responded to at this time, the mobile edge server UES generates a system state space S t at the end of the scheduling request acceptance period. The state space includes two parts: the remaining power ratio of each intelligent camera EC and the scheduling request, denoted as:

[0069] S t =[e 1 , e 2 , …, e N ; α 1 , α 2 , …, α N

[0070] e N is the remaining power ratio of the Nth intelligent camera EC, and α N is the scheduling request of the Nth intelligent camera EC. When α N = 1, it means EC N ​Request task scheduling. The mobile edge server UES receives call requests from the intelligent cameras EC in the system at a certain time period. Meanwhile, when a certain intelligent camera EC generates a video analysis task, it sends a scheduling request to the mobile edge server UES. If no response is received within a certain time interval, the task is processed locally. The scheduling request includes the type of task deployed, the coordinates, and the starting point location of the mobile edge server UES.

[0071] With the assistance of the mobile edge server UES, when the energy consumption of the intelligent camera EC is used as the reward for the mobile edge server UES, the reward function is as follows:

[0072]

[0073] Among them

[0074]

[0075]

[0076]

[0077] Among them: and respectively represent the local task processing energy consumption and local standby energy consumption of the intelligent camera EC t in the state S i ; and respectively represent the task transmission energy consumption of the intelligent camera EC t in the state S i and the standby energy consumption of the intelligent camera EC i when the mobile edge server UES assists in processing the task. respectively represent the standby power and transmission power of the intelligent camera EC i ; respectively represent the local computing time, the transmission time for task offloading, and the time for the mobile edge server UES to process the task of the intelligent camera EC i ; i ;

[0078] SC2. Design a trajectory planning algorithm for the mobile edge server UES according to the DDQN algorithm to enable the mobile edge server UES to assist the appropriate intelligent camera EC in processing tasks. The sum of the reduced energy consumption of all intelligent cameras EC is:

[0079]

[0080] st satisfies the following two conditions

[0081]

[0082] E UES >0; (2)

[0083] where T is the termination state, condition (1) is the limit on the number of intelligent cameras EC that the mobile edge server UES can assist each time; condition (2) is the limit on the on-board energy of the mobile edge server UES. When the mobile edge server UES loses power, the system also terminates operation.

[0084] SC3. After the DDQN model is trained, it is deployed on the mobile edge server UES. When it receives a scheduling request from an intelligent camera EC, it will generate a new state and input it into the trained model. The model will output an action to guide the mobile edge server UES to perform the next action.

[0085] In this architecture, the upper layer is the mobile edge server side. The mobile edge server UES makes full use of its mobility to provide effective computing offloading and communication services for areas with insufficient computing resources; the lower layer is the edge terminal. The edge node performs reasonable preprocessing operations on the original video data and transmits the data to the UES according to the scheduling requirements, or executes locally.

[0086] Such as Figure 3 shown, the specific application of the DDQN algorithm in step SC3 is as follows:

[0087] It is divided into four parts: initialization, data generation, data training, and network update. It should be noted that this model is essentially a neural network. The characteristic of the DDQN algorithm is to input the state into the prediction neural network to obtain the value prediction of each action in this state. Then, the predicted value of the action and the actual value are processed through the mean square error loss function, and the parameters of the prediction neural network are updated by the gradient descent method according to the result. When the update effect of the prediction network parameters is better, the output action value will be closer to the true value. In this way, the trained neural network model can correctly guide us to select good UES actions.

[0088] For the true value y t of the action space A t of a mobile edge server UES, it is considered in two cases. The first is that after the mobile edge server UES executes this action, the system reaches the termination state, and the reward value of the action at this time is R t . The second is that the system does not reach the termination state. Then this state is input into the target network, and the value of the action with the largest value is selected from the output to form a new true value. Specifically, in Figure 3Implementation of lines 8 - 14 of the algorithm. In this step, the traditional DQN algorithm also uses the prediction network, but directly selects the action with the maximum value on the prediction network. Since the parameters of the prediction network are not updated in a timely manner, it may lead to an overestimation of the action value output. Therefore, DDQN introduces a target network to solve this problem. And the algorithm updates the parameters of the prediction network to the target network at a certain frequency ( Figure 3 lines 16 - 17). Finally, the algorithm also outputs the target network.

[0089] The input of the algorithm is the number of iterations M and the randomly simulated trajectories of UES during the training process, and the number of simulated trajectories is M. At the beginning of training, the algorithm tends to explore. As the number of iterations increases, the algorithm gradually makes iterative trade - offs on beneficial UES trajectories and finally obtains the optimal UES trajectory. This phenomenon is caused by the ε - greedy strategy we used ( Figure 3 line 3). The training batch N is the number of samples taken from the experience replay at one time, and these N samples are used later for updating the parameters of the prediction network.

[0090] The algorithm first needs to perform initialization processing, mainly randomly generating the parameter matrices of the prediction network Q and the target network Q′, obtaining the starting state S as shown above 0 . And generating an experience replay pool D to store samples.

[0091] For each iteration, it represents a movement trajectory of the mobile edge server UES, which is essentially a state sequence. In Figure 3 lines 1 - 2 of the algorithm, the algorithm is used to control the start and end of a state sequence. Initially, the termination state identifier Terminate is false. If the system terminates after the UES executes the action A t , then Terminate is set to true. This indicates the end of this UES trajectory and randomly enters the next UES trajectory ( Figure 3 line 6 of the algorithm).

[0092] From Figure 3 lines 3 - 7 of the algorithm, it is the process for the algorithm to generate training data and store the generated data in the experience replay pool. First, the state is input into the Q network, and the values of all UES movement actions are output. Then, the action A t is selected through the ε - strategy. Subsequently, the UES moves to the corresponding offloading point, assists the corresponding EC in processing tasks, and obtains the reward R according to the reward function t . After the task of the current EC is completed, the UES listens to the requests in the system. When there is a task scheduling request issued by a certain EC, the UES generates a new state S t+1 (lines 3 - 4). In Figure 3In the fifth line of the algorithm, the obtained data is stored in the experience replay pool D. Subsequently, it is judged whether S t+1 is a termination state, and the termination descriptor Terminate is updated.

[0093] Figure 3 In the seventh to fourteenth lines of the algorithm, the algorithm randomly selects N samples from the experience replay pool, and then updates the parameter matrix w of the Q network through gradient backpropagation 1 . In the seventeenth line of the algorithm, it is judged whether the current episode meets the condition for updating the parameters of the target Q' network. If it meets the condition, the parameter matrix w 1 of the Q network is copied to the target network Q', and the update of the target network parameters is completed.

[0094] After M iterations, the DDQN algorithm model training is completed. For the setting of the M value, it needs to be adjusted according to the training results. If the M value is too large, it is easy to cause overfitting, and if it is too small, it is easy to cause underfitting. After the model training is completed, it is deployed to the UES. When a new state is input from the model, an action will be output to guide the UES to assist the UC in processing the video analysis task.

[0095] The above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution of the present invention and its inventive concept, makes equivalent replacements or changes, and should be covered by the protection scope of the present invention.

Claims

1. Unmanned Aerial Vehicle (UAV) Cooperative Video Processing Method Based on DDQN in Edge Environment, Characterized in that, It includes a Mobile Edge Server (UES), and the steps running in the Mobile Edge Server UES are as follows: SA1. The Mobile Edge Server UES collects system information; SA2. Input the system state into the model trained by the DDQN algorithm; SA3. Determine whether the Mobile Edge Server UES moves. When it moves, the Mobile Edge Server UES moves to the specified position and establishes a communication link with the selected Smart Camera (EC). Otherwise, go to step SA6; SA4. The Mobile Edge Server UES establishes a video analysis process according to the assigned port number; SA5. The Mobile Edge Server UES receives the data stream transmitted by the selected Smart Camera EC, returns the result to the selected Smart Camera EC after processing, and then determines whether there is a Smart Camera EC with exhausted power in the system. When there is, the work ends. When there is no, go to step SA6; SA6. The Mobile Edge Server UES hovers and waits, and goes to step SA1; Set a priority queue for all Smart Cameras EC. When the Mobile Edge Server UES finishes processing the task of one Smart Camera EC, re - sort the priorities of all Smart Cameras EC; The priority p of the i-th intelligent camera EC i is as follows: where e i represents the remaining battery percentage of the i-th intelligent camera EC, and represents the total remaining battery of all intelligent cameras EC; The DDQN algorithm is as follows: SC1. Find the optimal trajectory among several UES movement trajectories. Each UES trajectory corresponds to a state sequence, and the state sequence is composed of the action space, state space, and reward function of the Mobile Edge Server UES; SC2. Design a UES trajectory planning algorithm for the Mobile Edge Server UES according to the DDQN algorithm to enable the Mobile Edge Server UES to assist the appropriate Smart Camera EC in processing tasks, and calculate the sum of the reduced energy consumption of all Smart Cameras EC; SC3. After the DDQN model training is completed, deploy it to the Mobile Edge Server UES. When it receives a scheduling request from a Smart Camera EC, it will generate a new state and input it into the trained model. The model will output an action to guide the Mobile Edge Server UES to perform the next action.

2. The Unmanned Aerial Vehicle (UAV) Cooperative Video Processing Method Based on DDQN in Edge Environment according to Claim 1, Characterized in that, It further includes a Smart Camera EC, and the steps running in the Smart Camera EC are as follows: SB1. The Smart Camera EC generates a video analysis task and sends a scheduling request to the Mobile Edge Server UES; SB2. Determine whether the Mobile Edge Server UES responds to the request. If it responds to the request, establish a video stream transmission process according to the assigned IP address and port number. The Mobile Edge Server UES feeds back the execution time and energy consumption data of task processing to the Smart Camera EC; otherwise, perform the calculation task locally and send a completion information process according to the IP address and port of the Mobile Edge Server UES; SB3. Determine whether there is remaining power in the Smart Camera EC. When there is no remaining power, end the work. When there is remaining power, return to step SB1.

3. The method for collaborative video processing of drones based on DDQN in the edge environment according to claim 1, characterized in that, the action space in step SC1 is denoted as: A t = {1, 2, …, N}; where N is the position mapping of N intelligent cameras EC to N offloading points.

4. The method for collaborative video processing of drones based on DDQN in the edge environment according to claim 1, characterized in that, State space S in step SC1 t It includes two parts, namely the remaining battery percentage of each intelligent camera EC and the scheduling request, and is denoted as: S t = [e 1 , e 2 , …, e N ; α 1 , α 2 , …, α N ​ e N is the remaining battery level percentage of the Nth intelligent camera EC, α N is the scheduling request of the Nth intelligent camera EC. When α N = 1, it means that EC N requests task scheduling.

5. The method for collaborative video processing of drones based on DDQN in the edge environment according to claim 1, characterized in that, the reward function in step SC1 is the energy overhead reduced by the intelligent camera EC with the assistance of the mobile edge server UES, and the calculation is as follows: where Wherein: and respectively represent the local task processing energy consumption and the local standby energy consumption of the intelligent camera EC t in the state S i ; and respectively represent the task transmission energy consumption of the intelligent camera EC t in the state S i and the standby energy consumption of the intelligent camera EC i when the mobile edge server UES assists in processing tasks; respectively represent the standby power and the transmission power of the intelligent camera EC i ; respectively represent the local computing time, the transmission time for task offloading, and the time for the mobile edge server UES to process the tasks of the intelligent camera EC i ; i ​ 6. The method for collaborative video processing of drones based on DDQN in the edge environment according to claim 3, characterized in that, the sum of the energy consumption reduced by all intelligent cameras EC in step SC2 is: satisfying the following two conditions E UES ＞0； (2) where T is the termination state, condition (1) is the limit on the number of intelligent cameras EC that the mobile edge server UES can assist each time; condition (2) is the limit on the on-board energy of the mobile edge server UES. When the mobile edge server UES loses power, the system also terminates operation.

Images