A multi-unmanned aerial vehicle cooperative auxiliary communication optimization method based on deep reinforcement learning

Abstract

The application relates to a multi-unmanned aerial vehicle cooperative auxiliary communication optimization method based on deep reinforcement learning, which comprises the following steps: reconstructing the state information of a ground user and unmanned aerial vehicles in a two-dimensional scene; constructing a real-time total power consumption mathematical model of a group of unmanned aerial vehicles; constructing an exploration incentive module; establishing sigma deep reinforcement learning networks with the same structure; establishing a mixed network based on monotonic value function decomposition; inputting all unmanned aerial vehicle individual Q functions to form an unmanned aerial vehicle overall value Q function, and obtaining a joint flight path which maximizes the unmanned aerial vehicle system throughput and minimizes the energy consumption. The application enhances the active exploration efficiency of the unmanned aerial vehicles to the environment in the initial training stage, effectively improves the overall training effect, and effectively improves the sample utilization rate of the model in combination with the established priority experience playback pool; through training of the overall Q function of the multiple unmanned aerial vehicles, the unmanned aerial vehicles can plan the best path to reduce the energy consumption while better meeting the communication quality demand of the ground users.

Description

Technical Field

[0001] This invention relates to the fields of deep reinforcement learning and UAV communication technology, and in particular to a multi-UAV cooperative assisted communication optimization method based on deep reinforcement learning. Background Technology

[0002] In recent years, with the rapid development of aviation and automation technologies, unmanned aerial vehicles (UAVs) have been widely used in various fields such as military, agriculture, transportation, and public administration due to their superior maneuverability, low cost, and ease of operation. By installing small communication devices on UAVs, they can act as relay nodes, providing communication services to ground users. Simultaneously, as mobile base stations, UAVs can collect data from ground-based wireless sensor networks. By optimizing the UAV's trajectory, data collection from ground equipment can be maximized.

[0003] Currently, unmanned aerial vehicle (UAV)-assisted wireless communication has become a research hotspot, involving multiple directions, including buffer networks, wireless sensor networks, heterogeneous cellular networks, massive MIMO, device-to-device communication, and disaster recovery communication. However, in situations where users are widely and dispersed, a single independent UAV is inadequate and struggles to meet the communication needs of ground users. Therefore, it is necessary to conduct in-depth research on schemes for multiple UAVs to collaborate in communication assistance.

[0004] In their paper "Trajectory Design and Access Control for Air-Ground Coordinated Communications System with Multiagent Deep Reinforcement Learning," Ruijin Ding et al. proposed an AGPAMADDPG algorithm based on MADDPG to coordinate UAVs and GUs. To handle the mixed action space, AG-PAMADDPG transforms the discrete actions of GUs into a continuous probability distribution and samples actions from the distribution. Numerical results show that AG-PAMADDPG outperforms the distributed DQN algorithm and AC-PAMADDPG, which only optimizes GU access control, in terms of network throughput and GU fairness. For the joint optimization problem of UAV trajectory design and power allocation, R. Zhong et al., in their paper "Multi-agent Reinforcement Learning in Noma-Aided UAV Networks for Cellular Offloading," proposed a multi-agent MDQN algorithm to learn such policies. Simulation results show that in multi-agent scenarios, the proposed MDQN has a faster convergence speed than the baseline traditional DQN. Furthermore, by using MDQN to design 3D trajectories, the overall network summation rate can achieve gains of 142% and 56% compared to methods using circular trajectories and 2D trajectories, respectively. However, MDQN's intensive training may generate significant communication overhead for collecting drone experience.

[0005] However, based on existing research, some challenges remain: first, the efficiency of sample utilization of global scene information is limited, resulting in limited guidance for UAV flight path planning; second, the energy consumption of UAVs needs to be considered in designing energy-saving trajectory strategies. Summary of the Invention

[0006] To address the issues of limited efficiency in utilizing global scene information samples and the need to consider UAV energy consumption when designing energy-saving trajectory strategies, this invention aims to provide a multi-UAV collaborative assisted communication optimization method based on deep reinforcement learning. This method improves sample utilization, helps UAVs plan optimal paths to reduce energy consumption, and better meets the communication quality requirements of ground users.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a multi-UAV cooperative assisted communication optimization method based on deep reinforcement learning, the method comprising the following sequential steps:

[0008] (1) Reconstruct the two-dimensional scene from the status information of ground users and UAVs to obtain scene observation information;

[0009] (2) Construct a mathematical model of the real-time total power consumption of the UAV group to establish optimization objectives for UAV training;

[0010] (3) Construct an exploration incentive module. The UAV loads the exploration incentive module and outputs the reward value obtained from exploring the new environment. Through the reward value obtained from exploring the new environment, comprehensive scene observation information can be obtained.

[0011] (4) Establish σ deep reinforcement learning networks with the same structure as decision models for each UAV. The input of the UAV decision model is comprehensive scene observation information, and the output of the UAV decision model is the individual Q function of the UAV.

[0012] (5) Establish a hybrid network based on monotonic function decomposition, input all individual UAV Q functions into the hybrid network, and the hybrid network combines all individual UAV Q functions into the overall UAV value Q function. Train the overall UAV value Q function through the loss function, and obtain the joint flight path that maximizes the throughput of the UAV system and minimizes energy consumption based on the mathematical model of the real-time total power consumption of the UAV group.

[0013] Step (1) specifically refers to: reconstructing the state information of ground users and drones into a two-dimensional scene, and through matrix fitting, denoting the two-dimensional scene as a K*K equidistant grid, with Φ ground users randomly distributed at the center of the equidistant grid at a constant speed ν. φ In a two-dimensional scene, the user moves randomly. The direction of the user's random movement is discretized into up, down, left, right, upper left, lower left, upper right, lower right, and stationary. σ drones fly at a fixed altitude Η above the scene. The drones are numbered u1, u2, ... u σ The drone's flight speed is constant at ν. uav The drone's flight direction is discretized into up, down, left, right, upper left, lower left, upper right, lower right, and stationary. The drone provides communication services to ground users within a circular area of ​​radius D. The entire mission cycle of the drone is discretized into multiple individual time intervals, each with a duration of t. The current system time is T, where T = t1, t2…t end , t end The moment when the UAV mission terminates is represented by the scene matrix containing ground user information at time t, and the matrix containing the location and communication range of each UAV are stacked to obtain the scene observation information of σ UAVs.

[0014] Step (2) specifically refers to: the real-time total power consumption mathematical model of the UAV group includes the UAV flight power consumption model and the communication power consumption model between the UAV and the ground user:

[0015]

[0016] In the formula, This represents the total real-time power consumption of the drone at time t. This represents the flight power consumption of the drone at time t. Let t be the power consumption for communication between the UAV and the ground user. E max The maximum power consumption allowed by the drone's battery capacity;

[0017] The expression for the power consumption model of the UAV flight is as follows:

[0018]

[0019] In the formula, p v Indicates the flight mobility of the drone, ν uav Indicates the flight speed of the drone. This represents the horizontal coordinate projection of the drone in the scene at time t. This represents the horizontal coordinate projection of the drone in the scene at time t-1; This represents the vertical coordinate projection of the drone in the scene at time t. This represents the vertical coordinate projection of the drone in the scene at time t-1;

[0020] The expression for the communication power consumption model between the UAV and ground users is as follows:

[0021]

[0022] In the formula, V r denoted by , e represents the signal transmission rate of the UAV during communication; e represents the transmission efficiency of the communication process; c represents the bandwidth limitation during communication between the UAV and the ground user; d represents the communication distance between the UAV and the ground user, and its mathematical expression is shown below;

[0023]

[0024] In the formula, H represents the height of the drone above the horizontal ground. This represents the x-coordinate of the ground user in the scene at time t. This represents the vertical coordinate of the ground user in the scene at time t.

[0025] Step (3) specifically refers to: the exploration incentive module adopts a Siamese network, which is a convolutional neural network; the input of the exploration incentive module is the scene observation information obtained at the current time t and the past scene observation information stored in the historical scene information pool, and the output is a feature similarity. The triplet loss function is used as the update method of the loss function, and then the output of the exploration incentive module is integrated with the reward signal; the Siamese network of the exploration incentive module includes a shared convolutional layer, a fully connected layer and a metric learning layer:

[0026] Shared convolutional layer: Two input images are processed through the same convolutional layer to extract features and share weights. The goal of this layer is to capture the shared features of the input images. The input dimension is (K, K, 2*(σ+1)).

[0027] Fully connected layer: The fully connected layer has 1024 nodes and uses the ReLU function as the activation function;

[0028] Metric learning layer: Calculates the similarity between two input scene observations in the feature space, and uses Euclidean distance or cosine similarity to measure the difference between the two input scene observations;

[0029] The mathematical expression for the triplet loss function is as follows:

[0030] L(A,P,N)=max(||f(A)-f(p)|| 2 -||f(A)-f(N)|| 2 +a, 0)

[0031] In the formula, f(A) is the input of the anchor point sample, which uses the observation of past scene information in the historical scene information pool as the anchor point sample; f(p) is the input of the positive sample, which is defined as the current scene information when the UAV can reach any past scene in the historical scene information pool from the current scene state within k steps; f(N) is the input of the negative sample, which is defined as the current scene information when the UAV cannot reach any past scene in the historical scene information pool from the current scene state within k steps; and a represents a predefined boundary.

[0032] In step (3), the mathematical expression output by the exploration incentive module is:

[0033] F(ω,M)=max(μ1,μ2,…,μ m )

[0034] In the formula, ω represents the state information observation at the current time t, M represents all past scene observation information stored in the historical scene information pool, and m represents the number of past scene observation information contained in the historical scene information pool, (μ1, μ2, ..., μ...). m) represents the feature similarity between ω and all past scene observation information in M;

[0035] Then, the feature similarity is integrated with the reward signal:

[0036]

[0037] In the formula, R represents the reward value obtained from exploring a new environment. For hyperparameters, F(ω, M) represents the maximum feature similarity between ω and all past scene observation information in M; the higher the feature similarity between the current scene observation information and the historical scene observation information, the lower the reward value; the lower the feature similarity between the current scene observation information and the historical scene observation information, the higher the reward value.

[0038] Step (4) specifically includes the following steps:

[0039] (4a) Define the state space and action space: Define the state space of each UAV as the scene observation information, and define the action space of each UAV as A, the mathematical expression of which is as follows:

[0040] A = (Up, Down, Left, Right, Top Left, Bottom Left, Top Right, Bottom Right, Stationary)

[0041] (4b) Constructing a deep reinforcement learning network: The deep reinforcement learning network includes an input layer, three convolutional layers, one pooling layer, one fully connected layer, and one output layer. The input layer is a fully connected layer with a dimension of (K, K, 2) to receive scene observation information observed by the UAV. The number of nodes in the output layer is equal to the size of the UAV's action space, and the individual Q-function Q of each UAV during training is obtained. i (s t a i ), where s t a represents the scene observation information at time t. i This represents the action chosen by the i-th drone;

[0042] (4c) Define the experience replay buffer and the priority experience replay pool: Create an experience replay buffer Δ to store the previous state information set, joint action set, joint reward value and the next state information set for each UAV. This enables experience replay during the training of the deep reinforcement learning network, breaking the temporal correlation. The mathematical expression for the experience replay buffer Δ is as follows:

[0043] Δ=(s t u t R t s t+1 )

[0044] In the formula, ut This indicates that all drones are in state s t The following is a set of combined actions, R t Let s represent the set of joint rewards obtained by all drones at time t. t+1 This indicates that the drone is performing u t The scene observation information that arrives at the next moment after the arrival;

[0045] A priority experience replay pool is created, with the same structure as the experience replay buffer Δ described above. The priority experience replay pool is used to store the experience of the drone obtaining high-reward rewards in each training round. When the drone learns from the experience replay samples, the experience replay buffer Δ and the priority experience replay pool are combined. Forty percent of the experience is extracted from the experience replay buffer Δ and sixty percent of the experience is extracted from the priority experience replay pool. The mixture is then used as the input of the deep reinforcement learning network.

[0046] (4d) Define Instant Rewards and state transition probability

[0047] Instant rewards This represents the total communication transmission rate between the UAV and n ground users within its sensing range at time t, and the UAV's flight power consumption. The difference is expressed as follows:

[0048]

[0049] In the formula, This represents the flight power consumption of the drone at time t. The communication transmission rate between the drone and the user is expressed as follows:

[0050]

[0051] In the formula: W represents the bandwidth allocated to each ground communication user by the UAV, and SNR is the signal-to-noise ratio, which refers to the ratio of the UAV power to the noise power in the channel. The calculation formula is as follows:

[0052]

[0053] In the formula: Let t be the communication power consumption between the UAV and the ground user at time t, N be additive white Gaussian noise, and L be the average path loss.

[0054] State transition probability State transition probability Indicates that the drone is in state s t Transition to the next state s t+1 The probability of this is expressed as follows:

[0055]

[0056] Where η represents the greedy coefficient in the action selection process, a t This indicates the action selected by the drone at time t;

[0057] (4e) Determine the incentive mechanism for the entire mission cycle of the communication system:

[0058] Total reward R obtained by the drone total as follows:

[0059]

[0060] In the formula: Γ represents the time when the task ends, γ represents the discount factor, which is used to balance the proportion of future rewards in the current cumulative rewards. After the drone performs the action, it obtains the state space after the change of the scene environment state and the new reward value.

[0061] The establishment of the hybrid network based on monotonic value function decomposition in step (5) specifically includes the following steps:

[0062] (5a) Input representation: Collect all individual UAV Q functions obtained in step (4) i (s t ,a i );

[0063] (5b) Flattening: Q function of all individual drones Q i (s t ,a i Flatten the structure;

[0064] (5c) Connect: Connect the individual Q functions of all UAVs after flattening to obtain a large vector of joint individual Q functions;

[0065] (5d) Define fully connected layers: Add two fully connected layers to learn the complex relationships between individual Q-functions, and combine the overall Q-function Q. total (s t ,u t Mapped to a higher dimension;

[0066] (5e) Define the activation function: Use the ReLU function to introduce nonlinearity;

[0067] (5f) Define the output layer: Add an output layer to output the overall Q function Q. total (s t ,u t ), where s t u represents the scene observation information at time t. t This indicates that all drones are in state st The following is a set of combined actions;

[0068] (5g) Define the individual global maximum value constraint that needs to be satisfied:

[0069]

[0070] In the formula, Q i Denotes the individual Q-function of the i-th UAV; s 1 Represents the state space of the first drone, u 1 This indicates the action selected by the first drone, s n Let u represent the state space of the nth UAV. n This represents the action selected by the nth drone;

[0071] Hybrid networks need to be constrained by monotonic conditions:

[0072]

[0073] In the formula, σ represents the number of drones;

[0074] (5h) Define the loss function:

[0075]

[0076] Where b represents the number of samples sampled from the experience pool, and θ represents the parameters of the hybrid network. The mathematical expression is as follows:

[0077]

[0078] Where, θ - These are the parameters of the target network, where γ represents the discount factor; u t+1 This indicates the action selected by the drone at time t+1;

[0079] The error of the timing difference is then expressed as:

[0080] TD=(r+γQ total (target))-Q total (evaluate)

[0081] In the formula, Q total (target) represents the state s t+1 At that time, the maximum value that can be obtained from all the actions of the drone; Q total (evaluate) represents the total Q that can be obtained in state s based on the actions output by the deep reinforcement learning network for each drone. total, r represents the joint total reward value obtained by each drone after executing the action output by the deep reinforcement learning network in state s;

[0082] (5i) Backpropagation: Backpropagation is performed using the loss function to update the weight parameters of the individual Q-function and reduce Q. total (target) and Q total The difference between (evaluate) is calculated; steps (5a) to (5i) are repeated until the loss function L(θ) converges.

[0083] As can be seen from the above technical solution, the beneficial effects of the present invention are as follows: First, by establishing a mathematical model of the real-time total power consumption of the UAV group, the present invention establishes an optimization target for UAV training, effectively solving the problem of collaborative planning and communication among multiple UAVs under limited power constraints; Second, the present invention establishes an exploration incentive module, enhancing the UAV's active exploration of the environment in the initial stage of training, guiding the UAV to acquire key information more effectively, thus improving the overall training effect, and, combined with the established priority experience replay pool, effectively improving the model's utilization rate of samples; Third, by training the overall Q function of multiple UAVs, the present invention ultimately optimizes the individual Q function of the UAVs, enabling the UAVs to learn a better collaborative method, helping them plan the optimal path to reduce energy consumption while better meeting the communication quality requirements of ground users. Attached Figure Description

[0084] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0085] like Figure 1 As shown, a multi-UAV cooperative assisted communication optimization method based on deep reinforcement learning is proposed, which includes the following sequential steps:

[0086] (1) Reconstruct the two-dimensional scene from the status information of ground users and UAVs to obtain scene observation information;

[0087] (2) Construct a mathematical model of the real-time total power consumption of the UAV group to establish optimization objectives for UAV training;

[0088] (3) Construct an exploration incentive module. The UAV loads the exploration incentive module and outputs the reward value obtained from exploring the new environment. Through the reward value obtained from exploring the new environment, comprehensive scene observation information can be obtained.

[0089] (4) Establish σ deep reinforcement learning networks with the same structure as decision models for each UAV. The input of the UAV decision model is comprehensive scene observation information, and the output of the UAV decision model is the individual Q function of the UAV.

[0090] (5) Establish a hybrid network based on monotonic function decomposition, input all individual UAV Q functions into the hybrid network, and the hybrid network combines all individual UAV Q functions into the overall UAV value Q function. Train the overall UAV value Q function through the loss function, and obtain the joint flight path that maximizes the throughput of the UAV system and minimizes energy consumption based on the mathematical model of the real-time total power consumption of the UAV group.

[0091] Step (1) specifically refers to: reconstructing the state information of ground users and drones into a two-dimensional scene, and through matrix fitting, denoting the two-dimensional scene as a K*K equidistant grid, with Φ ground users randomly distributed at the center of the equidistant grid at a constant speed ν. φ In a two-dimensional scene, the user moves randomly. The direction of the user's random movement is discretized into up, down, left, right, upper left, lower left, upper right, lower right, and stationary. σ drones fly at a fixed altitude Η above the scene. The drones are numbered u1, u2, ... u σ The drone's flight speed is constant at ν. uav The drone's flight direction is discretized into up, down, left, right, upper left, lower left, upper right, lower right, and stationary. The drone provides communication services to ground users within a circular area of ​​radius D. The entire mission cycle of the drone is discretized into multiple individual time intervals, each with a duration of t. The current system time is T, where T = t1, t2…t end , t end The moment when the UAV mission terminates is represented by the scene matrix containing ground user information at time t, and the matrix containing the location and communication range of each UAV are stacked to obtain the scene observation information of σ UAVs.

[0092] Step (2) specifically refers to: the real-time total power consumption mathematical model of the UAV group includes the UAV flight power consumption model and the communication power consumption model between the UAV and the ground user:

[0093]

[0094] In the formula, This represents the total real-time power consumption of the drone at time t. This represents the flight power consumption of the drone at time t. Let t be the power consumption for communication between the UAV and the ground user. E max The maximum power consumption allowed by the drone's battery capacity;

[0095] The expression for the power consumption model of the UAV flight is as follows:

[0096]

[0097] In the formula, p vIndicates the flight mobility of the drone, ν uav Indicates the flight speed of the drone. This represents the horizontal coordinate projection of the drone in the scene at time t. This represents the horizontal coordinate projection of the drone in the scene at time t-1; This represents the vertical coordinate projection of the drone in the scene at time t. This represents the vertical coordinate projection of the drone in the scene at time t-1;

[0098] The expression for the communication power consumption model between the UAV and ground users is as follows:

[0099]

[0100] In the formula, V r denoted by , e represents the signal transmission rate of the UAV during communication; e represents the transmission efficiency of the communication process; c represents the bandwidth limitation during communication between the UAV and the ground user; d represents the communication distance between the UAV and the ground user, and its mathematical expression is shown below;

[0101]

[0102] In the formula, H represents the height of the drone above the horizontal ground. This represents the x-coordinate of the ground user in the scene at time t. This represents the vertical coordinate of the ground user in the scene at time t.

[0103] Step (3) specifically refers to: the exploration incentive module adopts a Siamese network, which is a convolutional neural network; the input of the exploration incentive module is the scene observation information obtained at the current time t and the past scene observation information stored in the historical scene information pool, and the output is a feature similarity. The triplet loss function is used as the update method of the loss function, and then the output of the exploration incentive module is integrated with the reward signal; the Siamese network of the exploration incentive module includes a shared convolutional layer, a fully connected layer and a metric learning layer:

[0104] Shared convolutional layer: Two input images are processed through the same convolutional layer to extract features and share weights. The goal of this layer is to capture the shared features of the input images. The input dimension is (K,K,2*(σ+1)).

[0105] Fully connected layer: The fully connected layer has 1024 nodes and uses the ReLU function as the activation function;

[0106] Metric learning layer: Calculates the similarity between two input scene observations in the feature space, and uses Euclidean distance or cosine similarity to measure the difference between the two input scene observations;

[0107] The mathematical expression for the triplet loss function is as follows:

[0108] L(A,P,N)=max(||f(A)-f(p)|| 2 -||f(A)-f(N)|| 2 +a, 0)

[0109] In the formula, f(A) is the input of the anchor point sample, which uses the observation of past scene information in the historical scene information pool as the anchor point sample; f(p) is the input of the positive sample, which is defined as the current scene information when the UAV can reach any past scene in the historical scene information pool from the current scene state within k steps; f(N) is the input of the negative sample, which is defined as the current scene information when the UAV cannot reach any past scene in the historical scene information pool from the current scene state within k steps; and a represents a predefined boundary.

[0110] In step (3), the mathematical expression output by the exploration incentive module is:

[0111] F(ω,M)=max(μ1,μ2,…,μ m )

[0112] In the formula, ω represents the state information observation at the current time t, M represents all past scene observation information stored in the historical scene information pool, and m represents the number of past scene observation information contained in the historical scene information pool, (μ1, μ2, ..., μ...). m ) represents the feature similarity between ω and all past scene observation information in M;

[0113] Then, the feature similarity is integrated with the reward signal:

[0114]

[0115] In the formula, R represents the reward value obtained from exploring a new environment. For hyperparameters, F(ω, M) represents the maximum feature similarity between ω and all past scene observation information in M; the higher the feature similarity between the current scene observation information and the historical scene observation information, the lower the reward value; the lower the feature similarity between the current scene observation information and the historical scene observation information, the higher the reward value.

[0116] Step (4) specifically includes the following steps:

[0117] (4a) Define the state space and action space: Define the state space of each UAV as the scene observation information, and define the action space of each UAV as A, the mathematical expression of which is as follows:

[0118] A = (Up, Down, Left, Right, Top Left, Bottom Left, Top Right, Bottom Right, Stationary)

[0119] (4b) Constructing a deep reinforcement learning network: The deep reinforcement learning network includes an input layer, three convolutional layers, one pooling layer, one fully connected layer, and one output layer. The input layer is a fully connected layer with a dimension of (K, K, 2) to receive scene observation information observed by the UAV. The number of nodes in the output layer is equal to the size of the UAV's action space, and the individual Q-function Q of each UAV during training is obtained. i (s t ,a i ), where s t a represents the scene observation information at time t. i This indicates the action chosen by the i-th drone;

[0120] (4c) Define the experience replay buffer and the priority experience replay pool: Create an experience replay buffer Δ to store the previous state information set, joint action set, joint reward value and the next state information set for each UAV. This enables experience replay during the training of the deep reinforcement learning network, breaking the temporal correlation. The mathematical expression for the experience replay buffer Δ is as follows:

[0121] Δ=(s t u t R t ,s t+1 )

[0122] In the formula, u t This indicates that all drones are in state s t The following is a set of combined actions, R t Let s represent the set of joint rewards obtained by all drones at time t. t+1 This indicates that the drone is performing u t The scene observation information at the next moment after arrival;

[0123] A priority experience replay pool is created, with the same structure as the experience replay buffer Δ described above. The priority experience replay pool is used to store the experience of the drone obtaining high-reward rewards in each training round. When the drone learns from the experience replay samples, the experience replay buffer Δ and the priority experience replay pool are combined. Forty percent of the experience is extracted from the experience replay buffer Δ and sixty percent of the experience is extracted from the priority experience replay pool. The mixture is then used as the input of the deep reinforcement learning network.

[0124] (4d) Define Instant Rewards and state transition probability

[0125] Instant rewards This represents the total communication transmission rate between the UAV and n ground users within its sensing range at time t, and the UAV's flight power consumption. The difference is expressed as follows:

[0126]

[0127] In the formula, This represents the flight power consumption of the drone at time t. The communication transmission rate between the drone and the user is expressed as follows:

[0128]

[0129] In the formula: W represents the bandwidth allocated to each ground communication user by the UAV, and SNR is the signal-to-noise ratio, which refers to the ratio of the UAV power to the noise power in the channel. The calculation formula is as follows:

[0130]

[0131] In the formula: Let t be the communication power consumption between the UAV and the ground user at time t, N be additive white Gaussian noise, and L be the average path loss.

[0132] State transition probability State transition probability Indicates that the drone is in state s t Transition to the next state s t+1 The probability of this process is expressed as:

[0133]

[0134] Where η represents the greedy coefficient in the action selection process, a t This indicates the action selected by the drone at time t;

[0135] (4e) Determine the incentive mechanism for the entire mission cycle of the communication system:

[0136] Total reward R obtained by the drone total as follows:

[0137]

[0138] In the formula: Γ represents the time when the task ends, γ represents the discount factor, which is used to balance the proportion of future rewards in the current cumulative rewards. After the drone performs the action, it obtains the state space after the change of the scene environment state and the new reward value.

[0139] The establishment of the hybrid network based on monotonic value function decomposition in step (5) specifically includes the following steps:

[0140] (5a) Input representation: Collect all individual UAV Q functions obtained in step (4) i (s t ,a i );

[0141] (5b) Flattening: Q function of all individual drones Q i (s t ,a i Flatten the structure;

[0142] (5c) Connect: Connect the individual Q functions of all UAVs after flattening to obtain a large vector of joint individual Q functions;

[0143] (5d) Define fully connected layers: Add two fully connected layers to learn the complex relationships between individual Q-functions, and combine the overall Q-function Q. total (s t ,u t Mapped to a higher dimension;

[0144] (5e) Define the activation function: Use the ReLU function to introduce nonlinearity;

[0145] (5f) Define the output layer: Add an output layer to output the overall Q function Q. total (s t ,u t ), where s t u represents the scene observation information at time t. t This indicates that all drones are in state s t The following is a set of combined actions;

[0146] (5g) Define the individual global maximum value constraint that needs to be satisfied:

[0147]

[0148] In the formula, Q i Let s represent the individual Q-function of the i-th UAV; 1 Represents the state space of the first drone, u 1 This indicates the action selected by the first drone, s n Let u represent the state space of the nth drone. n This represents the action selected by the nth drone;

[0149] Hybrid networks need to be constrained by monotonic conditions:

[0150]

[0151] In the formula, σ represents the number of drones;

[0152] (5h) Define the loss function:

[0153]

[0154] Where b represents the number of samples sampled from the experience pool, and θ represents the parameters of the hybrid network. The mathematical expression is as follows:

[0155]

[0156] Where, θ - These are the parameters of the target network, where γ represents the discount factor; u t+1 This indicates the action selected by the drone at time t+1;

[0157] The error of the timing difference is then expressed as:

[0158] TD=(r+γQ total (target))-Q total (evaluate)

[0159] In the formula, Q total (target) represents the state s t+1 At that time, the maximum value that can be obtained from all the actions of the drone; Q total (evaluate) represents the total Q that can be obtained in state s based on the actions output by the deep reinforcement learning network for each drone. total , r represents the joint total reward value obtained by each drone after executing the action output by the deep reinforcement learning network in state s;

[0160] (5i) Backpropagation: Backpropagation is performed using the loss function to update the weight parameters of the individual Q-function and reduce Q. total (target) and Q total The difference between (evaluate) is calculated; steps (5a) to (5i) are repeated until the loss function L(θ) converges.

[0161] In summary, this invention establishes a mathematical model of the real-time total power consumption of a drone group, creating an optimization objective for drone training and effectively solving the problem of collaborative planning and communication among multiple drones under limited power constraints. The invention also establishes an exploration incentive module, enhancing the drones' proactive exploration of the environment in the initial training phase, guiding them to acquire key information more effectively, thus improving the overall training effect. Furthermore, the established priority experience replay pool effectively improves the model's utilization rate of samples. Finally, by training the overall Q-function of multiple drones, this invention optimizes the individual Q-functions of each drone, enabling them to learn better collaborative methods. This helps drones plan optimal paths to reduce energy consumption while better meeting the communication quality requirements of ground users.

Claims

1. A multi-UAV cooperative assisted communication optimization method based on deep reinforcement learning, characterized in that: The method includes the following steps in sequence: (1) Reconstruct the two-dimensional scene from the status information of ground users and UAVs to obtain scene observation information; (2) Construct a mathematical model of the real-time total power consumption of the UAV group to establish optimization objectives for UAV training; (3) Construct an exploration incentive module. The UAV loads the exploration incentive module, and the exploration incentive module outputs the reward value obtained from exploring the new environment. The reward value obtained from exploring the new environment can be used to obtain comprehensive scene observation information. (4) Establish Each drone uses a deep reinforcement learning network with the same structure as its decision model. The input to the drone's decision model is comprehensive scene observation information, and the output of the drone's decision model is the individual drone's Q-function. (5) Establish a hybrid network based on monotonic value function decomposition, input all individual UAV Q functions into the hybrid network, the hybrid network combines all individual UAV Q functions into the overall UAV value Q function, train the overall UAV value Q function through loss function, and obtain the joint flight path that maximizes the UAV system throughput and minimizes energy consumption based on the mathematical model of real-time total power consumption of UAV group. The exploration incentive module employs a Siamese network, which is a convolutional neural network; the input to the exploration incentive module is the current... The scene observation information obtained at each moment and the past scene observation information stored in the historical scene information pool are output as a feature similarity. A triplet loss function is used as the update method for the loss function. Then, the output of the exploration incentive module is integrated with the reward signal. The Siamese network of the exploration incentive module includes shared convolutional layers, fully connected layers, and a metric learning layer. In step (3), the mathematical expression output by the exploration incentive module is: ； In the formula, Indicates the current Observation of state information at any given time This represents all past scene observation information that has been stored in the historical scene information pool. This indicates the amount of past scene observation information contained in the historical scene information pool. express and Feature similarity of all past scene observation information; Then, the feature similarity is integrated with the reward signal: ； In the formula, This represents the reward value gained from exploring new environments. For hyperparameters, ; express and The maximum feature similarity among all past scene observation information; The higher the feature similarity between the current scene observation information and the historical scene observation information, the lower the reward value; the lower the feature similarity between the current scene observation information and the historical scene observation information, the higher the reward value.

2. The multi-UAV cooperative assisted communication optimization method based on deep reinforcement learning according to claim 1, characterized in that: Step (1) specifically refers to: reconstructing the state information of ground users and drones into a two-dimensional scene, and then fitting the matrix to define the two-dimensional scene as follows: An equidistant grid, with randomly distributed [items] at the center of the equidistant grid. Ground users at a constant speed In a two-dimensional scene, the user moves randomly, and the direction of random movement on the ground is discretized into up, down, left, right, upper left, lower left, upper right, lower right, and stationary. The drone was set up at a fixed altitude Flying above the scene, the drone's number is The drone's flight speed is constant. The drone's flight direction is discretely categorized as up, down, left, right, upper left, lower left, upper right, lower right, and stationary; the drone has a radius of... The system provides communication services to ground users within a circular area, discretizing the entire mission cycle of the drone into multiple individual time slots, each with a duration of [duration missing]. The current system time is , The moment when the drone mission terminates; A scene matrix containing real-time ground user information is stacked with a matrix containing the location and communication range of each drone to obtain... Scene observation information from a drone.

3. The multi-UAV cooperative assisted communication optimization method based on deep reinforcement learning according to claim 1, characterized in that: Step (2) specifically refers to: the real-time total power consumption mathematical model of the UAV group includes the UAV flight power consumption model and the communication power consumption model between the UAV and the ground user: ； In the formula, express Real-time total power consumption of the drone express The flight power consumption of the drone at all times. for The power consumption of communication between the drone and ground users at any time. , The maximum power consumption allowed by the drone's battery capacity; The expression for the power consumption model of the UAV flight is as follows: ； In the formula, Indicates the flight mobility of the drone. Indicates the flight speed of the drone. express The horizontal coordinate projection of the drone in the scene. express The horizontal coordinate projection of the drone in the scene at any time; This represents the vertical coordinate projection of the drone in the scene at time t. express The vertical coordinate projection of the drone in the scene at any time; The expression for the communication power consumption model between the UAV and ground users is as follows: ； In the formula, This indicates the rate at which the drone transmits signals during communication. Indicates the transmission efficiency of the communication process; This indicates the bandwidth limitations during communication between the drone and ground users; The mathematical expression for the communication distance between the drone and the ground user is shown below; ； In the formula, Indicates the height of the drone above the horizontal ground. express The horizontal coordinate of the ground user in the scene at any time. express The vertical coordinate of the ground user in the scene at any given time.

4. The multi-UAV cooperative assisted communication optimization method based on deep reinforcement learning according to claim 1, characterized in that: Step (3) specifically refers to: Shared convolutional layer: The two input images are processed through the same convolutional layer for feature extraction to share weights. The goal of this layer is to capture the shared features of the input images, with the input dimension being... ; Fully connected layer: The fully connected layer has 1024 nodes and uses the ReLU function as the activation function; Metric learning layer: Calculates the similarity between two input scene observations in the feature space, and uses Euclidean distance or cosine similarity to measure the difference between the two input scene observations; The mathematical expression for the triplet loss function is as follows: ； In the formula, The input for the anchor point samples uses past scene information observations from the historical scene information pool as anchor point samples. It is a positive sample input, and the drone is in If the current scene information can reach any past scene in the historical scene information pool within one step from the current scene state, the current scene information is defined as a positive sample. It is a negative sample input, and the drone is in If it is impossible to reach any past scene from the historical scene information pool from the current scene state within one step, the current scene information is defined as a negative sample. It represents a predefined boundary.

5. The multi-UAV cooperative assisted communication optimization method based on deep reinforcement learning according to claim 1, characterized in that: Step (4) specifically includes the following steps: (4a) Define the state space and action space: Define the state space of each UAV as the scene observation information, and define the action space of each UAV as... Its mathematical expression is as follows: ； (4b) Constructing a deep reinforcement learning network: The deep reinforcement learning network includes an input layer, three convolutional layers, one pooling layer, one fully connected layer, and one output layer. The input layer is a fully connected layer with dimensions (K, K, 2) and is used to receive scene observation information observed by the UAV. The number of nodes in the output layer is the size of the UAV's action space, and the individual Q-function of each UAV during the training process is obtained. ,in, Indicates the current Scene observation information at any given moment This indicates the action chosen by the i-th drone; (4c) Define the experience replay buffer and priority experience replay pool: Create an experience replay buffer It is used to store the previous state information set, joint action set, joint reward value, and next state information set for each drone, enabling experience replay during the training of deep reinforcement learning networks, breaking temporal correlation, and serving as an experience replay buffer. The mathematical expression is as follows: ； In the formula, This indicates that all drones are in a certain state. The following combined action set, Indicates the current The combined reward set of all drones at any given time. Indicates that the drone is performing The scene observation information that arrives at the next moment after the arrival; Create a priority experience replay pool with the same structure as the experience replay buffer described above. Similarly, a priority experience replay pool is used to store the experience gained by the drone in each training round when it obtains high-reward rewards; an experience replay buffer is used when the drone learns from the experience replay samples. Combined with a priority experience replay pool, from the experience replay buffer Forty percent of the experience is extracted from the middle and sixty percent of the experience is extracted from the priority experience replay pool. The two are then mixed and finally used as the input of the deep reinforcement learning network. (4d) Define instant rewards and state transition probability : Instant rewards express Total communication transmission rate between the drone and n ground users within its sensing range and the drone's flight power consumption The difference is expressed as follows: ； In the formula, express The flight power consumption of the drone at all times. The communication transmission rate between the drone and the user is expressed as follows: ； In the formula: This indicates the bandwidth allocated to each ground communication user by the drone. The signal-to-noise ratio (SNR) refers to the ratio of the drone's power to the noise power in the channel. The calculation formula is as follows: ； In the formula: for The power consumption of communication between the drone and ground users at any time. It is additive white Gaussian noise. This represents the average path loss. State transition probability State transition probability Indicates the drone is in status Transition to the next state The probability of this is expressed as follows: ； in, This represents the greed coefficient during the action selection process. Indicates that drones are in The action selected at any given moment; (4e) Determine the incentive mechanism for the entire mission cycle of the communication system: Total rewards earned by drones as follows: ； In the formula: Indicates the time when the task ends. This represents the discount factor, used to balance the proportion of future rewards in the current cumulative rewards. After the drone performs an action, it obtains the state space after the change in the scene environment state and the new reward value.

6. The multi-UAV cooperative assisted communication optimization method based on deep reinforcement learning according to claim 1, characterized in that: The establishment of the hybrid network based on monotonic value function decomposition in step (5) specifically includes the following steps: (5a) Input representation: Collect the Q-functions of all individual UAVs obtained in step (4) ; (5b) Flattening: All individual drones' Q functions Flatten the design; (5c) Connect: Connect the individual Q functions of all UAVs after flattening to obtain a large vector of joint individual Q functions; (5d) Define fully connected layers: Add two fully connected layers to learn the complex relationships between individual Q functions, and integrate the overall Q function. Mapped to a higher dimension; (5e) Define the activation function: Use the ReLU function to introduce nonlinearity; (5f) Define the output layer: Add an output layer to output the overall Q function. ,in, Indicates the current Scene observation information at any given moment This indicates that all drones are in a certain state. The following is a set of combined actions; (5g) Define the individual global maximum value constraint that needs to be satisfied: ； In the formula, Let Q represent the individual Q-function of the i-th UAV; This represents the state space of the first drone. This indicates the action selected by the first drone. Let n be the state space of the nth drone. This represents the action selected by the nth drone; Hybrid networks need to be constrained by monotonic conditions: ； In the formula, Indicates the number of drones; (5h) Define the loss function: ； in, This indicates the number of samples taken from the experience pool. The parameters representing the hybrid network, The mathematical expression is as follows: ； in, These are the parameters of the target network. Indicates the discount factor; Indicates in The actions selected by the drone at that moment; The error of the timing difference is then expressed as: ； In the formula, Indicates that in the state of At that time, the maximum value that can be obtained from all the actions of the drone; Indicates the state In this case, the total amount that can be obtained based on the actions output by the deep reinforcement learning network of each drone is... , Indicates the state The joint total reward value obtained by each drone after executing the actions output by the deep reinforcement learning network; (5i) Backpropagation: Backpropagation is performed using the loss function to update the weight parameters of the individual Q-function and reduce the loss function. and The difference; repeat steps (5a) to (5i) until the loss function is found. convergence.

Images