Unmanned aerial vehicle communication system strategy optimization method based on deep reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle communication system strategy optimization method based on deep reinforcement learning, and the method comprises the steps: building an unmanned aerial vehicle communication system model which comprises an unmanned aerial vehicle, an RIS and a plurality of ground receiving users, carrying out the optimization of a maximum rate C through an improved TD3 algorithm improvement-TD3 by employing the deep reinforcement learning (DRL) technology under the condition of giving specific channel state information (CSI), and carrying out the optimization of the maximum rate C through the improved TD3 algorithm improvement-TD3. And solving an optimal beam forming matrix and an optimal phase shift matrix. According to the method, entropy regularization is introduced into an existing TD3 algorithm, so that exploratory performance is enhanced, an intelligent agent is promoted to explore more environments, the relation between exploration and utilization is balanced, finally, an optimal transmitting beam forming matrix and RIS phase shift are obtained, falling into a local optimal solution is avoided, and learning robustness and generalization ability are improved.

Description

Technical Field

[0001] The present invention relates to the fields of unmanned aerial vehicle communication and deep reinforcement learning, and in particular to a strategy optimization method for an unmanned aerial vehicle communication system based on deep reinforcement learning. Background Art

[0003] As an emerging technology, drone communication has shown great potential and application prospects in many fields. Drone communication uses drones as aerial base stations or communication relay stations to provide connection services for ground users. Its application areas cover military, civil, commercial and other aspects, providing new solutions for the development and security of modern society.

[0003] In recent years, UAV technology based on intelligent reflective surfaces (RIS) has also attracted widespread attention. As a low-cost and easy-to-install technology, RIS technology is used to mitigate interference and transmit the transmitted signal directly to the receiver. By optimizing the configuration of RIS and the communication strategy of the UAV, researchers have enhanced the reception signal for ground users, while reducing power consumption and extending the flight time of the UAV. At the same time, reinforcement learning algorithm (DRL) has also been widely used in UAV technology in recent years as one of the powerful tools to solve the problem of flexible adaptation to environmental changes and real-life constraints in wireless networks. Researchers have maximized the communication performance of UAVs using the DRL algorithm by optimizing strategies such as power allocation and phase shift optimization.

[0004] However, existing deep reinforcement learning methods have the following shortcomings:

[0005] 1. A key problem of the deep deterministic policy gradient method DDPG (Deep Deterministic Policy Gradient) is the overestimation problem, which mainly comes from two aspects: white suggestion and maximization of overestimation.

[0006] 2. Twin Delayed Deep Deterministic Policy Gradient TD3 uses delayed updates to improve learning stability and convergence speed. However, if the Critic network is not stable enough, the Actor network will be easily disturbed and oscillate.

[0007] 3. Lack of exploration, easy to fall into the local optimal solution. Summary of the invention

[0008] The purpose of this invention is to provide a UAV communication system strategy optimization method based on deep reinforcement learning. Entropy regularization is introduced into the existing TD3 algorithm to enhance exploration, prompt the intelligent agent to explore the environment more, balance the relationship between exploration and utilization, avoid falling into the local optimal solution, and improve the robustness and generalization ability of learning.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is: a UAV communication system strategy optimization method based on deep reinforcement learning, comprising the following steps: S1. Establish a UAV communication system model, including a UAV, an intelligent reflector RIS, and multiple ground receiving users, where the UAV has M antennas, the RIS has N reflective elements, and multiple users have K receiving terminals in total and all have single antennas; S2. Let the channel matrix from the UAV to the RIS be The channel vector from RIS to K receivers is The received signal of the user in the nth time slot is: In formula (1), s is a random variable with zero mean and unit variance, ε is a random variable with zero mean and unit variance, 2 The additive Gaussian white noise, G is the beamforming matrix of the UAV, Φ is the phase shift matrix of the UAV in the nth time slot; Φ=diag[φ1,φ2,...,φ n ],∈C (n×n) ; ρ n ∈[0,1] is the amplitude reflection coefficient, is an imaginary unit, is the phase shift of the nth reflection unit; The signal-to-noise ratio of the kth user is: The average throughput of the kth receiver in time slot t is: Γ k (t) = (1-T(t))Blog2(1+SNR k ), k∈K,t∈T (3); In formula (3), B is the bandwidth; S3. Take the total rate C as the standard for testing system performance: In formula (4), R k is the data rate of the kth user; The optimal G and Φ are obtained through the improved TD3 algorithm improvement-TD3. Improvement-TD3 optimizes the problem of maximizing the rate C to: In formula (5), Γ k(t) is the user throughput, Γ min is the minimum throughput of the user, P t is the total transmission power of the UAV, is the phase shift matrix of RIS; C1 constrains the minimum throughput of each receiver, C2 constrains the maximum power of the UAV, and C3 and C4 constrain each reflector element n in the RIS to only provide a phase shift without amplifying the input signal. Set the reward function to: In formula (6), is the variance of additive Gaussian white noise; The improvement-TD3 algorithm runs multiple rounds, each round iterates T steps. In the tth step, given the instantaneous channel H u,r 、h r,k and the action G obtained from the Actor network (t) and Φ (t) , and the total rate R max As a reward; For each round, when the maximum number of steps allowed is reached and the reward function does not reach the threshold, the algorithm is updated. At each time t, the Actor network generates an action a based on the current state and the exploration noise ε, and then obtains an instantaneous reward r after the algorithm executes the corresponding action, and then stores the tuple (s, a, r, s', d) in the experience replay buffer; Then, multiple N samples are sampled from the experience replay buffer and the error between the evaluation value and the target value is minimized using the gradient descent algorithm. Update the Critic network: Update the Actor network: a←π φ' (s')+ε (8); In formula (8), ε is the noise term; The entropy in the improvement-TD3 algorithm is expressed as: H = -∫p(a)logp(a)da (9); In formula (9), p(a) is the probability density function of the action, expressed as: Then, entropy regularization is introduced into the deterministic policy gradient method, and the policy function is expressed as: In formula (11), Z is the normalization constant, and α is a parameter used to control the degree of influence of entropy regularization; Finally, perform a soft update on the target network: In formula (12), τ is the learning rate, τ∈(0,1); After the target network is updated, the next round of iteration of the improvement-TD3 algorithm is performed until the reward function reaches the threshold and the algorithm terminates to obtain the optimal G opt and Φ opt As the action with the best immediate reward.

[0010] Preferably, improvement-TD3 initializes the experience replay buffer, beamforming matrix G, phase shift matrix Φ, actor network and critic network at the beginning of the algorithm, where π of the actor network is φ Initialized with random parameters φ, the critic network Initialized with random parameters δ1, δ2.

[0011] Compared with the prior art, the present invention has the following beneficial effects: The present invention is used in a RIS-assisted UAV communication system and can solve large-scale optimization problems. Compared with other existing deep reinforcement learning algorithms, the present invention enhances the exploratory nature of the algorithm on the basis of a double-delay deep deterministic policy gradient algorithm, thereby improving the robustness and generalization ability of learning, and ultimately obtaining the optimal transmit beamforming matrix and RIS phase shift. DETAILED DESCRIPTION

[0012] The embodiment provides a UAV communication system strategy optimization method based on deep reinforcement learning, comprising the following steps: S1. Establish a UAV communication system model, including a UAV, an intelligent reflector RIS, and multiple ground receiving users. The UAV has M antennas, the RIS has N reflective elements, and multiple users have K receiving terminals in total and all have single antennas.

[0013] Drones act as aerial base stations to provide downlink transmission services for multiple devices on the ground. Since the signal will attenuate quickly, we equip the drone with multiple antennas for beamforming to compensate for the attenuation of the signal. When there is a straight, unobstructed propagation path between the drone and the user, the reliability and efficiency of the transmission are high. However, if this path is blocked by obstacles, the quality of the signal received by the user will be significantly reduced. In order to improve the received power, a reconfigurable intelligent surface (RIS) can be deployed near the user to assist in signal transmission. RIS reflects the drone's signal to the user, thereby replacing the original direct non-linear transmission path with two continuous straight transmission paths, improving the quality of the received signal.

[0014] S2. Let the channel matrix from the UAV to the RIS be The channel vector from RIS to K receivers is The received signal of the user in the nth time slot is: In formula (1), s is a random variable with zero mean and unit variance, ε is a random variable with zero mean and unit variance, 2 The additive Gaussian white noise, G is the beamforming matrix of the UAV, Φ is the phase shift matrix of the UAV in the nth time slot; Φ=diag[φ1,φ2,...,φ n ],∈C (n×n) ; ρ n ∈[0,1] is the amplitude reflection coefficient, is an imaginary unit, is the phase shift of the nth reflection unit.

[0015] The signal-to-noise ratio of the kth user is:

[0016] According to Shannon's formula, the average throughput of the kth receiver in time slot t is: Γ k (t) = (1-T(t))Blog2(1+SNR k ),k∈K,t∈T (3); In formula (3), B is the bandwidth.

[0017] S3. Take the total rate C as the standard for testing system performance: In formula (4), R kis the data rate of the kth user.

[0018] The optimal G and Φ are obtained by the improved TD3 algorithm improvement-TD3. The goal is to use deep reinforcement learning (DRL) technology to find the optimal G and Φ by maximizing the rate C under a given specific CSI. Unlike traditional deep neural networks (DNNs), there is no need for offline training and online learning. The proposed DRL method uses one CSI to construct a state each time and runs the algorithm continuously to obtain the optimal G and Φ matrices.

[0019] Improvement-TD3 optimizes the problem of solving the maximum rate C to: In formula (5), Γ k(t) is the user throughput, Γ min is the minimum throughput of the user, P t is the total transmission power of the UAV, is the phase shift matrix of RIS; C1 constrains the minimum throughput of each receiver, C2 constrains the maximum power of the UAV, and C3 and C4 constrain each reflector element n in the RIS to only provide a phase shift without amplifying the input signal.

[0020] The goal is to optimize the efficiency and performance of the UAV communication system. In order to maximize the total rate over N time slots while ensuring the QoS requirements of each user, the reward function is set as: In formula (6), is the variance of additive white Gaussian noise.

[0021] The improvement-TD3 algorithm runs multiple rounds, each round iterates T steps. In the tth step, given the instantaneous channel H u,r 、h r,k and the action G obtained from the Actor network (t) and Φ (t) , and the total rate R max As a reward. At the beginning of the algorithm, the experience replay buffer, beamforming matrix G, phase shift matrix Φ, actor network and critic network are initialized, where the actor network's π φ Initialized with random parameters φ, the critic network Initialized with random parameters δ1, δ2.

[0022] For each round, when the maximum number of steps allowed is reached and the reward function does not reach the threshold, the algorithm is updated. At each time t, the Actor network generates an action a based on the current state and the exploration noise ε, and then obtains an instantaneous reward r after the algorithm executes the corresponding action. The tuple (s, a, r, s', d) is then stored in the experience replay buffer.

[0023] Then, multiple N samples are sampled from the experience replay buffer and the error between the evaluation value and the target value is minimized using the gradient descent algorithm. Update the Critic network: Update the Actor network: a←π φ' (s')+ε (8); In formula (8), ε is the noise term.

[0024] The entropy in the improvement-TD3 algorithm is expressed as: H = -∫p(a)logp(a)da (9); In formula (9), p(a) is the probability density function of the action, expressed as:

[0025] Then, entropy regularization is introduced into the deterministic policy gradient method, and the policy function is expressed as: In formula (11), Z is the normalization constant, and α is a parameter used to control the degree of influence of entropy regularization;

[0026] Finally, perform a soft update on the target network: In formula (12), τ is the learning rate, τ∈(0,1).

[0027] After the target network is updated, the next round of iteration of the improvement-TD3 algorithm is performed until the reward function reaches the threshold and the algorithm terminates to obtain the optimal G opt and Φ opt As the action with the best immediate reward.

[0028] The performance of the embodiment is evaluated by simulation using Python. First, the improvement-TD3 algorithm is compared with two other DRL algorithms, namely the deep deterministic policy gradient method DDPG (Deep Deterministic Policy Gradient) and the twin delayed deep deterministic policy gradient TD3 (Twin Delayed Deep Deterministic Policy Gradient). The three algorithms are compared in the same environment, which is a continuous action space provided by OpenAI Gym.

[0029] Comparing the average rewards of the improvement-TD3 algorithm and the DDPG algorithm after 1000 rounds of training, the DDPG algorithm tends to converge at about 400 steps, while the improvement-TD3 algorithm tends to converge at 200 steps, and its performance is relatively stable in the later stage, which is a significant improvement compared to the DDPG algorithm.

[0030] Comparing the average rewards of the improvement-TD3 algorithm and the TD3 algorithm, the improvement-TD3 algorithm showed a good trend in the early stage, with an average reward much higher than that of TD3. The improvement-TD3 algorithm reached convergence in about 300 rounds and performed steadily in the later stage, while the TD3 algorithm reached stability in about 500 rounds. The improvement-TD3 algorithm has a lot of performance improvements compared to the TD3 algorithm.

[0031] In summary, the present invention is applied to the scenario of RIS-assisted UAV communication, using the proposed algorithm to jointly design the transmit beamforming at the UAV and the phase shift of RIS to maximize the sum rate of the multi-user MISO system. The proposed algorithm adds some content on the basis of existing deep reinforcement learning, has a very standard form and low implementation complexity, so it is very easy to expand it to adapt to various system settings. In addition, the exploratory nature of the algorithm has been greatly improved, which can effectively improve the robustness and generalization ability of learning in the process of interaction with the environment.

Claims

1. A UAV communication system strategy optimization method based on deep reinforcement learning, characterized in that: The following steps are involved: S1. Establish a UAV communication system model, including a UAV, an intelligent reflector RIS, and multiple ground receiving users, where the UAV has M antennas, the RIS has N reflective elements, and multiple users have K receiving terminals in total and all have single antennas; S2. Let the channel matrix from the drone to the RIS be H u,r ∈￡ (M×N) , the channel vector from RIS to K receivers is h r,k ∈￡ (N×1) , the received signal of the user in the nth time slot is: In formula (1), s is a random variable with zero mean and unit variance, ε is a random variable with zero mean and unit variance, 2 The additive Gaussian white noise, G is the beamforming matrix of the UAV, G∈￡ (M×K) , Φ is the phase shift matrix of the UAV in the nth time slot; Φ=diag[φ1,φ2,...,φ n ],∈C (n×n) ; is the amplitude reflection coefficient, is an imaginary unit, is the phase shift of the nth reflection unit; The signal-to-noise ratio of the kth user is: The average throughput of the kth receiver in time slot t is: Γ k (t)=(1-T(t))Blog2(1+SNR k ), k∈K,t∈T (3); In formula (3), B is the bandwidth; S3. Take the total rate C as the standard for testing system performance: In formula (4), R k is the data rate of the kth user; The optimal G and Φ are obtained through the improved TD3 algorithm improvement-TD3. Improvement-TD3 optimizes the problem of maximizing the rate C to: In formula (5), Γ k(t) is the user throughput, Γ min is the minimum throughput of the user, P t is the total transmission power of the UAV, is the phase shift matrix of RIS; C1 constrains the minimum throughput of each receiver, C2 constrains the maximum power of the UAV, and C3 and C4 constrain each reflector element n in the RIS to only provide a phase shift without amplifying the input signal. Set the reward function to: In formula (6), is the variance of additive Gaussian white noise; The improvement-TD3 algorithm runs multiple rounds, each round iterates T steps. In the tth step, given the instantaneous channel H u,r 、h r,k and the action G obtained from the Actor network (t) and Φ (t) , and the total rate R max As a reward; For each round, when the maximum number of steps allowed is reached and the reward function does not reach the threshold, the algorithm is updated. At each time t, the Actor network generates an action a based on the current state and the exploration noise ε, and then obtains an instantaneous reward r after the algorithm executes the corresponding action, and then stores the tuple (s, a, r, s', d) in the experience replay buffer; Then, multiple N samples are sampled from the experience replay buffer and the error between the evaluation value and the target value is minimized using the gradient descent algorithm. Update the Critic network: Update the Actor network: a←π φ' (s')+ε(8); In formula (8), ε is the noise term; The entropy in the improvement-TD3 algorithm is expressed as: H = -∫p(a)logp(a)da (9); In formula (9), p(a) is the probability density function of the action, expressed as: Then, entropy regularization is introduced into the deterministic policy gradient method, and the policy function is expressed as: In formula (11), Z is the normalization constant, and α is a parameter used to control the degree of influence of entropy regularization; Finally, perform a soft update on the target network: In formula (12), τ is the learning rate, τ∈(0,1); After the target network is updated, the next round of iteration of the improvement-TD3 algorithm is performed until the reward function reaches the threshold and the algorithm terminates to obtain the optimal G opt and Φ opt As the action with the best immediate reward.

2. The method for optimizing the strategy of a UAV communication system based on deep reinforcement learning according to claim 1, characterized in that: At the beginning of the algorithm, improvement-TD3 initializes the experience replay buffer, beamforming matrix G, phase shift matrix Φ, actor network and critic network, where the π of the actor network φ Initialized with random parameters φ, the critic network Initialized with random parameters δ1, δ2.

Images