RIS-UAV network energy efficiency optimization method and device based on reinforcement learning

Abstract

The invention provides an RIS-UAV network energy efficiency optimization method and device based on reinforcement learning, and relates to the technical field of mobile communication. Comprising the following steps: determining data rates of all time slots of all users according to a Rayleigh channel from an unmanned aerial vehicle to each user and a diagonal reflection phase matrix of an RIS; constructing an optimization problem according to the data rates of all the time slots of all the users, the energy consumption of the base station and the energy consumption of the unmanned aerial vehicle; grouping all the users according to the correlation coefficient channels among the users and a preset threshold to obtain a plurality of user groups; according to the plurality of user groups, the optimization problem and the constraint parameter, determining an optimization problem containing a constraint condition; and according to the diagonal reflection phase matrix of the optimal RIS, a DDQN model in reinforcement learning is used to solve an optimization problem containing constraint conditions, an optimal target ratio is obtained, and an optimal unmanned aerial vehicle flight path is realized according to the optimal target ratio. In this way, the performance of the communication system can be improved, and the energy efficiency of the RIS-UAV can be improved.

Description

Technical Field

[0001] The present invention relates to the field of mobile communication technology, and in particular to a method and device for optimizing the network energy efficiency of a Reconfigurable Intelligent Surface-Unmanned Aerial Vehicle (RIS-UAV) based on reinforcement learning. Background Art

[0002] Drones are easy to deploy, highly maneuverable and low in cost, making drone-assisted communication a key role in the next generation of wireless communication systems, providing broader and deeper ubiquitous connections, and can be widely used in various communication scenarios to improve the quality of wireless networks. In addition, drones have flexible flight capabilities, and signal coverage in remote and special areas can be easily met through drones. Drone-assisted communication can play an important role in many application scenarios and provide people with better communication network services. When drone-assisted wireless network systems are operating, drones can generally be quickly deployed to the target area and establish line-of-sight links with ground communication terminals for communication. However, drone-assisted communication still faces serious challenges in practical applications. The size, weight and limited power supply of drones limit their working time in the air, and flight propulsion consumes most of the power. How to plan and design the flight trajectory of drones when they are working and maximize the system energy efficiency while ensuring the user communication rate is an important problem that needs to be solved in drone-assisted wireless network systems.

[0003] At present, the IRS-UAV-assisted downlink communication system is usually used to optimize the system energy efficiency by optimizing the beamforming at the base station and the reflection phase shift of the IRS. However, the IRS-UAV-assisted downlink communication system is a fixed deployment RIS, which limits the energy consumption and flight range of the drone, thereby reducing the energy efficiency of the IRS-UAV; in the IRS-UAV-assisted downlink communication system, there are situations where the antenna layout is not ideal or the propagation environment lacks sufficient scattering. In this case, the correlation between the channel vectors will reduce the performance of the communication system. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a RIS-UAV network energy efficiency optimization method and device based on reinforcement learning, so as to solve the problem that the prior art reduces the energy efficiency of IRS-UAV and reduces the performance of the communication system.

[0005] To solve the above technical problems, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of the present invention provides a RIS-UAV network energy efficiency optimization method based on reinforcement learning, and the RIS-UAV network energy efficiency optimization method based on reinforcement learning includes:

[0007] According to the Rayleigh channel from the UAV to each user and the diagonal reflection phase matrix of RIS, the data rate of all time slots of all users is determined;

[0008] Construct an optimization problem based on the data rates of all users in all time slots, the energy consumption of the base station, and the energy consumption of the drone;

[0009] According to the correlation coefficient channels and preset thresholds between users, all users are grouped to obtain multiple user groups;

[0010] Determine an optimization problem including constraints according to multiple user groups, optimization problems and constraint parameters;

[0011] According to the diagonal reflection phase matrix of the optimal RIS, the DDQN model in reinforcement learning is used to solve the optimization problem with constraints and obtain the optimal target ratio to achieve the optimal UAV flight trajectory according to the optimal target ratio. The optimal target ratio is the optimal ratio of the data rate of all time slots of all users in multiple user groups to the energy consumption of the UAV and the energy consumption of the base station.

[0012] A second aspect of the present invention provides a RIS-UAV network energy efficiency optimization device based on reinforcement learning, and the RIS-UAV network energy efficiency optimization device based on reinforcement learning includes:

[0013] The first determination module is used to determine the data rates of all time slots of all users according to the Rayleigh channel from the UAV to each user and the diagonal reflection phase matrix of the RIS;

[0014] A building module for constructing an optimization problem based on the data rates of all users in all time slots, the energy consumption of the base station, and the energy consumption of the drone;

[0015] A grouping module is used to group all users according to the correlation coefficient channels and preset thresholds between users to obtain multiple user groups;

[0016] A second determination module is used to determine the optimization problem including constraint conditions according to the multiple user groups, the optimization problem and the constraint parameters;

[0017] The solving module is used to solve the optimization problem containing constraints according to the diagonal reflection phase matrix of the optimal RIS by using the DDQN model in reinforcement learning to obtain the optimal target ratio, so as to achieve the optimal UAV flight trajectory according to the optimal target ratio. The optimal target ratio is the optimal ratio of the data rate of all time slots of all users in multiple user groups to the energy consumption of the UAV and the energy consumption of the base station.

[0018] Compared with the prior art, the RIS-UAV network energy efficiency optimization method and device based on reinforcement learning provided by the present invention determine the data rate of all time slots of all users according to the Rayleigh channel from the drone to each user and the diagonal reflection phase matrix of the RIS; construct an optimization problem according to the data rate of all time slots of all users, the energy consumption of the base station and the energy consumption of the drone; group all users according to the correlation coefficient channel and the preset threshold between users to obtain multiple user groups; determine the optimization problem containing constraints according to the multiple user groups, the optimization problem and the constraint parameters; according to the diagonal reflection phase matrix of the optimal RIS, use the DDQN model in reinforcement learning to solve the optimization problem containing constraints, and obtain the optimal target ratio, so as to achieve the optimal drone flight trajectory according to the optimal target ratio, and the optimal target ratio is the optimal ratio of the data rate of all time slots of all users in the multiple user groups to the energy consumption of the drone and the energy consumption of the base station. In this way, by grouping all users, the impact of user correlation is reduced, so as to reduce the correlation impact of the correlation coefficient channel between users and improve the performance of the communication system; by grouping all users, according to the diagonal reflection phase matrix of the optimal RIS, the DDQN model in reinforcement learning is used to solve the optimization problem containing constraints and obtain the optimal target ratio. According to the optimal target ratio, the optimal UAV flight trajectory can be achieved, and the optimal UAV flight trajectory can improve the energy efficiency of IRS-UAV. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0020] Figure 1 The schematic diagram of the downlink transmission system is schematically shown;

[0021] Figure 2 The flowchart of the RIS-UAV network energy efficiency optimization method based on reinforcement learning is schematically shown;

[0022] Figure 3 A performance convergence comparison diagram is schematically shown;

[0023] Figure 4 Schematic diagram showing the rewards and energy of different RIS elements;

[0024] Figure 5 A diagram schematically showing a comparison of rewards and energies for different numbers of base station antennas;

[0025] Figure 6 Schematic diagram showing different relative levels of component spacing in RIS;

[0026] Figure 7 Schematic diagram showing the reward and date rate for three systems with different numbers of users;

[0027] Figure 8 The energy efficiency comparison diagram under three different grouping situations is schematically shown;

[0028] Fig. 9 The correlation coefficients of the user groups corresponding to the three indicators varying from 0 to 1 are schematically shown;

[0029] Fig.10 The structure diagram of the RIS-UAV network energy efficiency optimization device based on reinforcement learning is schematically shown. DETAILED DESCRIPTION

[0030] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0031] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present invention should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0032] Figure 1 The schematic diagram of the downlink transmission system is schematically shown. The RIS-UAV network energy efficiency optimization method based on reinforcement learning of the present invention is applied to the downlink transmission system. The downlink transmission system includes a base station (BS), a RIS-UAV and multiple users. The BS is equipped with M antennas and serves K single-antenna user terminals, i.e., users. The RIS mounted on the drone is composed of R low-cost and nearly passive reflection array elements. Assume that the direct link between the BS and the user is completely blocked, so each user can only receive the signal reflected by the same RIS. Assume that the location coordinates of the base station are (x bs ,y bs ,z bs ), the position coordinates of the drone are (x u ,y u ,z u ), the location coordinates of the kth user are (x k ,y k ,z k ), k∈K, the distance between the base station and the drone The distance between the drone and the kth user

[0033] Assume that there is a Rician channel between BS and UAV represents the transpose of the Rayleigh channel from the UAV to the kth user. For each reflective element of the RIS, there is a discrete adjustable phase. Let Φ = diag{φ1,…,φ r ,…,φ R} is the diagonal reflection phase matrix of RIS, where The precoding vector from the base station to the kth user is is the beamforming matrix The kth column of x k is the transmission symbol transmitted to the kth user, E{x k 2}=1;n k is the additive Gaussian white noise at the kth user The noise variance is σ 2 . Where K is the number of users, is an N×M complex space, is a 1×N matrix, φ r is the phase value of the rth element in the diagonal reflection phase matrix of RIS, θ r is the reflection phase shift value of the RIS array element in the nth time slot, i is the set of phase values ​​of the rth element in the diagonal reflection phase matrix of the RIS, λ is the wavelength, is an M×1 complex space, is an M×K complex space, E{|x k | 2} is x k The mathematical expectation of the square of the absolute value of .

[0034] The method in the embodiment of the present invention is described in detail below.

[0035] Figure 2 The flowchart of the RIS-UAV network energy efficiency optimization method based on reinforcement learning in the embodiment of the present invention is schematically shown. Figure 2 As shown, the method may include:

[0036] S201. Determine the data rates of all time slots of all users according to the Rayleigh channel from the UAV to each user and the diagonal reflection phase matrix of the RIS.

[0037] Before determining the data rate of all time slots for all users based on the Rayleigh channel from the drone to each user and the diagonal reflection phase matrix of the RIS, first, the path loss model must be determined. The specific operations are as follows: the impact of the environment on the line of sight and the air-to-ground channel model in the urban environment need to be considered. The path loss calculation between the drone and the base station and the user adopts the probabilistic path loss model. The LoS connectivity probability between the drone and the kth target user at time slot t is for:

[0038]

[0039] Where a is a first constant value that depends on the environment and b is a second constant value that depends on the environment, z u is the drone altitude, is the horizontal distance between the UAV and the user. The path loss of the air-to-ground link depends on the vertical height of the UAV and the horizontal distance. Specifically, the path loss from the UAV to the user is expressed as A is the first constant and C is the second constant, which gives A = ηLOS - ηNLOS, f c is the carrier frequency (Hz), c is the speed of light (m / s), z u [n] is the coordinate of the UAV on the z-axis of the three-dimensional coordinate system in the nth time slot, η is the efficiency, which is used to measure the ratio of line-of-sight to non-line-of-sight, ηLOS and ηNLOS (unit: dB) are the losses corresponding to line-of-sight connection (LoS) and non-line-of-sight connection (Non-LoS, NLoS) respectively depending on the environment, then PL UK is the path loss from the UAV to the user.

[0040] Similarly, the path loss from the base station to the drone is:

[0041]

[0042]

[0043] in, is the LoS connectivity probability from the base station to the drone at time slot t, a is the first constant value and b is the second constant value depending on the environment, z u is the altitude of the drone, l BU is the horizontal distance between the base station and the drone, A is the first constant, C is the second constant, PL BU is the path loss from BS to UAV.

[0044] Secondly, the BS-RIS channel needs to be determined. The specific operation is as follows: Assuming that the channel from BS to UAV is a Rician channel, the channel from BS to UAV, i.e., the Rician channel, can be expressed as: Among them, PL BU is the path loss from BS to UAV, β is the Rice factor, H LOS represents the LoS component, a RIS represents the steering vector at RIS, represents the angle of arrival (AoA) at RIS, j represents an imaginary unit, a BS is the steering vector at BS, It represents the Angle of Departure (AoD) at the BS. Each NLoS component obeys a complex normal distribution with a mean of zero and a variance of 1.

[0045] Third, the RIS-user channel needs to be determined. The specific operation is as follows: The RIS-user channel is modeled as a correlated Rayleigh fading channel, where the fading is correlated on the RIS side. Specifically, the channel between the RIS and the kth user is modeled as h k :

[0046]

[0047] in, represents the distance and path loss index between RIS and the kth user. Each steering vector corresponds to a specific direction of departure (DoD). k is a Rayleigh fading channel vector with independent and identically distributed complex Gaussian components with zero mean and unit variance. k It is included with D k The transmit steering matrix of the steering vectors of the transmit antenna array corresponding to the directions. represents the i2th DOD of the kth user, i2∈1,…,D, where D is the maximum distance that the UAV can move in a time slot, is the i2th DOD of the kth user, and N is the number of time slots.

[0048] Finally, it is necessary to determine the user receiving signal. The specific operation is: the received signal y of the kth user in the nth time slot k for:

[0049]

[0050] Among them, the first is the expected signal of the kth user, and the second term The interference of other users' signals to the user. is the transpose of the Rayleigh channel from the UAV to the kth user, Φ is the diagonal reflection phase matrix of RIS, H[n] is the Ricean channel between the base station and the UAV, w k is the precoding vector from the base station to the kth user, w j′ is the precoding of the j′th user, K is the number of users, j′ is any one of the K users, x j′ is the transmission symbol transmitted to the j′th user.

[0051] Specifically, according to the Rayleigh channel from the UAV to each user and the diagonal reflection phase matrix of the RIS, the data rates of all time slots of all users are determined, including:

[0052] Step A1: Determine the signal-to-interference-noise ratio based on the Rayleigh channel from the UAV to each user and the diagonal reflection phase matrix of the RIS.

[0053] Specifically, the signal-to-interference-noise ratio is determined based on the Rayleigh channel from the drone to each user and the diagonal reflection phase matrix of the RIS, including:

[0054] According to the Rayleigh channel from the drone to each user and the diagonal reflection phase matrix of the RIS, the signal-to-interference-to-noise ratio is determined using the following first formula:

[0055]

[0056] Among them, SINR k [n] is the signal-to-interference-noise ratio of the kth user in the nth time slot, is the transpose of the Rayleigh channel from the UAV to the kth user, Φ is the diagonal reflection phase matrix of RIS, H[n] is the Ricean channel between the base station and the UAV, w k is the precoding vector from the base station to the kth user, is an M×1 complex space, w j′ is the precoding vector of the j′th user, σ 2 is the noise variance, K is the number of users, and j′ is any one of the K users.

[0057] Step A2: Determine the data rates of all time slots of all users according to the signal to interference and noise ratio.

[0058] Specifically, according to the signal to interference noise ratio, the data rates of all time slots of all users are determined, including:

[0059] According to the signal to interference noise ratio, the data rate of all time slots of all users is determined using the following second formula:

[0060]

[0061] Among them, R sum is the data rate of all users in all time slots, SINR k [n] is the signal to interference and noise ratio of the kth user in the nth time slot, k is the kth user, K is the number of users, and N is the number of time slots.

[0062] S202: Construct an optimization problem based on the data rate of all time slots of all users, the energy consumption of the base station and the energy consumption of the drone.

[0063] Before constructing the optimization problem based on the data rate of all time slots of all users, the energy consumption of the base station and the energy consumption of the drone, it is necessary to determine the energy consumption model. The energy consumption model includes the base station energy consumption model and the drone energy consumption model. The specific operation of determining the base station energy consumption model is: Assume that a base station with a transmission power of P is transmitting data, and the energy consumed when the transmission time is t in the nth time slot, that is, the energy consumption of the base station is

[0064] The specific operation of determining the UAV energy consumption model is as follows: Assume that the UAV flies from position L0 to L in a limited time span T. F For tracking purposes, T is divided into n time slots, i.e., T = N × t′, where t′ is the length of each time slot. Assuming that the flight range of the drone is limited, the position L of the drone at the nth time slot is n Indicated as L n =(x u [n],y u [n],z u [n]),n=1,…,N,x u [n] is the coordinate of the drone on the x-axis of the three-dimensional coordinate system in the nth time slot, y u [n] is the coordinate of the drone on the y-axis of the three-dimensional coordinate system in the nth time slot, z u [n] is the coordinate of the UAV on the z-axis of the three-dimensional coordinate system in the nth time slot.

[0065] The flight trajectory sequence of the UAV is represented as {L0,L1,...,L N ,L F}, L0 is the starting position of the UAV flight trajectory, L F is the final position of the UAV flight trajectory. The UAV flies in a limited area, and the distance moved between two time slots is constrained by the constraint condition ‖L n+1 -L n ‖ 2 ≤D 2 ,n=1,…,N-1,L n is the position of the UAV flight trajectory in the nth time slot, Ln+1 is the position of the UAV flight trajectory in the n+1th time slot, D = V max ×t, D is the maximum distance the drone can move in a time slot, V max It is the maximum flight speed of the UAV in a time slot.

[0066] The vertical flight speed of the drone in the nth time slot Expressed as The horizontal flight speed of the drone in the nth time slot It can be specifically expressed as z u [n+1] is the coordinate of the UAV on the z-axis of the three-dimensional coordinate system in the n+1th time slot, x u [n+1] is the coordinate of the drone on the x-axis of the three-dimensional coordinate system in the n+1th time slot, y u [n+1] is the coordinate of the drone on the y-axis of the three-dimensional coordinate system in the n+1th time slot.

[0067] if and If both are equal to 0, it means that the drone will be in a hovering state in the nth time slot. Since the power capacity of the drone is limited, its flight is limited by energy consumption. By calculating the flight speed in the vertical and horizontal directions, the flight power P of the drone can be calculated. uav for:

[0068] P uav =P a +P b +P c +P d ;

[0069]

[0070] Among them, P0 is the power consumed by the blade profile in the hovering state, P1 is the blade profile induced power in the hovering state, P2 is the power consumed when the drone rises or descends vertically, V tip is the tip rotation speed of the rotor blade, V0 is the rotation speed of the rotor in hovering, d0 represents the fuselage drag ratio, ρ is the air density, s is the rotor firmness, A uav represents the rotor disc area, P a is the power consumption of the UAV in horizontal flight, P b is the power consumption related to the induced drag of the UAV in the horizontal direction, P c is the power consumption related to the additional power consumption of the UAV in the horizontal direction, P d is the power consumption related to the UAV’s speed in the vertical direction.

[0071] The energy consumption of the drone flying in the nth time slot, that is, the energy consumption of the drone is expressed as P uav is the transmission power of the UAV, and t is the transmission time.

[0072] Specifically, based on the data rates of all time slots of all users, the energy consumption of base stations and the energy consumption of drones, an optimization problem is constructed, including:

[0073]

[0074] in, For the optimization problem, R sum is the data rate of all time slots for all users, is the energy consumption of the UAV in the nth time slot, is the energy consumption of the base station in the nth time slot, k is the kth user, K is the number of users, N is the number of time slots, L is the position of the UAV flight trajectory, Φ is the diagonal reflection phase matrix of RIS, and U k are all user groups containing k users.

[0075] S203: Group all users according to the correlation coefficient channels between users and a preset threshold to obtain multiple user groups.

[0076] In spatially correlated channels, the mutual interference between users may be strong. To address this difficulty, user grouping is introduced to divide users into different groups so that the channel correlation coefficient of users in the same group is below a preset threshold. Generally, if the spatial channels of multiple users are highly correlated, the interference between them will increase, which will reduce the signal quality and thus affect the overall performance of the system. Therefore, user grouping is used to deal with the channel correlation problem. The proposed user grouping method consists of two parts. The first dynamic grouping and the second static grouping. Dynamic grouping is a process of organizing users, grouping the channel conditions of that particular time slot in each time period according to the needs at that time. Following the grouping, the system identifies and serves the user group that is most beneficial for that particular time slot. On the other hand, dynamic and static grouping is the beginning of a process in which users are initially divided into several groups based on channel conditions. Once the initial grouping is completed, there is no re-grouping in the next time slot. Instead, the system always serves only one group of users in each time slot.

[0077] Specifically, all users are grouped according to the correlation coefficient channels and preset thresholds between users to obtain multiple user groups, including:

[0078] Step B1: According to the correlation coefficient channel between users, the correlation coefficient between any two users among all users is determined.

[0079] Pairwise users among all users refer to all combinations of paired users among all users.

[0080] The magnitude of the correlation can be determined by calculating the value of the spatial correlation system between all users. The larger the value, the stronger the correlation between two users.

[0081] Step B2: among all users, find out the pairs of preset users whose correlation coefficient differences between the two users are greater than a preset threshold.

[0082] The preset threshold is a value greater than 0 and less than 1.

[0083] If the difference in correlation coefficients between any two users is greater than a preset threshold, there are highly correlated channels between two of the preset users, and they need to be allocated to different groups.

[0084] Step B3: Divide two users in the pairwise preset users into different user groups to obtain a preset user group.

[0085] Step B4: among all users, find out pairs of target users whose correlation coefficient differences between each pair of users are not greater than a preset threshold.

[0086] Step B5: When any one of the two target users exists in the preset user group, the two users of the two target users are placed in the corresponding preset user group.

[0087] Exemplarily, the preset user group includes two groups, the first group includes user 1, the second group includes user 2, the target users are user 2 and user 3, and user 2 exists in the second group of the preset user group, so both are directly placed in the second group of the preset user group.

[0088] Step B6: When any one of the two target users does not exist in the preset user group, two of the two target users are placed in a new user group.

[0089] Exemplarily, the preset user group includes two groups, the first group includes user 1, the second group includes user 2, and the target users are user 4 and user 5. User 4 and user 5 do not exist in the first group and the second group of the preset user group, so a new user group is created and user 4 and user 5 are placed in the new user group.

[0090] Through the operations of step B1 to step B6 above, all users can be divided into user groups with appropriate relevance.

[0091] S204: Determine an optimization problem including constraint conditions according to multiple user groups, optimization problems and constraint parameters.

[0092] The constraint parameters include the phase value of the rth element in the diagonal reflection phase matrix of RIS, the movable distance of the UAV between two time slots, the starting position of the UAV flight trajectory, the final position of the UAV flight trajectory, the energy consumption of the UAV and the flight range of the UAV.

[0093] Specifically, according to multiple user groups, optimization problems and constraint parameters, an optimization problem including constraint conditions is determined, including:

[0094] According to multiple user groups, optimization problems, the phase value of the rth element in the diagonal reflection phase matrix of RIS, the movable distance of the UAV between two time slots, the starting position of the UAV flight trajectory, the final position of the UAV flight trajectory, the energy consumption of the UAV and the flight range of the UAV, the optimization problem with constraints is determined using the following third formula:

[0095]

[0096] in, is an optimization problem with constraints, R sum is the data rate of all time slots for all users, is the energy consumption of the UAV in the nth time slot, is the energy consumption of the base station in the nth time slot, k is the kth user, K is the number of users, N is the number of time slots, L is the position of the UAV flight trajectory, Φ is the diagonal reflection phase matrix of RIS, and U i1 is the i1th user group, U j1 is the j1th user group, i is the set of phase values ​​of the rth element in the diagonal reflection phase matrix of RIS, G is the number of all user groups, U k is all user groups containing k users, θ r is the reflection phase shift value of the RIS array element in the nth time slot, λ is the wavelength, L n+1 is the position of the UAV flight trajectory in the n+1th time slot, L n is the position of the UAV flight trajectory in the nth time slot, D is the maximum distance that the UAV can move in a time slot, and L F is the final position of the UAV flight trajectory, L1 is the position of the UAV flight trajectory in the first time slot, L0 is the starting position of the UAV flight trajectory, and E max is the maximum UAV energy consumption, u min is the minimum UAV flight range, u u is the flight range of the drone, u max is the maximum UAV flight range, {x u ,y u ,z u} is the position coordinate of the drone.

[0097] The third formula in S204 is an optimization problem including constraints.

[0098] Indicates that the two groups do not have common users. Indicates that the total number of users in the user group used is K users, represents the reflection phase shift value of the RIS array element in the nth time slot, ||L n+1 -L n || 2 ≤D 2 ,n=1,…,N-1 means that the movable distance of the drone between two time slots is limited, ||L n -L F || 2 ≤D 2 ,L1=L0 represents the starting and final positions of the UAV flight trajectory, E uav ≤E max represents the energy consumption limit of the drone, u min u max ,u={x u ,y u ,z u} indicates the flight range limit of the drone.

[0099] By constraint and The problem is not convex and there are discrete phase shift constraints. These constraints create a hybrid non-convex problem, which brings great uncertainty to the trajectory design of the UAV. This uncertainty makes it difficult for traditional convex optimization-based methods to solve this problem. In order to meet the above challenges, the Deep Reinforcement Learning (DRL) algorithm is used to solve the joint problem. DRL is called to enable the agent to learn from the environment and make decisions to complete the solution of the problem.

[0100] S205. According to the diagonal reflection phase matrix of the optimal RIS, the DDQN model in reinforcement learning is used to solve the optimization problem containing constraints to obtain the optimal target ratio, so as to achieve the optimal UAV flight trajectory according to the optimal target ratio.

[0101] Among them, the optimal target ratio is the optimal ratio of the data rate of all time slots of all users in multiple user groups to the energy consumption of the drone and the energy consumption of the base station.

[0102] ​​The deployment range of RIS systems is usually limited to linear or two-dimensional configurations, limiting the potential installation positions for each element in the diagonal reflection phase matrix of the RIS, i.e., the impact of each RIS element. In contrast, when the RIS can be deployed from a UAV in three-dimensional space, the number of feasible positions for each RIS device is greatly increased. Therefore, a simple method, namely a local optimization RIS passive beamforming search method, is used.

[0103] Before solving the optimization problem including constraints using a deep double Q-network (DDQN) model in reinforcement learning according to the diagonal reflection phase matrix of the optimal RIS to obtain the optimal target ratio and realize the optimal UAV flight trajectory according to the optimal target ratio, the method also includes:

[0104] Step C1: For the first element in the diagonal reflection phase matrix of RIS, set the first initial values ​​of the other elements except the first element.

[0105] Step C2: Determine all first phase values ​​of the first element according to the first initial value and the preset range.

[0106] All first phase values ​​of the first element, that is, the reflection phase shift values ​​of the first element of the RIS array element in the nth time slot are expressed as:

[0107] Step C3: Find the phase value that maximizes the rate among all first phase values ​​as the first optimal phase value.

[0108] Maximizing the rate is a measure of the quality of information transmission.

[0109] Step C4: Step of determining the second optimal phase value: for the rth element in the diagonal reflection phase matrix of RIS, set the second initial value of the elements other than the rth element; determine all the second phase values ​​of the rth element according to the second initial value and the preset range; find the phase value that maximizes the rate among all the second phase values ​​as the second optimal phase value.

[0110] All second phase values ​​of the rth element, that is, the reflection phase shift values ​​of the RIS array element in the nth time slot are expressed as:

[0111] Step C5: Repeat the step of determining the second optimal phase value until all elements in the diagonal reflection phase matrix of the RIS are processed, stop the repetitive operation, and obtain the diagonal reflection phase matrix of the optimal RIS.

[0112] The diagonal reflection phase matrix of the optimal RIS includes a first optimal phase value and a second optimal phase value.

[0113] Each element in the diagonal reflection phase matrix of the optimal RIS has an optimal phase value.

[0114] The above steps C1 to C4 determine the optimal phase setting for the entire RIS component array. Multiple searches are performed to ensure that the most favorable phase configuration is achieved on all components.

[0115] In each step of the reinforcement learning training process, the agent is used to collect the state of the RIS-UAV assisted wireless network system in the environment, that is, the position of the drone in the current time slot. The designed deep neural network learns and selects the movement direction of the drone as the subsequent action based on the state. The action taken will have a corresponding impact on the environment, so the state of the environment changes. At the same time, the environment gives the agent a reward value based on whether the change is in the desired direction. The operations in the scenario of the present invention are as follows:

[0116] Specifically, according to the diagonal reflection phase matrix of the optimal RIS, the DDQN model in reinforcement learning is used to solve the optimization problem containing constraints to obtain the optimal target ratio, so as to achieve the optimal UAV flight trajectory according to the optimal target ratio, including:

[0117] Step D1: Take the DDQN model as the agent, the state space as the environment, the position of the drone in the current time slot as the state, and the direction in which the drone moves as the action.

[0118] The state represents the position of the drone in the current time slot. The entire flight area is divided into multiple subspaces of equal size. Assuming that the drone is located in different subspaces, it means that the agent is in different states. The state s[n] of the drone in time slot n can be expressed as:

[0119] s[n]=(x u [n],y u [n],z u [n]);

[0120] Among them, x u [n], y u [n] and z u [n] represents the coordinates of the drone on the x-axis, y-axis, and z-axis of the three-dimensional coordinate system in the nth time slot.

[0121] The action is set to the direction in which the drone moves. The action a[n] in the nth time slot can be defined as:

[0122] a[n]=(U n ,Z n );

[0123] Among them, U n Indicates the UAV flight action in the horizontal direction, Zn Indicates the drone's flight action in the vertical direction.

[0124] In the present invention, the UAV has five actions in the horizontal direction, namely, moving to the left by a subspace, moving to the right by a subspace, moving forward by a subspace, flying backward by a subspace, and staying still. Assuming that x1 and y1 are the distances between two adjacent subspaces in the horizontal direction on the x-axis and y-axis, respectively, then U n Expressed as: U n ∈{(x1,0),(-x1,0),(0,0),(0,y1),(0,-y1)}. The drone has three actions in the vertical direction, namely, moving up a subspace distance, moving down a subspace distance, and staying still. Therefore, Z n It can be expressed as Z n ∈{z1,0,-z1}, z1 represents the distance between two adjacent subspaces on the z-axis in the vertical direction. The present invention considers that the drone flies and moves within a limited space. If it flies out of the limited space, a penalty value will be imposed on the reward function. During the training and learning process, the intelligent body will adopt the ε-greedy strategy to select the next action with the highest Q value relative to the state s[n].

[0125] Step D2: According to the diagonal reflection phase matrix of the optimal RIS, the data rates of all time slots of all users in the multiple user groups are determined, and the ratio of the data rates of all time slots of all users in the multiple user groups to the energy consumption of the drone and the energy consumption of the base station is determined as an expression of the reward.

[0126] When the drone takes an action, the phase values ​​of the elements in the diagonal reflection phase matrix of RIS are adjusted to maximize the data rate of all time slots for all users. Assuming that the current state of the drone is s[n], and the new state after performing action a[n] is s[n+1], the reward obtained in this process is:

[0127]

[0128] Among them, r[n] is the reward of the nth time slot, R s ' um is the data rate of all time slots of all users in the multiple user groups, is the energy consumption of the UAV in the nth time slot, is the energy consumption of the base station in the nth time slot, k is the kth user, K is the number of users, and N is the number of time slots.

[0129] In the above reward expression, the numerator is the data rate of all time slots of all users in multiple user groups, and the denominator is the sum of the energy consumption of the drone in the nth time slot and the energy consumption of the base station in the nth time slot. The time slots in the reward expression are from the 1st time slot to the current n+1th time slot.

[0130] DDQN is a learning method that combines the Q-learning algorithm with a deep neural network. Q-learning constructs a Q-table to store the Q value, i.e., Q(s,a). Q(s,a) is the expected reward for taking action a in state s, and the algorithm uses the Q-value update function to update the Q-table. The DDQN model uses a neural network to approximate the Q-value update function to Q(s,a|θ), and then inputs the action and state into the neural network, and obtains the Q value after the neural network calculates and processes it. In Q(s,a|θ), θ is the weight parameter of DDQN. The training goal is to find the optimal weight θ so that the neural network can be as close to the optimal Q-value function as possible.

[0131] The following steps D3 to D9 are the drone trajectory optimization algorithm based on the DDQN model used in the present invention. When the algorithm is running, the hyperparameters of the two Q networks (evaluation network and target network) and other environmental parameters are first initialized. Secondly, in each round of training, N time slots are trained. At the beginning of each round of training, the time slot n=1 is reset and the environmental parameters are initialized, that is, the position of the drone is restored to the starting point. The input is: the parameters θ of the evaluation network and the parameters θ of the target network - , let θ - =θ; the number of training rounds E, the number of time slots N, the size of the experience replay pool, the output is the optimal UAV flight trajectory, which can be expressed as {L0,L1…L F}.

[0132] Step D3: Initialize environmental parameters and obtain the current state.

[0133] Step D4: Use the agent to adopt a greedy strategy, select the current action relative to the current state and execute the current action.

[0134] The current action is the best action that maximizes the reward.

[0135] Step D5: After executing the current action, the agent interacts with the environment according to the expression of the current action and reward, and returns the function value of the reward and the new state.

[0136] Specifically, after executing the current action, according to the expression of the current action and reward, the agent interacts with the environment, adjusts the diagonal reflection phase matrix of RIS to maximize the data rate of all time slots of all users, and obtains the function value of the reward and the new state.

[0137] Step D6: The current state, current action, function value of reward and new state are stored in the experience replay pool as experience data. When the amount of experience data reaches the preset amount, the storage is stopped.

[0138] Step D7: Randomly extract a target number of experience data from the experience replay pool as training data for training the DDQN model to train the DDQN model.

[0139] Step D8: Calculate the objective function of the DDQN model and update the parameters of the evaluation network according to the loss function.

[0140] The calculation update formula for the parameter θ of the evaluation network is:

[0141]

[0142] Among them, Q(s n ,a n |θ) is the state in state s n Take action a n y is the expected return, y is the actual return, Q(s n+1 ,a|θ - ) is the next state s n+1 Take action a n The expected return at time θ - is the network parameter used to select the action, i.e., the parameter of the target network, s n is the current state, a n is the immediate reward obtained at time step n, is the gradient of θ, and δ is the learning rate.

[0143] The loss function used to train θ is as follows:

[0144]

[0145] Among them, Loss(θ) is the loss function of θ, To find the mathematical expectation, is the objective function of DDQN.

[0146] Objective Function of DDQN It is expressed as:

[0147]

[0148] Among them, r n+1 is the reward at the n+1th time, γ is the discount factor used to measure the importance of future rewards, and s n+1 is the next state, θ - are network parameters used to select actions.

[0149] Step D9: Update the target network parameters. When the maximum number of iterations is reached, terminate the iteration operation and obtain the optimal target ratio to achieve the optimal UAV flight trajectory according to the optimal target ratio.

[0150] The purpose of the present invention is to use a UAV to install a RIS to assist a wireless network in the air, in order to establish an indirect line-of-sight link between the BS and the user end. The mobile UAV communicates with the ground user along its planned trajectory, and the signal sent by the BS is reflected to the user through the RIS installed on the UAV. The DDQN model is used to optimize the UAV trajectory. The close spacing between RIS elements in the RIS-UAV network can enhance the spatial correlation, resulting in smaller user signals at different angles and channel correlation between users. In addition, the BS-RIS link may not have rich scattering, so there may be correlation in the channels of cascaded different users. The present invention adopts a method of grouping users to reduce the impact of user correlation and places users in different groups with high correlation in the grouping algorithm. Two grouping methods are also provided: dynamic grouping and static grouping.

[0151] Figure 3 The performance convergence comparison diagram is schematically shown, with the horizontal axis being the number of training cycles and the vertical axis being the reward, corresponding to energy efficiency. The blue line represents the Double Deep Q-Network (Double_DQN), the red line represents the Dueling Deep Q-Network (Dueling DQN), the green line represents the Deep Q-Network (DQN), and the yellow line represents the Deep Q-Network with Prioritized Experience Replay (DQNPR). The RIS-UAV network energy efficiency optimization method based on reinforcement learning of the present invention is used and compared with solutions based on DQNPR, DQN and Dueling DQN algorithms. By setting the parameters of all four network models to the same value, Figure 3 It is clearly shown that once the DDQN model is fully trained, it outperforms the other two models and effectively improves the reward value. As the training progresses, the intelligent system learns the characteristics of the working environment, thereby generating a reward value that is stable over time. Therefore, the RIS-UAV network energy efficiency optimization method based on reinforcement learning proposed in this invention is proven to be an effective method.

[0152] Figure 4The reward and energy diagram of different RIS elements are schematically shown, with the horizontal axis being the number of training cycles, the vertical axis on the left being the reward, and the vertical axis on the right being the energy. The blue solid line represents the energy consumption of a UAV with 64 elements, the red solid line represents the energy consumption of a UAV with 32 elements, and the green solid line represents the energy consumption of a UAV with 16 elements. The blue dotted line represents the user and data rate with 64 elements, the red dotted line represents the user and data rate with 32 elements, and the green dotted line represents the user and data rate with 16 elements. As the number of RIS units increases, the system capacity of the RIS-UAV network can be increased. When N=16$ changes to N=32$, the benefit component brought by the increase in the number of RIS is not significant enough. As the number of elements further increases, when $N=64$, the system and rate will increase significantly. By intelligently adjusting the phase and amplitude of the RIS reflected signal, the network can amplify the signal strength in the target direction. This optimization of the signal path not only enhances the coverage and quality of wireless communication, but also significantly improves the energy efficiency of the system.

[0153] Figure 5 The comparison diagram of rewards and energy for different numbers of base station antennas is schematically shown. The horizontal axis is the number of training cycles, the vertical axis on the left is the reward, and the vertical axis on the right is the energy. The blue solid line represents the reward value for 128 antennas, the red solid line represents the reward value for 64 antennas, and the green solid line represents the reward value for 32 antennas. The blue dotted line represents the energy consumed by the drone with 128 antennas, the red dotted line represents the energy consumed by the drone with 64 antennas, and the green dotted line represents the energy consumed by the drone with 32 antennas. It can be seen that, especially when the number of BS antennas is not too large, the average rate increases with the increase in the number of BS antennas. When $M=128$, the average rate does not increase with the increase in the number of BS antennas, because increasing the number of BS antennas will reduce the correlation between UTs. When the grouping threshold is high, strongly correlated UTs will not be grouped into different groups or grouping will not be performed, and all UTs will be simultaneously subject to more severe interference between service UTs, resulting in a decrease in system performance.

[0154] Figure 6The schematic diagram shows different correlation levels of component spacing in RIS, with the horizontal axis being the number of training cycles, the vertical axis on the left being the reward, and the vertical axis on the right being the user and data rate. The blue solid line represents the reward value when the array element spacing is 1 / 8, the red solid line represents the reward value when the array element spacing is 1 / 4, the green solid line represents the reward value when the array element spacing is 1 / 2, the blue dotted line represents the user and data rate when the array element spacing is 1 / 8, the red dotted line represents the user and data rate when the array element spacing is 1 / 4, and the green dotted line represents the user and data rate when the array element spacing is 1 / 2. By changing this parameter, the reward and date rate of the system are effectively changed. As the distance between elements increases, the correlation increases, and the reward and data rate decrease, resulting in reduced system energy efficiency.

[0155] Figure 7 The schematic diagram of the reward and data rate of three systems with different numbers of users is shown schematically, the horizontal axis is the number of training cycles, the vertical axis on the left is the reward, and the vertical axis on the right is the user data rate. The blue solid line represents the reward value when the number of users is 40, the red solid line represents the reward value when the number of users is 30, the green solid line represents the reward value when the number of users is 20, the blue dotted line represents the reward value when the number of users is 40, the red dotted line represents the reward value when the number of users is 30, and the green dotted line represents the reward value when the number of users is 20. It can be seen that as the number of users increases, the reward and dating rate of the system will also increase. In theory, the more users there are, the stronger the user correlation is. The experimental results show that the grouping scheme of the present invention is effective. Our grouping can reduce the interference of highly correlated users and further improve the system and data rate by dividing them into different groups as the number of users increases.

[0156] Figure 8 The energy efficiency comparison diagram under three different grouping conditions is schematically shown. The horizontal axis is the number of training cycles, the vertical axis is the reward, the blue solid line represents the case of no grouping, the red solid line represents the case of static grouping, and the green solid line represents the case of dynamic grouping. The grouping includes dynamic grouping, static grouping, and no grouping. It can be seen that static grouping is better than dynamic grouping and non-grouping, and dynamic grouping is close to the non-grouping scheme.

[0157] Fig. 9 The schematic diagram shows the correlation coefficient of the user group corresponding to the three indicators changing from 0 to 1. The horizontal axis is the number of training cycles, the vertical axis is the average value, the blue dot line represents the average reward value, the red square line represents the average data rate, and the green triangle line represents the average energy efficiency. The three indicators are the average energy of the system efficiency, the average data rate, and the average energy consumption of the drone. Fig. 9 It can be seen that these three indicators reach their maximum value when the correlation increases to a maximum value of 0.7, and then gradually decrease. The values ​​corresponding to 0 and 1 are equal. Because at this time, users are no longer grouped.

[0158] Based on the above Figure 2 It can be seen from the implementation method that the embodiment of the present invention determines the data rate of all time slots of all users according to the Rayleigh channel from the drone to each user and the diagonal reflection phase matrix of the RIS; constructs an optimization problem according to the data rate of all time slots of all users, the energy consumption of the base station and the energy consumption of the drone; groups all users according to the correlation coefficient channel and the preset threshold between users to obtain multiple user groups; determines the optimization problem containing constraints according to the multiple user groups, the optimization problem and the constraint parameters; according to the diagonal reflection phase matrix of the optimal RIS, uses the DDQN model in reinforcement learning to solve the optimization problem containing constraints, and obtains the optimal target ratio, so as to achieve the optimal drone flight trajectory according to the optimal target ratio, and the optimal target ratio is the optimal ratio of the data rate of all time slots of all users in the multiple user groups to the energy consumption of the drone and the energy consumption of the base station. In this way, by grouping all users, the impact of user correlation is reduced, so as to reduce the correlation impact of the correlation coefficient channel between users and improve the performance of the communication system; by grouping all users, according to the diagonal reflection phase matrix of the optimal RIS, the DDQN model in reinforcement learning is used to solve the optimization problem containing constraints and obtain the optimal target ratio. According to the optimal target ratio, the optimal UAV flight trajectory can be achieved, and the optimal UAV flight trajectory can improve the energy efficiency of IRS-UAV.

[0159] Based on the same inventive concept, as an implementation of the above-mentioned RIS-UAV network energy efficiency optimization method based on reinforcement learning, an embodiment of the present invention also provides a RIS-UAV network energy efficiency optimization device based on reinforcement learning. Fig.10 is a structural diagram of the device in the embodiment of the present invention, see Fig.10 As shown, the RIS-UAV network energy efficiency optimization device based on reinforcement learning may include:

[0160] The first determination module 1001 is used to determine the data rate of all time slots of all users according to the Rayleigh channel from the drone to each user and the diagonal reflection phase matrix of the RIS;

[0161] A construction module 1002 is used to construct an optimization problem based on the data rates of all time slots of all users, the energy consumption of the base station and the energy consumption of the drone;

[0162] A grouping module 1003 is used to group all users according to the correlation coefficient channels between users and a preset threshold value to obtain multiple user groups;

[0163] A second determination module 1004 is used to determine an optimization problem including constraint conditions according to multiple user groups, optimization problems and constraint parameters;

[0164] The solution module 1005 is used to solve the optimization problem containing constraints according to the diagonal reflection phase matrix of the optimal RIS using the DDQN model in reinforcement learning to obtain the optimal target ratio, so as to achieve the optimal UAV flight trajectory according to the optimal target ratio. The optimal target ratio is the optimal ratio of the data rate of all time slots of all users in multiple user groups to the energy consumption of the UAV and the energy consumption of the base station.

[0165] The first determination module 1001 is specifically used to determine the signal to interference plus noise ratio according to the Rayleigh channel from the drone to each user and the diagonal reflection phase matrix of the RIS; and determine the data rate of all time slots of all users according to the signal to interference plus noise ratio.

[0166] The second determination module 1004, the constraint parameters include the phase value of the rth element in the diagonal reflection phase matrix of the RIS, the movable distance of the drone between two time slots, the starting position of the drone flight trajectory, the final position of the drone flight trajectory, the energy consumption of the drone and the flight range of the drone; according to multiple user groups, optimization problems and constraint parameters, determining the optimization problem containing constraint conditions, including: according to multiple user groups, optimization problems, the phase value of the rth element in the diagonal reflection phase matrix of the RIS, the movable distance of the drone between two time slots, the starting position of the drone flight trajectory, the final position of the drone flight trajectory, the energy consumption of the drone and the flight range of the drone, using the following third formula to determine the optimization problem containing constraint conditions:

[0167]

[0168] in, For the optimization problem, R sum is the data rate of all time slots for all users, is the energy consumption of the UAV in the nth time slot, is the energy consumption of the base station in the nth time slot, k is the kth user, K is the number of users, N is the number of time slots, L is the position of the UAV flight trajectory, Φ is the diagonal reflection phase matrix of RIS, and U i1 is the i1th user group, U j1 is the j1th user group, i is the set of phase values ​​of the rth element in the diagonal reflection phase matrix of RIS, G is the number of user groups, U k is all user groups containing k users, θ r is the reflection phase shift value of the RIS array element in the nth time slot, λ is the wavelength, L n+1 is the position of the UAV flight trajectory in the n+1th time slot, L n is the position of the UAV flight trajectory in the nth time slot, D is the maximum distance that the UAV can move in a time slot, and L Fis the final position of the UAV flight trajectory, L1 is the position of the UAV flight trajectory in the first time slot, L0 is the starting position of the UAV flight trajectory, and E max is the maximum UAV energy consumption, u min is the minimum UAV flight range, u u is the flight range of the drone, u max is the maximum UAV flight range, {x u ,y u ,z u} is the position coordinate of the drone.

[0169] The grouping module 1003 is specifically used to determine the correlation coefficients between two users among all users according to the correlation coefficient channels between users; find out the two preset users among all users whose difference in correlation coefficients between two users is greater than a preset threshold; divide two users among the two preset users into different user groups to obtain a preset user group; find out the two target users among all users whose difference in correlation coefficients between two users is not greater than the preset threshold; when any one of the two target users exists in the preset user group, put the two users among the two target users into the corresponding preset user group; when any one of the two target users does not exist in the preset user group, put the two users among the two target users into a new user group.

[0170] The RIS-UAV network energy efficiency optimization device based on reinforcement learning also includes: a third determination module, which is used to set the first initial value of the first element in the diagonal reflection phase matrix of the RIS except the first element before solving the optimization problem containing constraints based on the diagonal reflection phase matrix of the optimal RIS using the DDQN model in reinforcement learning to achieve the optimal UAV flight trajectory based on the optimal target ratio; determine all the first phase values ​​of the first element according to the first initial value and the preset range; find the phase value that maximizes the rate among all the first phase values ​​as the first optimal phase value. The second optimal phase value determination operation: set the second initial value of the other elements except the rth element in the diagonal reflection phase matrix of the RIS; determine all the second phase values ​​of the rth element according to the second initial value and the preset range; find the phase value that maximizes the rate among all the second phase values ​​as the second optimal phase value. Repeat the operation of determining the second optimal phase value until all elements in the diagonal reflection phase matrix of the RIS are processed, stop the repeated operation, and obtain the diagonal reflection phase matrix of the optimal RIS, which includes the first optimal phase value and the second optimal phase value.

[0171] The solution module 1005 is specifically used to use the DDQN model as an intelligent agent, the state space as an environment, the position of the drone in the current time slot as a state, and the direction of the drone's movement as an action; determine the data rate of all time slots of all users in multiple user groups according to the diagonal reflection phase matrix of the optimal RIS, and determine the ratio of the data rate of all time slots of all users in multiple user groups to the energy consumption of the drone and the energy consumption of the base station as an expression of the reward; initialize environmental parameters to obtain the current state; use the intelligent agent to adopt a greedy strategy to select the current action relative to the current state and execute the current action; after executing the current action, according to the expression of the current action and the reward, The intelligent agent interacts with the environment and returns the function value of the reward and the new state; the current state, current action, function value of the reward and the new state are stored in the experience replay pool as experience data, and the storage is stopped when the amount of experience data reaches the preset amount; the target amount of experience data is randomly extracted from the experience replay pool as the training data for training the DDQN model to train the DDQN model; the objective function of the DDQN model is calculated, and the parameters of the evaluation network are updated according to the loss function; the target network parameters are updated, and the iteration operation is terminated when the maximum number of iterations is reached to obtain the optimal target ratio, so as to achieve the optimal UAV flight trajectory according to the optimal target ratio.

[0172] It should be pointed out here that the above description of the embodiment of the RIS-UAV network energy efficiency optimization device based on reinforcement learning is similar to the description of the embodiment of the RIS-UAV network energy efficiency optimization method based on reinforcement learning, and has similar beneficial effects as the embodiment of the RIS-UAV network energy efficiency optimization method based on reinforcement learning. For technical details not disclosed in the embodiment of the RIS-UAV network energy efficiency optimization device based on reinforcement learning of the embodiment of the present invention, please refer to the description of the embodiment of the RIS-UAV network energy efficiency optimization method based on reinforcement learning of the present invention for understanding.

[0173] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A RIS-UAV network energy efficiency optimization method based on reinforcement learning, characterized in that: The RIS-UAV network energy efficiency optimization method based on reinforcement learning includes: According to the Rayleigh channel from the UAV to each user and the diagonal reflection phase matrix of RIS, the data rate of all time slots of all users is determined; Constructing an optimization problem based on the data rates of all time slots of all users, the energy consumption of the base station, and the energy consumption of the drone; Grouping all the users according to the correlation coefficient channels between the users and a preset threshold to obtain a plurality of user groups; Determining an optimization problem including constraint conditions according to the multiple user groups, the optimization problem and constraint parameters; According to the diagonal reflection phase matrix of the optimal RIS, the DDQN model in reinforcement learning is used to solve the optimization problem containing constraints to obtain an optimal target ratio, so as to achieve an optimal UAV flight trajectory according to the optimal target ratio, and the optimal target ratio is an optimal ratio of the data rate of all time slots of all users in multiple user groups to the energy consumption of the UAV and the energy consumption of the base station.

2. The RIS-UAV network energy efficiency optimization method based on reinforcement learning according to claim 1 is characterized in that: The data rates of all time slots of all users are determined according to the Rayleigh channel from the drone to each user and the diagonal reflection phase matrix of the RIS, including: Determine a signal-to-interference-and-noise ratio according to a Rayleigh channel from the drone to each user and a diagonal reflection phase matrix of the RIS; The data rates of all time slots of all users are determined according to the signal to interference plus noise ratio.

3. The RIS-UAV network energy efficiency optimization method based on reinforcement learning according to claim 2 is characterized in that: The determining of the signal to interference and noise ratio according to the Rayleigh channel from the drone to each user and the diagonal reflection phase matrix of the RIS comprises: According to the Rayleigh channel from the drone to each user and the diagonal reflection phase matrix of the RIS, the signal to interference noise ratio is determined using the following first formula: Among them, SINR k [n] is the signal-to-interference-noise ratio of the kth user in the nth time slot, is the transpose of the Rayleigh channel from the UAV to the kth user, Φ is the diagonal reflection phase matrix of the RIS, H[n] is the Rice channel between the base station and the UAV, is the k-th user precoding vector, is an M×1 complex space, w j ' is the j'th user precoding vector, any one of the precoding vectors of K users, σ 2 is the noise variance, K is the number of users, and j' is any one of the K users.

4. The RIS-UAV network energy efficiency optimization method based on reinforcement learning according to claim 2 is characterized in that: The determining, according to the signal to interference and noise ratio, the data rates of all time slots of all users comprises: According to the signal to interference and noise ratio, the data rates of all time slots of all users are determined using the following second formula: Among them, R sum is the data rate of all time slots of all users, SINR k [n] is the signal to interference and noise ratio of the kth user in the nth time slot, k is the kth user, K is the number of users, and N is the number of time slots.

5. The RIS-UAV network energy efficiency optimization method based on reinforcement learning according to claim 1 is characterized in that: The constraint parameters include the phase value of the rth element in the diagonal reflection phase matrix of the RIS, the movable distance of the UAV between two time slots, the starting position of the UAV flight trajectory, the final position of the UAV flight trajectory, the energy consumption of the UAV and the flight range of the UAV; The step of determining the optimization problem including constraint conditions according to the multiple user groups, the optimization problem and the constraint parameters includes: According to the multiple user groups, the optimization problem, the phase value of the rth element in the diagonal reflection phase matrix of the RIS, the movable distance of the drone between the two time slots, the starting position of the drone flight trajectory, the final position of the drone flight trajectory, the energy consumption of the drone and the flight range of the drone, the optimization problem containing constraints is determined using the following third formula: in, is the objective function of the optimization problem, R sum is the data rate of all time slots of all users, is the energy consumption of the UAV in the nth time slot, is the energy consumption of the base station in the nth time slot, k is the kth user, K is the number of users, N is the number of time slots, L is the position of the UAV flight trajectory, Φ is the diagonal reflection phase matrix of the RIS, and U i1 is the i1th user group, U j1 is the j1th user group, i is the set of phase values ​​of the rth element in the diagonal reflection phase matrix of the RIS, G is the number of all user groups, U k is all user groups containing the kth user, θ r is the reflection phase shift value of the RIS array element in the rth time slot, λ is the wavelength, L n+1 is the position of the UAV flight trajectory in the n+1th time slot, L n is the position of the UAV flight trajectory in the nth time slot, D is the maximum distance that the UAV can move in a time slot, and L F is the final position of the UAV flight trajectory, L1 is the position of the UAV flight trajectory in the first time slot, L0 is the starting position of the UAV flight trajectory, and E max is the maximum UAV energy consumption, u min is the minimum UAV flight range, u u is the flight range of the drone, u max is the maximum flight range of the UAV, {x u ,y u ,z u } is the position coordinate of the drone.

6. The RIS-UAV network energy efficiency optimization method based on reinforcement learning according to claim 1 is characterized in that: The method of grouping all users according to the correlation coefficient channel between users and the preset threshold value to obtain multiple user groups includes: According to the correlation coefficient channel between users, the correlation coefficient between any two users among all users is determined; Among all the users, find out the two preset users whose difference of correlation coefficient between the two users is greater than a preset threshold; Dividing two users among the pairwise preset users into different user groups to obtain a preset user group; Among all the users, find the pair of target users whose difference in correlation coefficient between the two users is not greater than a preset threshold; When any one of the two target users exists in the preset user group, two of the two target users are placed in the corresponding preset user group; When any one of the two target users does not exist in the preset user group, two of the two target users are placed in a new user group.

7. The RIS-UAV network energy efficiency optimization method based on reinforcement learning according to claim 1 is characterized in that: Before solving the optimization problem including constraints by using the DDQN model in reinforcement learning according to the diagonal reflection phase matrix of the optimal RIS to obtain the optimal target ratio so as to realize the optimal UAV flight trajectory according to the optimal target ratio, the method further includes: For a first element in a diagonal reflection phase matrix of the RIS, setting first initial values ​​of other elements except the first element; Determining all first phase values ​​of the first element according to the first initial value and a preset range; Finding a phase value that maximizes the rate among all the first phase values ​​as the first optimal phase value; The step of determining the second optimal phase value is as follows: for the rth element in the diagonal reflection phase matrix of the RIS, setting the second initial values ​​of the other elements except the rth element; determining all second phase values ​​of the rth element according to the second initial value and the preset range; finding the phase value that maximizes the rate among all the second phase values ​​as the second optimal phase value; Repeat the step of determining the second optimal phase value until all elements in the diagonal reflection phase matrix of the RIS are processed, stop the repeated operation, and obtain the diagonal reflection phase matrix of the optimal RIS, wherein the diagonal reflection phase matrix of the optimal RIS includes the first optimal phase value and the second optimal phase value.

8. The RIS-UAV network energy efficiency optimization method based on reinforcement learning according to claim 7 is characterized in that: The diagonal reflection phase matrix of the optimal RIS is used to solve the optimization problem containing constraints by using the DDQN model in reinforcement learning to obtain the optimal target ratio, so as to achieve the optimal UAV flight trajectory according to the optimal target ratio, including: The DDQN model is used as the agent, the state space is used as the environment, the position of the drone in the current time slot is used as the state, and the direction of the drone's movement is used as the action; Determine the data rates of all time slots of all users in the multiple user groups according to the diagonal reflection phase matrix of the optimal RIS, and determine the ratio of the data rates of all time slots of all users in the multiple user groups to the energy consumption of the drone and the energy consumption of the base station as an expression of the reward; Initialize environment parameters and obtain the current status; Using the agent to adopt a greedy strategy, select a current action relative to the current state and execute the current action; After executing the current action, the agent interacts with the environment according to the current action and the expression of the reward, and returns a function value of the reward and a new state; The current state, the current action, the function value of the reward and the new state are stored as experience data in an experience replay pool, and when the amount of the experience data reaches a preset amount, storage is stopped; Randomly extracting a target number of experience data from the experience replay pool as training data for training the DDQN model, so as to train the DDQN model; Calculate the objective function of the DDQN model and update the parameters of the evaluation network according to the loss function; The target network parameters are updated, and when the maximum number of iterations is reached, the iteration operation is terminated to obtain the optimal target ratio so as to achieve the optimal UAV flight trajectory according to the optimal target ratio.

9. The RIS-UAV network energy efficiency optimization method based on reinforcement learning according to claim 8 is characterized in that: The reward expression is: Among them, r[n] is the reward of the nth time slot, R sum is the data rate of all time slots of all users in the plurality of user groups, is the energy consumption of the UAV in the nth time slot, is the energy consumption of the base station in the nth time slot, k is the kth user, K is the number of users, and N is the number of time slots.

10. A RIS-UAV network energy efficiency optimization device based on reinforcement learning, characterized in that: The RIS-UAV network energy efficiency optimization device based on reinforcement learning includes: The first determination module is used to determine the data rate of all time slots of all users according to the Rayleigh channel from the drone to each user and the diagonal reflection phase matrix of the RIS; A construction module, used for constructing an optimization problem according to the data rates of all time slots of all users, the energy consumption of the base station and the energy consumption of the drone; A grouping module, used to group all the users according to the correlation coefficient channels between the users and a preset threshold value to obtain a plurality of user groups; A second determination module, configured to determine an optimization problem including constraint conditions according to the multiple user groups, the optimization problem and constraint parameters; A solution module is used to solve the optimization problem containing constraints according to the diagonal reflection phase matrix of the optimal RIS by using the DDQN model in reinforcement learning to obtain an optimal target ratio, so as to achieve an optimal UAV flight trajectory according to the optimal target ratio, wherein the optimal target ratio is an optimal ratio of the data rate of all time slots of all users in multiple user groups to the energy consumption of the UAV and the energy consumption of the base station.

Images