RIS-assisted unmanned aerial vehicle communication control method and related device
Through Markov decision-making process model and TD3 algorithm, the drone movement trajectory and RIS phase shift angle are optimized, and the signal attenuation and battery life problems in high-rise buildings are solved, and efficient drone communication and energy utilization are achieved. Ensure communication quality and security.
Patent Information
- Application Number
- CN202411125152.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-16
AI Technical Summary
In dense and complex high-rise buildings, the signal propagation between drones and users is attenuated by the building, and the drone's battery technology cannot meet the needs of long-term and high-intensity communication, which has security risks and dynamic channel changes. Communication quality issues.
The Markov decision-making process model is used to combine TD3 algorithm and AO algorithm to optimize the drone's movement trajectory, transmission power and RIS phase shift angle, and convert the optimization problem into joint optimization problem, improve the communication capability and energy efficiency of the drone to ensure that there is no Reasonable arrangement of human-machine flight paths and communication quality.
Under strict time and energy limitations, reasonable arrangement of drone flight paths can be achieved, communication throughput and energy utilization efficiency can be maximized, and discrete phase shift angles of RIS can be optimized to improve the environment quality of wireless channel.
Smart Images

Figure CN119095057A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a control method and a related device for RIS-assisted unmanned aerial vehicle communication, belonging to the field of unmanned aerial vehicle-assisted wireless communication. Background Art
[0002] In drone-assisted wireless communications, drones are mainly used as mobile aerial communication platforms to provide wireless access to ground users from high altitudes. However, in densely populated and complex areas with high-rise buildings, signal propagation between drones and users will still be attenuated due to obstruction by buildings, which brings certain challenges to drone deployment.
[0003] Reconfigurable smart metasurface (RIS) technology has been proven to be the most promising technology to improve data transmission efficiency. As an important auxiliary means for 5G and 6G wireless communication networks, signal propagation can be optimized by adjusting the phase shift angle of each reflective element of RIS. RIS is usually fixedly deployed on the walls of buildings to improve the quality of signal transmission to supplement wireless communication networks. In recent years, the integration of RIS with drone technology has attracted more and more attention, and has raised many challenging issues such as drone trajectory design, resource allocation, system capacity, etc., and has been widely used in different scenarios.
[0004] In urban environments, it is very important for RIS to assist drones in information dissemination and data collection, but there are still some problems and challenges. First, the energy of drones mainly comes from the batteries they carry, and the growth of battery technology still cannot meet the long-term and high-intensity communication needs of drones. Therefore, improving the energy efficiency of drones is an urgent problem to be solved in the field of wireless communications. Furthermore, in reality, for reasons of security and privacy, some buildings or environmental areas are prohibited for drones to cross, which leads to safety risks during the flight of drones, which requires reasonable arrangements for the flight path of drones. Finally, the random movement of target users in the environment can easily lead to dynamic changes in the channel, which in turn affects the communication quality. Summary of the invention
[0005] The present invention provides a control method and related devices for RIS-assisted unmanned aerial vehicle communication, which solve the problems disclosed in the background technology.
[0006] According to one aspect of the present disclosure, a control method for RIS-assisted UAV communication is provided, comprising: The elements of the Markov decision process model are updated by using the three-dimensional coordinates of the UAV, the three-dimensional coordinates of the user and the RIS phase shift matrix at the current time step; wherein the Markov decision process model is transformed through an optimization problem, which is a problem of jointly optimizing the UAV movement trajectory, the UAV transmission power and the RIS phase shift deflection, with the goal of improving the communication capability of the UAV and the data collection of energy efficiency; The TD3 algorithm is used to solve the updated Markov decision process model to obtain the UAV movement control instructions and transmission power control instructions of the current time step, as well as the phase shift angle control instructions of each RIS component; wherein, during training, the TD3 algorithm adjusts the probability of sampling experience data by assigning priority weights to the data in the experience pool; when using the TD3 algorithm for solving, the AO algorithm is used to calculate the optimal phase shift angle for the current time step, and the optimal phase shift angle is the phase shift angle that maintains the maximum communication rate between the UAV and the user.
[0007] In some embodiments of the present disclosure, the Markov decision process model is pre-constructed, and the construction process includes: According to the environmental parameters of RIS-assisted UAV communication, the communication model and energy consumption model are constructed; According to the communication model and energy consumption model, an optimization problem is constructed and converted into a Markov decision process model.
[0008] In some embodiments of the present disclosure, the communication model is: ; In the formula, r u,k,m is the communication rate between the UAV and the mth user in the kth user cluster, B is the total bandwidth, is the average bandwidth from the drone to each user, P is the transmission power of the drone, σ is the additive Gaussian white noise, is the average channel gain between the UAV and the mth user in the kth user cluster, p u,k,m is the blocking probability between the UAV and the mth user in the kth user cluster, g u,k,m is the channel gain when the mth user in the kth user cluster communicates directly with the drone, g u,r,k,m = g r,k,m Θg r,u is the channel gain when the mth user in the kth user cluster communicates indirectly with the drone, g r,k,m is the channel gain from the mth user in the kth user cluster to the RIS, g r,u is the channel gain from UAV to RIS, Θ is the RIS phase shift matrix; The energy consumption model is: E u =E mov +E t ; In the formula, E u is the total energy consumption of the UAV, E t is the launch energy consumption of the UAV, E mov is the energy consumption of the UAV’s movement, , P0 and P1 are the blade power and derived power of the UAV in hovering state, Utip is the tip speed of the UAV rotor blade, v0 is the average rotor induced speed of the UAV in the hovering state, v is the moving speed of the UAV, d0, ρ, s0 and A are the parameters related to the UAV fuselage drag ratio, air density, rotor solidity and rotor disc area, respectively.
[0009] In some embodiments of the present disclosure, the optimization problem is: ; st C1:0≤x u (t) ≤L; C2:0≤y u (t) ≤L; C3:L u (0)={(x u ,y u )|(x u ,y u )∈A ini}; C4:L u (N-1)={(x u ,y u )|(x u ,y u )∈A des}; C5:|| L u (t-1)- L u (t)||2≤V max δ t ; C6: 0≤P(t) ≤P max ; C7: ; C8: ; Where, L u is the three-dimensional coordinate of the drone, K is the number of user clusters, M is the number of users in the user cluster, x u (t) is the three-dimensional coordinate L of the UAV at time step t u (t) is the x-axis coordinate, y u (t) is the three-dimensional coordinate L of the UAV at time step t u (t) is the Y-axis coordinate, L is the maximum boundary value of the UAV flight area, and A is ini , A des are the starting and ending position ranges of the drone, L u (0) and L u (N-1) are the three-dimensional coordinates of the drone at the initial and final time steps, respectively, and L u (t-1) are the three-dimensional coordinates of the UAV at time step t-1, Vmax is the maximum speed of the UAV, δ t is the duration of time step t, P(t) is the transmission power of the UAV at time step t, P max is the upper limit of the UAV’s transmission power, E u (t) is the total energy consumption of the UAV at time step t, For the The phase shift angle of each element, n is the number of quantization bits.
[0010] In some embodiments of the present disclosure, in the Markov decision process model: The state space is represented as S ={s t =( S 1, S 2 )}, where s t is the state at time step t, S 1 is the two-dimensional coordinate of the UAV at time step t, S 2 is the distance set between the drone and the user, and between the drone and the destination at time step t; The action space is represented as A ={a t =(θ(t),l(t),P(t)}, where, a t is the action at time step t, θ(t) is the flight direction of the UAV at time step t, l(t) is the flight distance of the UAV at time step t, and P(t) is the transmission power of the UAV at time step t; The reward function is expressed as: ; Where R(t) is the reward value at time step t, δ c1 ~δ c4 is a constant, r u,k,m is the communication rate between the UAV and the mth user in the kth user cluster, E u is the total energy consumption of the UAV, R des are the guidance parameters for the UAV to reach its destination, , d des (t) is the horizontal distance between the UAV and the destination center at time step t, d des (t-1) is the horizontal distance between the UAV and the destination center at time step t-1, R ser The additional reward for the drone to complete the user data collection, R pun Penalty for abnormal conditions.
[0011] In some embodiments of the present disclosure, the priority of data allocation in the experience pool is the absolute value of the TD error δ plus ɛ, and the probability of the jth sample data being sampled is ;in, is the total number of sample data for each sampling, p i and p j are the priorities of the i-th and j-th sample data respectively, is the priority level, and ɛ is a small constant.
[0012] According to another aspect of the present disclosure, a control device for RIS-assisted UAV communication is provided, comprising: An update module uses the drone's 3D coordinates, the user's 3D coordinates, and the RIS phase shift matrix at the current time step to update the elements of the Markov decision process model; the Markov decision process model is transformed through an optimization problem, which is a problem of jointly optimizing the drone's movement trajectory, the drone's transmission power, and the RIS phase shift angle deflection, with the goal of improving the drone's communication capability and energy-efficient data collection; The solution module uses the TD3 algorithm to solve the updated Markov decision process model to obtain the UAV movement control instructions and transmission power control instructions of the current time step, as well as the phase shift angle control instructions of each RIS component; wherein, during training, the TD3 algorithm adjusts the probability of sampling experience data by assigning priority weights to the data in the experience pool; when using the TD3 algorithm for solution, the AO algorithm is used to find the optimal phase shift angle for the current time step, and the optimal phase shift angle is the phase shift angle that maintains the maximum communication rate between the UAV and the user.
[0013] In some embodiments of the present disclosure, the Markov decision process model in the update module is pre-constructed, and the construction process includes: According to the environmental parameters of RIS-assisted UAV communication, the communication model and energy consumption model are constructed; According to the communication model and energy consumption model, an optimization problem is constructed and converted into a Markov decision process model.
[0014] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, enable the computing device to perform a control method for RIS-assisted drone communication.
[0015] According to another aspect of the present disclosure, a computer device is provided, comprising one or more processors and one or more memories, wherein one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for executing a control method for RIS-assisted drone communication.
[0016] The beneficial effects achieved by the present invention are as follows: 1. The present invention converts the joint optimization problem of the UAV movement trajectory, the UAV transmission power and the RIS phase shift deflection into a Markov decision process model, updates the elements of the Markov decision process model at each time step, and uses the improved TD3 algorithm to solve the updated model to obtain the UAV movement control instructions and transmission power control instructions at each time step, as well as the phase shift angle control instructions of each RIS component, thereby ensuring that under strict time and energy constraints, the UAV flight path is reasonably arranged, the communication throughput (i.e., providing communication quality) and energy utilization efficiency are maximized; 2. The present invention uses the improved TD3 algorithm to solve the model, introduces the PER (prioritized experience replay) technology on the traditional TD3 algorithm, can improve the UAV's exploration of the environment and the use of effective data, and uses the AO algorithm to optimize the discrete phase shift angle of the RIS to improve the quality of the wireless channel environment. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 A flow chart of the control method for RIS-assisted UAV communication; Figure 2 This is a schematic diagram of the RIS-assisted UAV communication system; Figure 3 It is the structural diagram of TD3-PER-AO algorithm; Figure 4 It is a two-dimensional diagram of the first type of user cluster distribution in a simple barrier-free environment; Figure 5 It is a two-dimensional diagram of the second type of user cluster distribution in a simple barrier-free environment; Figure 6 It is a two-dimensional diagram of the third type of user cluster distribution in a simple barrier-free environment; Figure 7 3D diagram of the first scenario of user cluster distribution in an environment full of obstacles; Figure 8 3D diagram of the second scenario of user cluster distribution in an environment full of obstacles; Fig. 9 3D diagram of the third scenario of user cluster distribution in an environment full of obstacles; Fig.10 Comparison chart of TD3 algorithm with / without PER in barrier-free environment; Fig.11 This is a reward diagram with / without RIS model based on TD3-PER algorithm in an accessible environment; Fig.12 Comparison chart of TD3 algorithm with / without PER in obstacle environment; Fig.13This is the reward diagram with / without RIS model based on TD3-PER algorithm in obstacle environment; Fig.14 A comparison chart of drone energy consumption in different environments and user distribution scenarios; Fig.15 A comparison chart of drone energy efficiency in different environments and user distribution scenarios; Fig.16 This is a comparison chart of the total rate of drones in different environments and user distribution scenarios; Fig.17 Block diagram of the control device for RIS-assisted UAV communication. DETAILED DESCRIPTION
[0018] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. It is obvious that the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.
[0019] Unless otherwise specified, the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure.
[0020] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0021] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.
[0022] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0023] It should be noted that similar symbols and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it does not require further discussion in subsequent figures.
[0024] In order to achieve reasonable arrangement of UAV flight paths, maximize communication throughput (i.e., provide communication quality) and energy utilization efficiency, the present disclosure proposes a RIS-assisted UAV communication control method and related devices, specifically, the joint optimization problem of UAV movement trajectory, UAV transmission power and RIS phase shift angle deflection is converted into a Markov decision process model, and the RIS-assisted UAV communication is controlled by solving the Markov decision process model.
[0025] Figure 1 This is a schematic diagram of an embodiment of the control method of RIS-assisted UAV communication disclosed in the present invention. Figure 1 The embodiment can be executed by a server in a control center.
[0026] like Figure 1 As shown, step 1 of the embodiment uses the three-dimensional coordinates of the drone, the three-dimensional coordinates of the user and the RIS phase shift matrix at the current time step to update the elements of the Markov decision process model; wherein the Markov decision process model is transformed through an optimization problem, and the optimization problem is a problem of jointly optimizing the drone movement trajectory, the drone transmission power and the RIS phase shift deflection, with the goal of improving the drone communication capability and energy efficient data collection.
[0027] It should be noted that the Markov decision process model needs to be built in advance, and the construction process may include: 1) According to the environmental parameters of RIS-assisted UAV communication, the communication model and energy consumption model are constructed.
[0028] The environment in which RIS assists UAV communication can be as follows: like Figure 2 , the communication system is defined as a UAV and several users (i.e., mobile user terminals). The flight altitude of the UAV is fixed at H1. The users are divided into K clusters, each of which has M users. ue The speed of the RIS randomly moves in different communication areas. It is assumed that the RIS is a uniform linear array composed of S elements, which is fixed at a certain position, and several no-fly zones / obstacles are set in the environment.
[0029] It should be noted that there are two types of communication between drones and users: one is direct communication between the two, and the other is indirect communication through RIS. Therefore, the constructed communication model can be defined as follows: Define the three-dimensional coordinates of the drone at time step t as L u (t)=[x u (t),y u (t),H1],x u (t) and y u (t) are L uThe X-axis coordinates and Y-axis coordinates in (t), the three-dimensional coordinates of the m-th user in the k-th user cluster at time step t are L k,m (t)=[x k,m (t),y k,m (t),0],x k,m (t) is L k,m (t) is the x-axis coordinate, y k,m (t) is L k,m (t), then the distance between the drone and the mth user in the kth user cluster at time step t is d u,k,m (t)=|| L k,m (t)- L u (t)||2.
[0030] The channel gain g when the mth user in the kth user cluster communicates directly with the drone u,k,m It can be expressed as: ; Where ρ0 is the channel gain at the reference distance of 1m, α1 is the path loss factor, and d u,k,m is the distance between the drone and the mth user in the kth user cluster.
[0031] Define the three-dimensional coordinates of the RIS fixed position as L r =[x r ,y r ,H2],x r and r L r The X-axis coordinates and Y-axis coordinates in the figure are as follows: H2 is the fixed height of RIS, and the RIS phase shift matrix is represented by , for The phase shift of the element, The phase shift of each element is expressed as a complex number: , without loss of generality, , For the The phase shift angle of each element, .
[0032] The link from the UAV to the user can be indirectly represented by two links from the user to the RIS and from the RIS to the UAV. The two links are modeled using the Rice channel fading model. Then the channel gain g from the UAV to the RIS is r,u , and the channel gain g from the mth user in the kth user cluster to the RIS r,k,m It can be expressed as: ; ; ; ; Where, d r,u is the distance from the UAV to the RIS, d r,k,m is the distance between the user and RIS, α2 and α3 are path loss factors, K1 and K2 are Rice factors, is a random NLoS component, which conforms to a cyclic complex Gaussian distribution, is the LoS component in the link, which can be specifically expressed as: ; ; Where λ and d are the carrier wavelength and the spacing between RIS elements, respectively, and the cosine of the departure angle is , the cosine of the angle of arrival .
[0033] Therefore, the channel gain of the mth user in the kth user cluster when communicating indirectly with the drone is g u,r,k,m =g r,k,m Θg r,u .
[0034] The probability of direct link being blocked is evaluated using the air-to-ground channel model in an urban environment. a and b are two constants determined by environmental factors. The blocking probability p between the UAV and the mth user in the kth user cluster is u,k,m It can be expressed as: .
[0035] Define P as the transmission power of the drone, the total bandwidth as B, and the average bandwidth from the drone to each user as , then the average channel gain h between the UAV and the mth user in the kth user cluster is u,k,m It can be expressed as: ; The communication rate r between the drone and the mth user in the kth user cluster u,k,m It can be expressed as: ; Where σ is additive white Gaussian noise.
[0036] It should be noted that there are two main types of drone energy consumption: one is drone movement energy consumption, and the other is launch energy consumption. Therefore, the energy consumption model constructed can be defined as follows: The blade power and derived power of the drone in the hovering state are defined as P0 and P1 respectively, and the tip speed of the drone rotor blade is U tip, the average rotor induced speed of the drone in the hovering state is v0, and the moving speed of the drone is v, then the moving energy consumption of the drone is E mov It can be expressed as: ; Where d0, ρ, s0 and A are parameters related to the UAV fuselage drag ratio, air density, rotor solidity and rotor disc area, respectively.
[0037] The total energy consumption can be expressed as: E u =E mov +E t ; In the formula, E u is the total energy consumption of the UAV, E t is the launch energy consumption of the UAV, E t =Pδ t , P is the transmission power of the UAV, δ t is the duration of time step t (assuming that the entire communication cycle T is divided into N time steps, and the duration of each step is δ t =T / N).
[0038] 2) Based on the communication model and energy consumption model, an optimization problem is constructed and converted into a Markov decision process model.
[0039] It should be noted that, by analyzing the above model, a joint optimization problem of UAV movement trajectory, UAV transmission power and RIS phase shift deflection is constructed to improve the communication capability and energy-efficient data collection of UAVs. Specifically, it strives to improve the energy efficiency while optimizing the total communication rate of the system to meet the needs of all users in the cluster during the data collection operation, while minimizing the total propulsion energy consumption of the UAV at all time steps.
[0040] The optimization problem can be expressed as: ; st C1:0≤x u (t) ≤L; C2:0≤y u (t) ≤L; C3:L u (0)={(x u ,y u )|(x u ,y u )∈A ini}; C4:L u (N-1)={(x u ,y u )|(x u,y u )∈A des}; C5:|| L u (t-1)- L u (t)||2≤V max δ t ; C6: 0≤P(t) ≤P max ; C7: ; C8: ; Where, L u is the three-dimensional coordinate of the UAV, L is the maximum boundary value of the UAV flight area, A ini , A des are the starting and ending position ranges of the drone, L u (0) and L u (N-1) are the three-dimensional coordinates of the drone at the initial and final time steps, respectively, and L u (t-1) are the three-dimensional coordinates of the UAV at time step t-1, V max is the maximum speed of the UAV, P(t) is the transmission power of the UAV at time step t, and P max is the upper limit of the UAV’s transmission power, E u (t) is the total energy consumption of the drone at time step t, and n means the number of quantization bits (due to the limitation of current hardware capabilities, the phase shift angle is discretized into 2 n corners).
[0041] Among the above constraints, C1 and C2 stipulate that the drone flies within the defined boundaries, C3 and C4 restrict its initial and final positions, C5 sets the maximum speed of the drone flight, C6 limits the power allocation of the user cluster, C7 ensures that the energy consumption of the drone remains within the battery capacity range, and C8 represents the range of the phase shift angle.
[0042] It should be noted that the above optimization problem is transformed into a Markov decision process (MDP) model, in which the state transition is completely dependent on the previous state and is not affected by the entire historical sequence. MDP uses a five-tuple ( S , A , R , P ,γ) to represent, where Ѕ represents the state space, A represents the action space, R represents the reward function, P represents the state transition probability, γ is used as a discount factor, and its value range is 0~1.
[0043] According to the characteristics of the problem, the state space, action space and reward function are summarized as follows: The state space is represented as S ={s t =( S 1, S 2 )}, a series of observations used to characterize the wireless environment, where s t is the state at time step t, S 1 is the two-dimensional coordinate of the UAV at time step t, S 1={x u (t),y u (t)}, S 2 is the distance set between the drone and the user, and between the drone and the destination at time step t, S 2 ={d 1,1 (t),…, d K,M (t), d des (t)},d K,M (t) is the distance between the drone and the Mth user in the Kth user cluster at time step t, d des (t) is the distance between the UAV and the destination at time step t.
[0044] The action space is represented as A ={a t =(θ(t),l(t),P(t)}, where, a t is the action at time step t, θ(t)∈(0,π] is the flight direction of the drone at time step t, l(t)∈[0, V max δ t ] is the flight distance of the UAV at time step t, P(t)∈[0, P max ] is the UAV transmission power at time step t. In addition, each time the UAV performs an action, the RIS controller operates the RIS phase shift component, thereby adjusting the phase shift angle accordingly.
[0045] The reward function is divided into four parts: the communication rate between the drone and the user, the energy consumed by the drone during flight (i.e., E u ), distance-based destination rewards and abnormal penalties, as described below: ; Where R(t) is the reward value at time step t, δ c1 ~δ c4 is a constant, r u,k,m With E uThe ratio of is the energy efficiency, which plays a vital role in the flight of drones. In addition, once the drone collects enough information from the mth user in the kth user cluster, such as r u,k,m ≥r min , r min is the lower limit of the communication rate, the user is marked as completed, thus excluding subsequent visits by the drone.
[0046] R des is the guidance parameter for the UAV to reach its destination, defined as: ; Where, d des (t) is the horizontal distance between the UAV and the destination center at time step t, d des (t-1) is the horizontal distance between the UAV and the destination center at time step t-1, R ser Additional rewards for drones to complete user data collection, mainly to speed up information collection, setting R ser =1, prevent the drone from staying in a certain position range, R pun It is a penalty for abnormal conditions. It is mainly used to prevent the drone from overtime, crossing the boundary, entering the no-fly zone and other abnormal behaviors during flight. pun = -1, and at the same time c4 Set to a larger value and use the termination flag to end the exploration of the current scene when any of the above exceptions are encountered.
[0047] return Figure 1 In step 2 of the embodiment, the TD3 algorithm is used to solve the updated Markov decision process model to obtain the UAV movement control instructions and transmission power control instructions of the current time step, as well as the phase shift angle control instructions of each RIS component; wherein, during training, the TD3 algorithm adjusts the probability of sampling experience data by assigning priority weights to the data in the experience pool; when the TD3 algorithm is used for solving, the AO algorithm is used to find the optimal phase shift angle for the current time step, and the optimal phase shift angle is the phase shift angle that maintains the maximum communication rate between the UAV and the user.
[0048] It should be noted that in order to solve the mixed action space problem of UAV transmission power and trajectory continuous action, as well as RIS discrete phase shift, an improved TD3 algorithm is pre-designed, specifically the TD3-PER-AO algorithm, see Figure 3 The details are as follows: For the UAV control problem, the TD3-PER algorithm is used to optimize the transmission power and trajectory in the process. Specifically, the Prioritized Experience Replay (PER) technology is introduced on the traditional double-delayed deep deterministic policy gradient (TD3).
[0049] In the traditional TD3 algorithm training process, after receiving the state, the drone executes the action specified by the algorithm, and then receives the feedback reward, new state and task state, and stores this data in the experience replay pool. However, the traditional TD3 algorithm uses random sampling, which reduces the correlation between experience data, but reduces the utilization rate of important experience data. The utilization of data can only be guaranteed through sufficient sampling. Here, the PER method is introduced to adjust the probability of sampling experience data by assigning priority to the data in the experience pool, that is, data with higher values are extracted first. Specifically, the priority p is determined by the TD error δ, which is the absolute value of the TD error plus ɛ. The formula is p=|δ|+ɛ. The probability of the jth sample data being sampled is ;in, is the total number of sample data for each sampling, p i and p j are the priorities of the i-th and j-th sample data respectively, is the importance of the priority, and ɛ is a small constant used to prevent the probability from being 0.
[0050] During the training process of this algorithm, the data containing the priorities is stored in PER. At each network update, sampling is performed according to P(j) and the importance sampling (IS) weights w are applied. j to resolve the deviation.
[0051] Actor-main network in TD3-PER algorithm and its target network , Critics Network and its target network The principles for updating parameters are as follows: ; ; ; ; Among them, the critic main network is The loss function is updated using the gradient descent method. Indicates adding a small amount of random noise The action of the target network is to achieve a smoother target strategy. Every κ time steps, the actor network and all target networks are updated according to the above formula. represents the critic network's estimate of the value of the state-action pair, is the strategy of the actor network, which represents the mapping relationship between states and continuous actions. It follows a Gaussian distribution that is clipped to a small range (-c,c), with a mean of 0 and a variance of , is the objective function of the actor network, by maximizing The value is used to update the actor network, Rj is the reward value of the jth sample data, Parameters for soft update of the target network, It means to find the gradient of action a. Find the gradients with respect to the actor network parameters.
[0052] When the TD3 algorithm is used for solving, the phase shift angles in the calculation process are all optimal phase shift angles. Specifically, the AO algorithm is used to find the current optimal phase shift angle. The optimal phase shift angle is the phase shift angle that maintains the maximum communication rate between the drone and the user, that is, the communication rate between the drone and the user is kept at the best at each time step. , is the optimal phase shift angle, is the RIS adopted at the tth time step The communication rate between the drone and the user during the phase, Constraint C8 is satisfied. In each iteration of the AO algorithm, the communication rate is maximized iteratively by optimizing the phase shift of each component one by one. This approach ensures that the overall performance of the system is enhanced by fine-tuning the phase shift to achieve the highest communication rate.
[0053] In each round of the algorithm, the drone executes the actions specified by the TD3 algorithm, namely, flight direction, flight distance, and transmission power. At this time, the RIS controller cooperates with the drone to optimize the RIS phase shift element, obtain the reward, new state, and task state of environmental feedback, store the data in PER, and update the algorithm network. Through continuous iteration of the TD3-PER-AO algorithm, the optimal strategy of the drone is finally obtained, thereby maximizing long-term returns.
[0054] The above method adopts an improved TD3 algorithm to solve the model and introduces PER (Prioritized Experience Replay) technology on the traditional TD3 algorithm, which can improve the UAV's exploration of the environment and the utilization of effective data. The AO algorithm is used to optimize the discrete phase shift angle of RIS to improve the quality of the wireless channel environment.
[0055] In order to verify the above method, the following simulation experiments were conducted: A 1000m×1000m urban area is designed, with user clusters randomly distributed throughout the designated area and users moving randomly. To measure the effectiveness of the method, three different scenarios with different user cluster distributions are designed: a simple scenario includes a single cluster accommodating 4 users, and two more complex scenarios with two clusters located nearby or far away, each accommodating 7 users. In addition, two environments are configured to increase the complexity of the UAV mission, one is an obstacle-free environment that enables the UAV to fly freely over buildings, and the other is a no-fly zone established over specific high-rise buildings, which requires the UAV to adjust its heading when encountering obstacles, thereby increasing the difficulty of the mission. Results are shown in Figure 2. Figures 4 to 9 As shown, Figures 4 to 6 It is a two-dimensional representation of the flight trajectory of the UAV in an obstacle-free environment, wherein the orange area is the starting and ending positions, the gray area is the user clustering area, and the green area is the location of the RIS. The trajectory in the figure is the flight trajectory generated by guiding the UAV using the present invention; Figures 7-9 A more detailed three-dimensional description of the UAV flight trajectory in an obstacle environment, where the red path corresponds to the TD3-PER-AO algorithm, the blue path corresponds to the traditional TD3 algorithm, and the green path corresponds to the TD3-PER algorithm without the RIS model.
[0056] The TD3-AO algorithm and the TD3-PER algorithm are established to verify the performance of the TD3-PER-AO algorithm. Fig.10 and Fig.11 The reward curves of the proposed optimization scheme and two baseline algorithms in an obstacle-free environment are shown. It is obvious that the TD3-PER-AO algorithm shows excellent convergence speed and task performance in different scenarios, verifying the effectiveness of the state space and reward formula. However, in a simplified obstacle-free environment, the performance difference of the algorithms shown by the drone in the task is not very prominent. To solve this problem, no-fly zones / obstacles were introduced in the experiment, which increased the complexity of the environment and enhanced the mobility of the user, thus exacerbating the challenge of drone exploration. Fig.12 and Fig.13 The reward curves of the proposed optimization scheme and two baseline algorithms in an environment full of obstacles are shown. Obviously, the drone cannot fully explore the environment under the guidance of the baseline algorithm and falls into a local optimal state; while under the guidance of the proposed scheme, it avoids obstacles, completes the data collection task, and reaches the destination quickly. Through comparative analysis of two different environments, all three algorithms show that they can complete the assigned tasks in a simple obstacle-free environment, but the training of the proposed algorithm is slightly better than that of the baseline algorithm. In more complex environments, the limitations of the baseline algorithm hinder its ability to fully complete the assigned tasks, while the proposed algorithm shows adaptability by effectively handling the complexity of the environment, thereby achieving the desired goals in the face of these challenges.
[0057] In order to comprehensively evaluate the effectiveness of the TD3-PER-AO algorithm in different environments and user scenarios, Figures 14 to 16 The focus is on the comparative analysis of three key indicators of drone energy consumption, energy efficiency, and total rate. Compared with the obstacle-free environment, the values of all indicators are significantly higher when the proposed algorithm is running in an environment with obstacles, which shows that drones operating in an environment full of obstacles show longer flight distances, higher energy consumption levels, higher environmental complexity, and higher mission difficulty levels.
[0058] The above method converts the joint optimization problem of UAV movement trajectory, UAV transmission power and RIS phase shift deflection into a Markov decision process model, updates the elements of the Markov decision process model at each time step, and uses the TD3 algorithm to solve the updated model to obtain the UAV movement control instructions and transmission power control instructions at each time step, as well as the phase shift angle control instructions of each RIS element, thereby ensuring that the UAV flight path is reasonably arranged under strict time and energy constraints, maximizing the communication throughput (i.e., providing communication quality) and energy utilization efficiency.
[0059] Fig.17 This is a schematic diagram of an embodiment of a control device for RIS-assisted UAV communication disclosed in the present invention. Figure 3 The embodiment is a virtual device that can be loaded and executed by a server in a control center, and includes an update module and a solution module.
[0060] The update module of the embodiment is configured to update the elements of the Markov decision process model using the three-dimensional coordinates of the drone, the three-dimensional coordinates of the user and the RIS phase shift matrix at the current time step; wherein the Markov decision process model is transformed through an optimization problem, and the optimization problem is a problem of jointly optimizing the drone movement trajectory, the drone transmission power and the RIS phase shift angle deflection, with the goal of improving the drone communication capability and energy efficient data collection.
[0061] It should be noted that the Markov decision process model in the update module is pre-constructed, and the construction process may include: constructing a communication model and an energy consumption model according to the environmental parameters of the RIS-assisted UAV communication; constructing an optimization problem according to the communication model and the energy consumption model, and converting the optimization problem into a Markov decision process model.
[0062] The solution module of the embodiment is configured to use the TD3 algorithm to solve the updated Markov decision process model to obtain the UAV movement control instructions and transmission power control instructions of the current time step, as well as the phase shift angle control instructions of each RIS component; wherein, during training, the TD3 algorithm adjusts the probability of sampling experience data by assigning priority weights to the data in the experience pool; when using the TD3 algorithm for solving, the AO algorithm is used to find the optimal phase shift angle for the current time step, and the optimal phase shift angle is the phase shift angle that maintains the maximum communication rate between the UAV and the user.
[0063] Similar to the above method, the above device converts the joint optimization problem of the UAV movement trajectory, the UAV transmission power and the RIS phase shift deflection into a Markov decision process model, updates the elements of the Markov decision process model at each time step, and uses the TD3 algorithm to solve the updated model to obtain the UAV movement control instructions and transmission power control instructions at each time step, as well as the phase shift angle control instructions of each RIS element, thereby ensuring that under strict time and energy constraints, the UAV flight path is reasonably arranged, and the communication throughput (i.e., providing communication quality) and energy utilization efficiency are maximized.
[0064] Based on the same technical solution, the present disclosure also relates to a computer-readable storage medium, which stores one or more programs, and the one or more programs include instructions. When the instructions are executed by a computing device, the computing device executes a control method for RIS-assisted drone communication.
[0065] Based on the same technical solution, the present disclosure also relates to a computer device, including one or more processors and one or more memories, one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing a control method for RIS-assisted drone communication.
[0066] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0067] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0068] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0069] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0070] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.
Claims
1. A control method for RIS-assisted UAV communication, characterized in that: include: The elements of the Markov decision process model are updated by using the three-dimensional coordinates of the UAV, the three-dimensional coordinates of the user and the RIS phase shift matrix at the current time step; wherein the Markov decision process model is transformed through an optimization problem, which is a problem of jointly optimizing the UAV movement trajectory, the UAV transmission power and the RIS phase shift deflection, with the goal of improving the communication capability of the UAV and the data collection of energy efficiency; The TD3 algorithm is used to solve the updated Markov decision process model to obtain the UAV movement control instructions and transmission power control instructions of the current time step, as well as the phase shift angle control instructions of each RIS component; wherein, during training, the TD3 algorithm adjusts the probability of sampling experience data by assigning priority weights to the data in the experience pool; when using the TD3 algorithm for solving, the AO algorithm is used to calculate the optimal phase shift angle for the current time step, and the optimal phase shift angle is the phase shift angle that maintains the maximum communication rate between the UAV and the user.
2. The control method of RIS-assisted UAV communication according to claim 1, characterized in that: The Markov decision process model is pre-built, and the construction process includes: According to the environmental parameters of RIS-assisted UAV communication, the communication model and energy consumption model are constructed; According to the communication model and energy consumption model, an optimization problem is constructed and converted into a Markov decision process model.
3. The control method of RIS-assisted UAV communication according to claim 2, characterized in that: The communication model is: ; In the formula, r u,k,m is the communication rate between the UAV and the mth user in the kth user cluster, B is the total bandwidth, is the average bandwidth from the drone to each user, P is the transmission power of the drone, σ is the additive Gaussian white noise, is the average channel gain between the UAV and the mth user in the kth user cluster, p u,k,m is the blocking probability between the UAV and the mth user in the kth user cluster, g u,k,m is the channel gain when the mth user in the kth user cluster communicates directly with the drone, g u,r,k,m = g r,k,m Θg r,u is the channel gain when the mth user in the kth user cluster communicates indirectly with the drone, g r,k,m is the channel gain from the mth user in the kth user cluster to the RIS, g r,u is the channel gain from UAV to RIS, Θ is the RIS phase shift matrix; The energy consumption model is: AND u =E mov +E t ; In the formula, E u is the total energy consumption of the UAV, E t is the launch energy consumption of the UAV, E mov is the energy consumption of the UAV’s movement, , P0 and P1 are the blade power and derived power of the UAV in hovering state, U tip is the tip speed of the UAV rotor blade, v0 is the average rotor induced speed of the UAV in the hovering state, v is the moving speed of the UAV, d0, ρ, s0 and A are the parameters related to the UAV fuselage drag ratio, air density, rotor solidity and rotor disc area, respectively.
4. The control method for RIS-assisted UAV communication according to claim 3, characterized in that: The optimization problem is: ; s.t. C1:0≤x u (t) ≤L; C2:0≤y u (t) ≤L; C3:L u (0)={(x u ,y u )|(x u ,y u )∈A ini }; C4:L u (N-1)={(x u ,y u )|(x u ,y u )∈A des }; C5:|| L u (t-1)- L u (t)||2≤V max δ t ; C6: 0≤P(t) ≤P max ; C7: ; C8: ; Where, L u is the three-dimensional coordinate of the drone, K is the number of user clusters, M is the number of users in the user cluster, x u (t) is the three-dimensional coordinate L of the UAV at time step t u (t) is the x-axis coordinate, y u (t) is the three-dimensional coordinate L of the UAV at time step t u (t) is the Y-axis coordinate, L is the maximum boundary value of the UAV flight area, and A is ini , A des are the starting and ending position ranges of the drone, L u (0) and L u (N-1) are the three-dimensional coordinates of the drone at the initial and final time steps, respectively, and L u (t-1) are the three-dimensional coordinates of the UAV at time step t-1, V max is the maximum speed of the UAV, δ t is the duration of time step t, P(t) is the transmission power of the UAV at time step t, P max is the upper limit of the UAV’s transmission power, E u (t) is the total energy consumption of the UAV at time step t, For the The phase shift angle of each element, n is the number of quantization bits.
5. The control method for RIS-assisted UAV communication according to claim 1 or 2, characterized in that: In the Markov decision process model: The state space is represented as S ={s t =(S1, S2)}, where s t is the state at time step t, S1 is the two-dimensional coordinate of the drone at time step t, and S2 is the distance set between the drone and the user, and between the drone and the destination at time step t; The action space is represented as A ={a t =(θ(t),l(t),P(t)}, where, a t is the action at time step t, θ(t) is the flight direction of the UAV at time step t, l(t) is the flight distance of the UAV at time step t, and P(t) is the transmission power of the UAV at time step t; The reward function is expressed as: ; ; Where R(t) is the total reward value at time step t, δ c1 ~δ c4 is a constant, r u,k,m is the communication rate between the UAV and the mth user in the kth user cluster, E u is the total energy consumption of the UAV, R des is the guidance parameter for the UAV to reach the destination, d des (t) is the horizontal distance between the UAV and the destination center at time step t, d des (t-1) is the horizontal distance between the UAV and the destination center at time step t-1, R ser The additional reward for the drone to complete the user data collection, R pun Penalty for abnormal conditions.
6. The control method of RIS-assisted UAV communication according to claim 1, characterized in that: The priority of data allocation in the experience pool is the absolute value of the TD error δ plus ɛ. The probability of the jth sample data being sampled is ;in, is the total number of sample data for each sampling, p i and p j are the priorities of the i-th and j-th sample data respectively, is the priority level, and ɛ is a small constant.
7. A control device for RIS-assisted UAV communication, characterized in that: include: An update module uses the drone's 3D coordinates, the user's 3D coordinates, and the RIS phase shift matrix at the current time step to update the elements of the Markov decision process model; the Markov decision process model is transformed through an optimization problem, which is a problem of jointly optimizing the drone's movement trajectory, the drone's transmission power, and the RIS phase shift angle deflection, with the goal of improving the drone's communication capability and energy-efficient data collection; The solution module uses the TD3 algorithm to solve the updated Markov decision process model to obtain the UAV movement control instructions and transmission power control instructions of the current time step, as well as the phase shift angle control instructions of each RIS component; wherein, during training, the TD3 algorithm adjusts the probability of sampling experience data by assigning priority weights to the data in the experience pool; when using the TD3 algorithm for solution, the AO algorithm is used to find the optimal phase shift angle for the current time step, and the optimal phase shift angle is the phase shift angle that maintains the maximum communication rate between the UAV and the user.
8. The control device for RIS-assisted UAV communication according to claim 7, characterized in that: The Markov decision process model in the update module is pre-built, and the construction process includes: According to the environmental parameters of RIS-assisted UAV communication, the communication model and energy consumption model are constructed; According to the communication model and energy consumption model, an optimization problem is constructed and converted into a Markov decision process model.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs include instructions. When the instructions are executed by a computing device, the computing device executes any one of the methods of claims 1 to 6.
10. A computer device, characterized in that: include: One or more processors and one or more memories, one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the methods described in claims 1 to 6.
Citation Information
Patent Citations
6G RIS assisted full duplex unmanned aerial vehicle system energy efficiency optimization method and device
CN115694583A
Unmanned aerial vehicle carried RIS assisted AF relay collaborative construction and optimization method
CN115955264A
Multi-objective optimization method, system and equipment for RIS auxiliary communication
CN116436512A
Energy efficiency optimization method of unmanned aerial vehicle and IRS auxiliary wireless charging edge network
CN118233926A
Cited By
Anti-interference unmanned aerial vehicle-IOS auxiliary wireless communication energy collection method
CN119743782A
Energy efficiency optimization method of unmanned aerial vehicle auxiliary communication system based on deep reinforcement learning
CN121508617A
Energy efficiency optimization method for unmanned aerial vehicle assisted communication system based on deep reinforcement learning
CN121508617B
