Fixed-wing unmanned aerial vehicle communication coverage energy efficiency optimization method based on reflection intelligent surface assistance

Abstract

The invention discloses an energy efficiency optimization method for communication coverage of a fixed-wing unmanned aerial vehicle based on reflection intelligent surface assistance, and the method comprises the steps: S1, building a problem P with maximization of system energy efficiency as a target, and setting a multi-constraint condition; s2, dividing the problem P into two sub-problems; s3, planning the trajectory of the unmanned aerial vehicle by adopting an MICH algorithm to solve the sub-problem I; and S4, firstly, an AS-VRE mechanism is adopted to improve the exploration capability of the DRL, and secondly, an NsHQPN algorithm is adopted to improve the value precision of the hybrid action space. According to the method, the system energy efficiency of the fixed-wing unmanned aerial vehicle for assisting the GN communication can be improved as much as possible under the limited energy condition, and the communication service can be better and more persistently provided for the GN in the emergency communication scene.

Description

Technical Field

[0001] The present invention relates to the field of emerging information technology, and in particular provides an energy efficiency optimization method for fixed-wing unmanned aerial vehicle communication coverage based on the assistance of reflective intelligent surfaces. Background Art

[0002] Unmanned aerial vehicles (UAVs) have attracted much attention due to their high mobility and flexible aerial deployment. They are widely used in aerial inspections, emergency rescue, mobile edge computing and other fields. In terms of wireless network communication, traditional communication methods are difficult to achieve stable and comprehensive signal coverage due to factors such as complex terrain and fixed communication distance. In view of the limitations of traditional communication methods, drones can be used as wireless relay stations or mobile base stations to provide more effective communication services for ground nodes (GNs). However, since drones fly at a certain altitude, the link between them and ground nodes is often blocked by tall buildings, trees, etc. The probability of such blockage increases significantly, especially in urban environments. Due to blockage, the channel gain of the communication link is greatly reduced, thus affecting the communication quality. In recent years, reflective smart surfaces (RIS) based on 5G and 6G can effectively solve the impact of blockage on channel gain by adjusting the reflection angle and phase. In addition, this phase adjustment can also achieve signal reflection, thereby reducing energy consumption, improving communication efficiency and coverage, and maximizing the signal-to-noise ratio. Many research works use RIS to assist UAVs to improve communication gain. Provide a reliable visual link.

[0003] The energy efficiency of the system is greatly affected by the trajectory of the UAV. Therefore, it is necessary to pay attention to the optimization problem of trajectory planning. In UAV-assisted communication, UAVs can be divided into rotary-wing UAVs and fixed-wing UAVs according to their wing structure and power system. Compared with rotary-wing UAVs, the advantages of fixed-wing UAVs are long flight time and large coverage. Since fixed-wing UAVs cannot hover, the steering is completed by the centripetal force generated by the inclination of the fuselage, which will lead to insufficient steering angle and inability to change direction or turn around in time. Therefore, the flight trajectory should be continuous and smooth, not twisted. However, algorithms based on time slot optimization decisions, such as deep reinforcement learning algorithms (DRL), are usually unable to plan such smooth trajectories. In addition, it is also necessary to design trajectories based on the distribution of GNs, so that they are more adapted to the distribution shape of GNs and cover more GNs while reducing the flight distance. In some works, the flight trajectory of fixed-wing UAVs is set to a circle. However, this circular trajectory has the following two disadvantages. First, when the GN distribution is narrow relative to its length, the circular trajectory will have a situation where there are few or no GNs near some flight positions. This results in a waste of setting some flight positions on the circular trajectory. Second, setting a circular trajectory may cause the flight radius of the UAV to be too large, making the UAV too far away from the GN at the center. As a result, the communication quality of the GN at the center will be reduced. Therefore, finding a trajectory design method that is suitable for the distribution of GNs is a major challenge.

[0004] In addition to the flight trajectory of the drone, the flight speed of the drone and the service scheduling of the GN also have a profound impact on the energy efficiency of the system. In some works, the DRL algorithm is often used for optimization. However, the traditional DRL algorithm has two shortcomings. The first is that the exploration of effective experience is limited. It is easy to learn a lot of negative experience in the early stage of learning, which makes the training fall into the local optimum. The second disadvantage is that when the traditional DRL solves the mixed action space (for example, the speed is a continuous action, and the scheduling of the GN is a discrete action), the action value accuracy will be insufficient. This will also cause the decision to fall into the local optimum, resulting in a decrease in the energy efficiency of the system. Therefore, it becomes particularly important to solve the existing defects of the DRL algorithm. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide an energy efficiency optimization method for fixed-wing UAV communication coverage assisted by reflective intelligent surfaces, so as to improve the system energy efficiency of fixed-wing UAV assisted GN communication as much as possible under limited energy conditions. Extend the service time of communication tasks. In emergency communication scenarios, better and more lasting communication services can be provided for GN.

[0006] The technical solution provided by the present invention is: an energy efficiency optimization method for fixed-wing UAV communication coverage assisted by reflective intelligent surfaces, wherein a fixed-wing UAV is used to provide communication coverage for the GN, and a RIS is placed on the outer wall of a ground building for link reflection;

[0007] The energy efficiency optimization method comprises:

[0008] S1: With the goal of maximizing system energy efficiency, construct problem P and set multiple constraints;

[0009] S2: Divide problem P into two sub-problems: Sub-problem 1: Plan a smooth hovering trajectory that can adapt to the distribution of ground nodes; Sub-problem 2: On the planned trajectory, use the DRL algorithm to plan the flight speed of the drone and the service scheduling of the GN to further improve energy efficiency;

[0010] S3: The midpoint iterative convex hull (MICH) algorithm is used to plan the trajectory of the UAV to solve sub-problem 1;

[0011] S4: Solve sub-problem 2: First, the action screening virtual real experience (AS-VRE) mechanism is adopted to improve the exploration ability of DRL, and then the policy network (NsHQPN) algorithm is adopted to improve the accuracy of the value of the mixed action space.

[0012] Specifically, the problem P is described in S1 as:

[0013]

[0014]

[0015]

[0016]

[0017]

[0018] (1) indicates that the goal is to maximize the system energy efficiency. Under this goal, there are constraints (1a)-(1d); (1a) indicates that the service state of each GN in each time slot can only be the served state and the unserved state, which are recorded as 1 and 0 respectively; (1b) indicates that in a time slot, the drone can only serve one GN, which is the constraint of Time Division Multiple Access (TDMA); (1c) indicates that the cumulative amount of data transmitted to each GN must be greater than the amount of data it requested (1d) represents the upper and lower limits of the flight speed of the fixed-wing UAV.

[0019] Specifically, the MICH algorithm described in S3 includes:

[0020] First, several key trajectory nodes are solved according to the distribution of GN;

[0021] Secondly, trajectory fitting is performed on the key trajectory nodes to make the trajectory curve smoother to adapt to the special direction-changing constraints of the fixed-wing UAV.

[0022] Specifically, the trajectory of a fixed-wing drone is expressed by the following equation:

[0023]

[0024] where a 0 ∈[0,3000], b 0 ∈[0,3000],x 0 ∈[0,3000],y 0 ∈[0,3000], and θ∈[-π,π] represent the unknown parameters of the fitting curve;

[0025] The predicted data is calculated according to the defined fitting curve function. The calculation process formula is as follows:

[0026]

[0027] Finally, the loss equation is used for gradient descent optimization to obtain specific parameters;

[0028] Construct a Markov decision process:

[0029] State space: The job of the fixed-wing UAV is to meet the communication needs of all GNs. Since the distance between the UAV and the selected GN directly affects the communication quality, a state space is defined The position of the UAV is Indicates that K reflects the service status of all GNs. is a binary variable indicating whether the service to the GN is completed in the current time slot. When The UAV continues to provide communication services to the GN. When , it means that the GN has completed the service and no longer needs to provide services in this round; the service status of all GNs can be mapped by using the binary list to calculate K, that is, Thus, it is determined whether the task for each GN has been completed;

[0030] Action space: The action space A = {v, Θ} contains the flight speed v of the UAV and the service schedule Θ of the GN. In the k time slot, the specific action a(k) is defined as the flight speed v[k] and the service schedule Θ in the current time slot, that is, The flight speed v[k] of the UAV is limited to [v max ,vmin ]; the GN selected in the k time slot is used express;

[0031] Reward function: The DRL algorithm is used to evaluate the action a(k) performed by the UAV in state s(k). Therefore, the reward r k It can be expressed as

[0032]

[0033] The data transmission rate in time slot k is The energy consumed by the fixed-wing UAV in the k time slot is When selecting a GN that has served again, a penalty term is introduced

[0034] Specifically, the AS-VRE mechanism is used to optimize DRL:

[0035] The AS-VRE mechanism consists of two basic processes: the action screening process and the process of combining virtual and real experience;

[0036] First, the state observation is input into the neural network, and the output of the neural network evaluates the rewards of all possible actions;

[0037] Perform the action screening step to identify and exclude all illegal actions; when selecting actions, select the action with the highest reward value from the actions that meet the requirements as the action to be executed, which is the screening process;

[0038] Introduce some virtual experience into the experience pool, that is, the experience of illegal actions that have not been executed and have low reward values, so that the neural network can fully learn the experience of illegal actions;

[0039] Add virtual rewards for illegal actions v , so that the reward for illegal actions is lower than the reward for compliant actions; the definition of virtual rewards is as follows:

[0040] r v =min(Q(s(k+1),a uc ))-1. (5)

[0041] Violating actions are marked as a uc ; Since the illegal action is not executed when the illegal action is selected, r v It does not constitute real action value.

[0042] Specifically, the AS-VRE mechanism is as follows: the neural network scores each action according to the input state; then, the neural network determines the illegal actions, screens out the actions that meet the requirements, and selects the action with the largest reward from the actions that meet the requirements to execute; the experience after execution is added to the experience pool; at the same time, if an illegal action is encountered, the virtual experience that has not been actually executed needs to be added to the experience pool; finally, a batch of experiences are selected from the experience pool for neural network learning;

[0043] The NsHQPN algorithm is used to optimize the decision-making of speed and GN service scheduling:

[0044] The action set consists of discrete action set C and continuous action set A c Among them, the action a(k) can be expressed as a(k)=(c(k),a c (k)), where c(k) represents a discrete action, c∈C, a c (k) is the continuous action corresponding to the discrete action c(k), a c ∈A c The action a(k) taken in state s(k) is expressed by the state-action value function Q(s(k), c(k), a c (k)) to evaluate; the Bellman equation is expressed as:

[0045]

[0046] Where γ∈(0,1] represents the discount factor;

[0047] Calculate the optimal continuous action corresponding to each discrete action and use the policy network to make the optimal decision;

[0048]

[0049] Among them, pol(·) is a deterministic policy function used to select a continuous action set A from c Select the best action a c ; The parameter set of the policy network is represented by Φ;

[0050] The Q network then makes decisions based on the continuous actions and determines the discrete actions;

[0051]

[0052] In each time slot, the agent starts from the current state s(k), accumulates the rewards obtained by performing N steps of actions, and takes the Q value of the state s(k+N) reached after these N steps of actions as the target expected value y k , used to update the Q value of state s(k) and action a(k);

[0053]

[0054] Where N represents the number of steps from the current state to the target state; the parameter set of the Q network is represented by Ψ;

[0055] Define the loss function used to update the network parameters Φ and Ψ:

[0056]

[0057] The present invention provides an energy efficiency optimization method for communication coverage of fixed-wing UAVs assisted by reflective intelligent surfaces, which solves the energy efficiency optimization problem of RIS-assisted fixed-wing UAVs that provide communication services for GNs in urban environments. The method solves the flight trajectory planning problem of fixed-wing UAVs based on the MICH algorithm of Graham scanning. The algorithm can consider the distribution shape of GNs to better cover GNs. The present invention proposes to place the AS-VRE mechanism at the output end of the neural network to solve the problem that taboos cannot be added in the action space in the DRL algorithm. The present invention also proposes an NsHQPN algorithm that can process mixed action spaces to solve the problem that the traditional DRL algorithm falls into local optimality due to insufficient action accuracy when processing mixed action spaces. Experiments show that the MICH algorithm, AS-VRE mechanism and NsHQPN algorithm proposed in the present invention can effectively improve the energy efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments:

[0059] Figure 1 A system scenario diagram provided by the present invention;

[0060] Figure 2 It is a schematic diagram of the solution process of the problem P of the present invention;

[0061] Figure 3 Schematic diagram of the solution of the MICH algorithm of the present invention;

[0062] Figure 4 Schematic diagram of the AS-VRE mechanism of the present invention;

[0063] Figure 5 It is the principle diagram of the NsHQPN algorithm of the present invention;

[0064] Figure 6 is the average reward for reconstructing the performance comparison of the smart surface in the embodiment of the present invention;

[0065] Figure 7 is the energy efficiency CDF of the reconfigurable smart surface performance comparison in the embodiment of the present invention;

[0066] Figure 8 is a comparison of different trajectories in the embodiments of the present invention;

[0067] Fig. 9 is the average reward of different trajectories in the embodiment of the present invention;

[0068] Fig.10 is the energy efficiency CDF of different trajectories in the embodiment of the present invention;

[0069] Fig.11 is the average reward of different mechanisms in the embodiment of the present invention;

[0070] Fig.12 is the energy efficiency CDF of different mechanisms in the embodiments of the present invention;

[0071] Fig.13 is the average reward of different DRL algorithms in the embodiment of the present invention;

[0072] Fig.14 : is the energy efficiency CDF of different DRL algorithms in the embodiments of the present invention. DETAILED DESCRIPTION

[0073] The present invention will be further explained below in conjunction with specific implementation schemes, but the present invention is not limited thereto.

[0074] This implementation scheme provides an energy efficiency optimization method for fixed-wing UAV communication coverage based on reflective intelligent surface assistance. The scene diagram is as follows: Figure 1 As shown, a fixed-wing UAV is used to provide communication coverage for the GN, wherein in order to avoid the influence of obstacles on the communication link channel gain, a RIS is placed on the outer wall of the ground building for link reflection. The purpose of the present invention is to maximize the system energy efficiency. The problem description is as follows:

[0075]

[0076] (1) indicates that the goal is to maximize the system energy efficiency. Under such a goal, there are so many constraints (1a)-(1d). (1a) The service state of each GN in each time slot can only be the served state and the unserved state, which are recorded as 1 and 0 respectively. (1b) In a time slot, the drone can only serve one GN, which is the constraint of time division multiple access technology (TDMA). (1c) The cumulative amount of data transmitted by each GN must be approximately the amount of data requested by it. (1d) represents the upper and lower limits of the flight speed of the fixed-wing UAV.

[0077] In problem P, the speed of the drone and the service scheduling of the GN have a profound impact on the energy efficiency of the system. However, due to the particularity of the flight trajectory of fixed-wing drones, that is, the steering of fixed-wing drones is maintained by the centripetal force provided by the tilt of the drone body, so its trajectory is smooth. However, algorithms such as DRL that optimize based on time slots will distort the trajectory. It cannot be truly implemented in actual situations. Therefore, problem P is divided into two sub-problems to solve. Sub-problem 1: Plan a smooth circling trajectory that can adapt to the distribution of ground nodes to avoid unnecessary path loss and increase propulsion energy consumption, thereby improving system energy efficiency. Sub-problem 2: On the planned trajectory, use the DRL algorithm to plan the flight speed of the drone and the service scheduling of the GN to further improve energy efficiency. However, in sub-problem 2, the use of the traditional DRL algorithm will cause the energy efficiency value to fall into the local optimum due to the low value accuracy of the mixed action space and the low ability to explore effective experience.

[0078] Therefore, in order to make a more energy-efficient decision for problem P, the present invention proposes a MICH algorithm for trajectory planning for sub-problem 1. In view of the two defects of DRL in sub-problem 2, the AS-VRE mechanism is first proposed to improve the exploration ability of DRL, and then the N-step hybrid deep Q and policy network (NsHQPN) algorithm is proposed to improve the value accuracy of the hybrid action space. The schematic diagram of the solution process of problem P is shown in Figure 2 shown.

[0079] MICH algorithm: Considering the smoothness of the fixed-wing UAV trajectory and the communication task of energy efficiency optimization, the trajectory needs to fit the distribution of GN as much as possible while reducing the path loss. Therefore, the MICH algorithm is proposed. The schematic diagram of the solution of the MICH algorithm is as follows Figure 3 As shown. It is divided into two major steps. First, several key trajectory nodes are solved according to the distribution of GN. Secondly, trajectory fitting is performed on these key trajectory nodes to make the trajectory curve smoother and able to adapt to the special direction-changing constraints of fixed-wing UAVs. The specific steps are shown in Algorithm 1.

[0080]

[0081]

[0082] The UAV trajectory is represented by the following functional equation:

[0083]

[0084] where a 0 ∈[0,3000], b 0 ∈[0,3000],x 0 ∈[0,3000],y 0∈[0,3000], and θ∈[-π,π] represent the unknown parameters of the fitting curve. The predicted data is calculated according to the defined fitting curve function. The calculation process formula is as follows:

[0085]

[0086] Finally, the loss equation is used for gradient descent optimization to obtain specific parameters.

[0087] Construction of Markov Decision Process:

[0088] State Space: The job of the fixed-wing UAV is to meet the communication needs of all GNs. Since the distance between the UAV and the selected GN directly affects the communication quality, a state space is defined The position of the UAV is Indicates that K reflects the service status of all GNs. GN service status is a binary variable indicating whether the service to the GN is completed in the current time slot. When , it means that the ιth GN has not completed the service, and the UAV can continue to provide communication services for it. When , it means that the GN has completed the service and no longer needs to provide services in this round. The service status of all GNs can be mapped by using a binary list to calculate K, that is, Thereby determining whether the task for each GN has been completed.

[0089] Action space: The action space A = {v, Θ} contains the flight speed v of the UAV and the service schedule Θ of the GN. In the k time slot, the specific action a(k) is defined as the flight speed v[k] and the service schedule Θ in the current time slot, that is, The flight speed v[k] of the UAV is limited to [v max ,v min ] range. The GN selected in the k time slot is used express.

[0090] Reward function: In communication systems, energy efficiency is an important indicator for evaluating system performance. It is defined as the ratio between the amount of data transmitted and the energy consumed. In order to maximize the energy efficiency of fixed-wing UAV-assisted wireless communication systems, the DRL algorithm is adopted and energy efficiency is used as a reward function to evaluate the action a(k) performed by the UAV in state s(k). Therefore, the reward r k It can be expressed as

[0091]

[0092] The data transmission rate in time slot k is The energy consumed by the fixed-wing UAV in the k time slot is When selecting a GN that has served again, a penalty term is introduced

[0093] AS-VRE mechanism: The AS-VRE mechanism aims to improve the efficiency of the agent's exploration of the environment so that the deep neural network (DNN) can learn better. Each GN has different data service requirements, and the UAV needs to avoid repeatedly providing communication services to GNs that have already been served. To solve this problem, action space planning is introduced in the DRL algorithm to optimize the scheduling of GNs.

[0094] In a round, each time slot will filter the actions that meet the requirements in the action space, and prohibit the filtered illegal actions in this round, so as to effectively avoid re-serving the GN that has met the requested data volume. The illegal actions are marked as a uc . After the action screening (AS) mechanism, the experiences available for learning in the experience pool are all compliant and positive, so DNN will not learn negative experiences. There are two problems with only the screening mechanism: First, since DNN will not learn negative experiences, it will not give low scores to non-compliant actions. In fact, for DNN, giving low scores to non-compliant actions is as important as giving high scores to compliant actions. This effectively avoids negative factors, allowing the agent to learn better. Secondly, simply screening illegal actions will not affect DNN's scoring of illegal actions, because this is just an avoidance process. In addition, under the influence of the screening mechanism, DNN has no difference in the score of illegal actions. The only identifier of this difference is the state value K. The setting of the state value K is invalid because it is meaningless. This confusion occurs in the process of DNN scoring actions, resulting in poor training results and falling into local optimality. Therefore, it is also necessary to add some virtual and bad experiences that have not been actually executed to the experience pool for the agent to learn.

[0095] Therefore, a virtual reward r is added for the illegal action v , so that the reward for illegal actions is lower than the reward for legal actions. In this way, the learning effect of the neural network can be improved during the training process. The definition of virtual reward is as follows:

[0096] r v =min(Q(s(k+1),a uc ))-1. (5)

[0097] Since the illegal action is not executed when the illegal action is selected, r v It does not constitute real action value.

[0098] The schematic diagram of the AS-VRE mechanism is shown in Figure 4As shown in Figure 1, its core contains two basic processes: the action screening process and the process of combining virtual and real experience. First, the state observation value is input into the neural network, and the output of the neural network will evaluate the rewards of all possible actions. Next, the action screening step is performed to identify and exclude all illegal actions. When selecting an action, the action with the highest reward value is selected from the actions that meet the requirements as the action to be executed. This process is called the screening process. However, since the action screening process limits the experience pool and only includes the experience of compliant actions, this may cause the neural network to be unable to fully learn the experience of illegal actions. In order to make up for this deficiency, it is also necessary to introduce a part of virtual experience (that is, the experience of illegal actions that have not been executed and have low reward values) into the experience pool. In this way, the neural network can be exposed to both compliant and non-compliant experiences during the learning process, so as to have a more comprehensive understanding of the environment and make more accurate decisions.

[0099] NsHQPN algorithm: NsHQPN algorithm is a DRL algorithm that can handle mixed action spaces. NsHQPN algorithm aims to separate discrete actions and continuous actions and optimize them separately using Q network and policy network. Subsequently, the results of these separate optimizations are integrated to obtain the optimization result of the overall strategy.

[0100] The action set consists of discrete action set C and continuous action set A c Among them, the action a(k) can be expressed as a(k)=(c(k),a c (k)), where c(k) represents a discrete action, c∈C, a c (k) is the continuous action corresponding to the discrete action c(k), a c ∈A c The action a(k) taken in state s(k) is expressed by the state-action value function Q(s(k), c(k), a c (k)) to evaluate. The Bellman equation is expressed as:

[0101]

[0102] Where γ∈(0,1] represents the discount factor.

[0103] In the NsHQPN algorithm, it is usually necessary to first calculate the optimal continuous action corresponding to each discrete action to achieve the decision-making process of mixed actions. This process is achieved by using a policy network to make the optimal decision.

[0104]

[0105] Among them, pol(·) is a deterministic policy function used to select a continuous action set A from c Select the best action a cThe parameter set of the policy network is denoted by Φ. The Q network then makes decisions based on the continuous actions and determines the discrete actions.

[0106]

[0107] In each time slot, the agent starts from the current state s(k), accumulates the rewards obtained by performing N steps of actions, and takes the Q value of the state s(k+N) reached after these N steps of actions as the target expected value y k , used to update the Q value of state s(k) and action a(k).

[0108]

[0109] Where N is the number of steps from the current state to the target state. The parameter set of the Q network is denoted by Ψ.

[0110] Define the loss function used to update the network parameters Φ and Ψ:

[0111]

[0112] The principle diagram of the NsHQPN algorithm is as follows Figure 5 As shown. The neural network scores each action according to the input state. Subsequently, it needs to judge the illegal actions, filter out the actions that meet the requirements, and select the action with the largest reward from the actions that meet the requirements to execute. The experience after execution must be added to the experience pool. At the same time, if an illegal action is encountered, the virtual experience that is not actually executed needs to be added to the experience pool. Finally, a batch of experience is selected from the experience pool for neural network learning. The detailed algorithm flow is as follows:

[0113]

[0114]

[0115] With the assistance of RIS, the present invention uses the MICH algorithm to plan the flight trajectory of fixed-wing UAVs. The AS-VER mechanism and the NsHQPN algorithm are used to plan the speed of UAVs and the scheduling of GNs to improve communication services for GNs. Four sets of simulation experiments were conducted to verify the performance of the proposed scheme. First, the impact of RIS on the system energy efficiency was compared. Secondly, the impact of three different fixed-wing UAV flight trajectories on the system energy efficiency was compared. Then, the impact of the AS-VRE mechanism on the experimental results of the DRL algorithm was compared. Finally, the NsHQPN algorithm was compared with the traditional DRL algorithm.

[0116] Comparison of RIS performance: The specific impact of RIS on reward and energy efficiency in UAV-assisted communication systems is explored. Figure 6 and Figure 7As shown in the figure, under the same flight trajectory and algorithm conditions, the UAV system equipped with RIS significantly improves the communication effect. RIS reflects the communication signal by adjusting the phase shift, avoiding obstacles from blocking the link between the UAV and the GN, thereby improving the channel gain and system energy efficiency. Figure 6 and Figure 7 The reward and energy efficiency data in verify that the energy efficiency of UAV systems equipped with RIS is much higher than that of systems without RIS.

[0117] Comparison of different trajectories: The impact of different UAV trajectories on the experiment was compared. Trajectories planned by different algorithms are as follows: Figure 8 As shown in the figure, it can be seen that the circular trajectory solution is less favorable. Since the GNs are distributed in a narrow shape, the circular trajectory causes a lot of waste in the trajectory position setting. A large part of the trajectory positions are set in places where GNs are sparsely distributed or even where there are no GNs. Many trajectory positions are set in places where GNs are sparsely distributed or even where there are no GNs. Fig. 9 and Fig.10 The rewards and energy efficiency in can also be confirmed, and this trajectory setting cannot obtain too high reward returns and energy efficiency returns. Then, the trajectory planned by the CH scheme has a characteristic, that is, it can surround all GNs as much as possible. This surround is also set relative to the outermost GN, so there is still a certain distance from the center part of the GN. Therefore, its reward and energy efficiency are slightly higher than the circular trajectory. However, it still cannot reach the optimal state. Compared with the trajectory planned by the CH algorithm and the circular trajectory, the trajectory planned by the MICH algorithm can continue to shrink toward the center on the basis of the CH algorithm, reduce the average distance between the UAV and the GN, reduce the path loss of the UAV flight, and increase the channel gain. Therefore, this scheme is superior to the previous two schemes in terms of reward returns and energy efficiency returns.

[0118] Comparison of different mechanisms: This implementation compares the performance of AS-VRE, AS mechanism and traditional penalty DRL. Fig.11 and Fig.12It can be seen that the performance of traditional penalty learning is the worst, and DRL using the AS mechanism is second. DRL using the AS-VRE mechanism has the best performance. The reason is that traditional penalty learning is difficult to explore effective action behaviors during the exploration process due to the large solution space. Therefore, after a period of training, it will fall into a local optimal state. Then, the AS mechanism can remove illegal actions through action screening, thereby significantly improving the probability of selecting effective action behaviors during the exploration process and converting them into better reward returns. The AS-VRE mechanism adds a virtual experience mechanism on the basis of the AS mechanism. This false experience is a bad experience with low reward returns. These false experiences may not be executed. In other words, it not only tells the agent to find effective decisions as much as possible, but also tells the agent which behaviors are bad and not to choose, thereby improving the efficiency of the DRL exploration process from both positive and negative aspects. Therefore, DRL under the AS-VRE mechanism is also the best.

[0119] Comparison of different DRL algorithms: This implementation evaluates the performance of four algorithms, NsHQPN, HQPN, Dueling DQN and DDPG, in dealing with mixed action space problems. Compared with NsHQPN and HQPN, Dueling DQN and DDPG face the challenge of insufficient action acquisition accuracy when dealing with such problems. Since Dueling DQN discretizes continuous actions, when the size of the discretized action is small, the action space will expand dramatically, affecting the efficiency of the algorithm. At the same time, if the size of the discretized action is too large, the accuracy of the action will be reduced. When processing discrete actions, DDPG may encounter some situations where the values ​​do not make sense. If you need to make it meaningful, you need to round it off, thereby reducing the accuracy. From Fig.13 and Fig.14 As can be seen from the figure, in the exploration phase, there is no significant difference in the performance of the four algorithms. However, after a period of training, the convergence effects of DuelingDQN and DDPG algorithms are not as good as those of NsHQPN and HQPN. In addition, Fig.13 and Fig.14 The reward and energy efficiency data in verify that NsHQPN is superior to HQPN in both convergence speed and height. Part of the reason for this phenomenon is that the one-step update mechanism is insufficient in dealing with delayed rewards and cannot fully utilize existing data, resulting in poor convergence. In contrast, the N-step update strategy adopted by NsHQPN can more effectively utilize information from multiple states, improve sample utilization, and reduce deviations in the update process. This mechanism enables the algorithm to better perform long-term planning, making training more stable and converging more efficiently.

[0120] The detailed description of the present invention is written in a progressive manner, emphasizing the differences between the various implementations, and similar parts thereof can be referred to each other.

[0121] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. An energy efficiency optimization method for fixed-wing UAV communication coverage based on reflective smart surface assistance, characterized in that: Use fixed-wing drones to provide communication coverage for GN, and place RIS on the outer walls of ground buildings for link reflection; The energy efficiency optimization method comprises: S1: With the goal of maximizing system energy efficiency, construct problem P and set multiple constraints; S2: Divide problem P into two sub-problems: Sub-problem 1: Plan a smooth hovering trajectory that can adapt to the distribution of ground nodes; Sub-problem 2: On the planned trajectory, use the DRL algorithm to plan the flight speed of the drone and the service scheduling of the GN to further improve energy efficiency; S3: Use the MICH algorithm to plan the trajectory of the UAV to solve sub-problem 1; S4: Solve sub-problem 2: First, adopt the AS-VRE mechanism to improve the exploration ability of DRL, and then adopt the NsHQPN algorithm to improve the accuracy of the value of the mixed action space.

2. The energy efficiency optimization method for fixed-wing UAV communication coverage based on reflective intelligent surface assistance according to claim 1 is characterized in that: The problem P is described in S1 as: (1) indicates that the goal is to maximize the system energy efficiency. Under this goal, there are constraints (1a)-(1d); (1a) indicates that the service state of each GN in each time slot can only be the served state and the unserved state, which are recorded as 1 and 0 respectively; (1b) indicates that in a time slot, the UAV can only serve one GN, which is the constraint of TDMA; (1c) indicates that the cumulative amount of data transmitted to each GN must be greater than the amount of data requested by it σ ι ; (1d) represents the upper and lower limits of the flight speed of the fixed-wing UAV.

3. The energy efficiency optimization method for fixed-wing UAV communication coverage based on reflective intelligent surface assistance according to claim 1 is characterized in that: The MICH algorithm described in S3 includes: First, several key trajectory nodes are solved according to the distribution of GN; Secondly, trajectory fitting is performed on the key trajectory nodes to make the trajectory curve smoother to adapt to the special direction-changing constraints of the fixed-wing UAV.

4. The energy efficiency optimization method for fixed-wing UAV communication coverage based on reflective intelligent surface assistance according to claim 1 is characterized in that: The trajectory of a fixed-wing drone is expressed by the following equation: where a0∈[0,3000], b0∈[0,3000], x0∈[0,3000], y0∈[0,3000], and θ∈[-π,π] represent the unknown parameters of the fitting curve; The predicted data is calculated according to the defined fitting curve function. The calculation process formula is as follows: Finally, the loss equation is used for gradient descent optimization to obtain specific parameters; Construct a Markov decision process: State space: The job of the fixed-wing UAV is to meet the communication needs of all GNs. Since the distance between the UAV and the selected GN directly affects the communication quality, a state space is defined The position of the UAV is Indicates that K reflects the service status of all GNs. The service status of GN K ι (k) is a binary variable indicating whether the service to the GN is completed in the current time slot. ι When (k) = 0, it means that the ιth GN has not completed the service, and the UAV continues to provide communication services for it. On the contrary, when K l When (k) = 1, it means that the GN has completed the service and no longer needs to provide services in this round; the service status of all GNs can be mapped by using a binary list to calculate K, that is, Thus, it is determined whether the task for each GN has been completed; Action space: The action space A = {v, Θ} contains the flight speed v of the UAV and the service schedule Θ of the GN. In the k time slot, the specific action a(k) is defined as the flight speed v[k] and the service schedule Θ in the current time slot, that is, The flight speed v[k] of the UAV is limited to [v max ,v min ]; the GN selected in the k time slot is used express; Reward function: The DRL algorithm is used to evaluate the action a(k) performed by the UAV in state s(k). Therefore, the reward r k It can be expressed as The data transmission rate in time slot k is The energy consumed by the fixed-wing UAV in the k time slot is When selecting a GN that has served again, a penalty term is introduced 5. The energy efficiency optimization method for fixed-wing UAV communication coverage based on reflective intelligent surface assistance according to claim 1 is characterized in that: Using AS-VRE mechanism to optimize DRL: The AS-VRE mechanism consists of two basic processes: the action screening process and the process of combining virtual and real experience; First, the state observation is input into the neural network, and the output of the neural network evaluates the rewards of all possible actions; Perform the action screening step to identify and exclude all illegal actions; when selecting actions, select the action with the highest reward value from the actions that meet the requirements as the action to be executed, which is the screening process; Introduce some virtual experience into the experience pool, that is, the experience of illegal actions that have not been executed and have low reward values, so that the neural network can fully learn the experience of illegal actions; Add virtual rewards for illegal actions v , so that the reward for illegal actions is lower than the reward for compliant actions; the definition of virtual rewards is as follows: r v =min(Q(s(k+1),a uc ))-1. (5) Violating actions are marked as a uc ; Since the illegal action is not executed when the illegal action is selected, r v It does not constitute real action value.

6. The energy efficiency optimization method for fixed-wing UAV communication coverage based on reflective intelligent surface assistance according to claim 1 is characterized in that: The NsHQPN algorithm is as follows: the neural network scores each action according to the input state; then, the neural network determines the illegal actions, screens out the actions that meet the requirements, and selects the action with the largest reward from the actions that meet the requirements to execute; the experience after execution is added to the experience pool; at the same time, if an illegal action is encountered, it is necessary to add the virtual experience that has not been actually executed to the experience pool; finally, a batch of experiences are selected from the experience pool for neural network learning; The NsHQPN algorithm is used to optimize the decision-making of speed and GN service scheduling: The action set consists of discrete action set C and continuous action set A c Among them, the action a(k) can be expressed as a(k)=(c(k),a c (k)), where c(k) represents a discrete action, c∈C, a c (k) is the continuous action corresponding to the discrete action c(k), a c ∈A c The action a(k) taken in state s(k) is expressed by the state-action value function Q(s(k), c(k), a c (k)) to evaluate; the Bellman equation is expressed as: (c(k),a c (k))), (6) Where γ∈(0,1] represents the discount factor; Calculate the optimal continuous action corresponding to each discrete action and use the policy network to make the best decision; Among them, pol(·) is a deterministic policy function used to select a continuous action set A from c Select the best action a c ; The parameter set of the policy network is represented by Φ; The Q network then makes decisions based on the continuous actions and determines the discrete actions; In each time slot, the agent starts from the current state s(k), accumulates the rewards obtained by performing N steps of actions, and takes the Q value of the state s(k+N) reached after these N steps of actions as the target expected value y k , used to update the Q value of state s(k) and action a(k); Where N represents the number of steps from the current state to the target state; the parameter set of the Q network is represented by Ψ; Define the loss function used to update the network parameters Φ and Ψ:

Images