Data collection design method of intelligent reflection surface assisted unmanned aerial vehicle

Through intelligent reflective surface assistance, the combined optimization of the flight trajectory of the drone and the scheduling decision of the sensor node and the reflective phase shift of the intelligent reflective surface, the problem of data collection channel fading of the drone in complex environments is solved, and the sensor node is realized The freshness of data is guaranteed and the communication quality is improved.

CN119071812APending Publication Date: 2024-12-03NINGBO UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411002559.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In complex environments, the channel fading problem caused by obstacles during data collection of drones leads to a decline in communication quality, making it difficult to ensure the data freshness of sensor nodes.

Method used

Through intelligent reflective surface assistance, the flight trajectory of the drone, the scheduling decision of the sensor nodes and the reflective phase shift vector of the intelligent reflective surface are jointly optimized, and the MDP model is constructed and the PPO-RAC algorithm is used to optimize the flight trajectory and intelligence of the drone. The phase shift of the reflective surface to minimize the average information age of the sensor node.

Benefits of technology

While ensuring the communication data quality of sensor nodes, the average information age of sensor nodes is significantly reduced, keeping the data fresh, and improving the communication quality of drones.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071812A_ABST
    Figure CN119071812A_ABST
Patent Text Reader

Abstract

The invention discloses a data collection design method for an intelligent reflection surface-assisted unmanned aerial vehicle, and the method comprises the steps: enabling the average information age of all sensor nodes to be minimum, optimizing the trajectory of the unmanned aerial vehicle, and carrying out the association of the sensor nodes and a phase shift control strategy of the intelligent reflection surface in a combined manner; the intelligent reflecting surface is provided with a small number of active reflecting elements for sampling cascade channel states to form partial estimated channel states, and the phase shift of the intelligent reflecting surface is adjusted according to the sampled channel states; an enhanced near-end strategy optimization method is provided, a redesigned Actor-Critic network framework is integrated, the framework comprises a Critic network and two Actor networks, one Actor network is used for estimating the phase shift of an intelligent reflection surface, and the other Actor network is used for optimizing the flight path of an unmanned aerial vehicle and the association between the flight path and sensor nodes; the method has the advantages that the average information age of all the sensor nodes is minimized while the communication data quality of each sensor node is ensured, so that the data is kept fresh.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a data collection technology, and in particular to a data collection design method for an unmanned aerial vehicle (UAV) assisted by an intelligent reflective surface (IRS). Background Art

[0002] In recent years, with the rapid development of 5G mobile communication technology, the Internet of Things (IoT) technology represented by Wireless Sensor Networks (WSNs) has been widely used in various fields of digital economy and life. In wireless sensor networks, timely and reliable collection of status information is the basis of network operation, and ensuring the freshness of the collected data is very critical in some delay-sensitive applications (for example, autonomous driving).

[0003] Unmanned Aerial Vehicle (UAV) is usually used as a mobile data collector to efficiently collect the perception data of ground sensor nodes due to its flexibility, maneuverability and ease of operation. In recent years, with the rapid development of drone technology, people's demand for online data collection has continued to increase. However, although the existing drone systems perform well in some aspects, there are still some challenges in complex environments and special conditions. For example, in a complex environment with many obstacles such as buildings and trees, the drone air-to-ground wireless communication channel will suffer from shadow fading and small-scale fading, resulting in severe path loss and greatly reducing the strength of the received signal.

[0004] As a new type of 6G communication technology, Intelligent Reflective Surfaces (IRS) technology presents great application potential. By controlling the phase shift of each element of the intelligent reflective surface and intelligently reflecting the incident signal, the wireless propagation environment can be reconfigured. Therefore, intelligent reflective surfaces can be used to enhance the communication quality of drones during data collection. Usually, the age of information (AoI) is used as a standard to measure the freshness of data. The drone technology is combined with the intelligent reflective surface technology to ensure the freshness of the sensor node's perception data.

[0005] With the characteristics of IRS reconfigurable wireless communication environment, using IRS to enhance the communication performance of UAVs has become a major research hotspot recently. Some literatures consider an IRS-assisted UAV communication downlink transmission system, and propose a continuous convex approximation algorithm to jointly optimize the flight trajectory of the UAV and the reflection phase shift of the IRS to maximize the average achievable rate. Some literatures consider the scenario of IRS assisting UAVs to serve multiple ground users, and propose an alternating optimization algorithm to jointly optimize the resource allocation strategy, the flight trajectory of the UAV and the reflection phase shift of the IRS to minimize the average total power consumption of the system, proving that the use of IRS is beneficial to reducing the energy consumption of UAVs. The above work focuses on designing IRS-assisted UAV communication solutions to cope with various application scenarios. Under the premise that the idealized channel state information is completely known, a variety of optimization strategies are used to determine the passive beamforming configuration of the IRS and the flight trajectory of the UAV. It is worth emphasizing that the IRS does not have a radio frequency chain, and its function is to passively reflect the incident signal rather than actively transmit. Therefore, in UAV communication systems, especially under dynamically changing environmental conditions, achieving ideal wireless channel conditions through IRS has become a very challenging task. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a data collection design method for an intelligent reflective surface-assisted UAV, which jointly optimizes the flight trajectory of the UAV, the scheduling decision of the sensor node and the reflection phase shift vector of the intelligent reflective surface, thereby ensuring the communication data quality of each sensor node while minimizing the average information age of all sensor nodes to keep the data fresh.

[0007] The technical solution adopted by the present invention to solve the above technical problems is: a data collection design method for an intelligent reflective surface-assisted drone, characterized by comprising the following steps:

[0008] Step S1: Construct a UAV data collection system assisted by a smart reflective surface: The system includes Num sensor nodes, a UAV, and a smart reflective surface. The total working time of the system is time slots, and set the fixed length of each time slot to seconds; Num sensor nodes are randomly distributed in a And the width is The ground rectangular target area is divided into N square grids of the same size, and the side length of the square grid is l s , the center coordinate position of the nth square grid is recorded as Ω n ,Ω n =[X n ,Y n], the coordinate position of the i-th sensor node is recorded as q i ,q i =[x i ,y i ]; The maximum battery capacity of the drone is The drone is set to fly only in the airspace above the rectangular target area on the ground and along the center of the square grid. The drone is set to always fly at a height of H above the rectangular target area on the ground. U The drone is set to collect data in the sky above the , and it is set that in any time slot, the drone is associated with at most one sensor node and collects data. The drone is set to collect data when the remaining energy is less than the energy threshold e th Stop collecting data at t, and record the projection coordinate position of the drone at the tth time slot as Smart reflective surface is an M x ×M y The uniform plane array is set to deploy the intelligent reflective surface at a height of H above the rectangular target area on the ground. R The projection coordinate position of the smart reflective surface is recorded as q R ,q R =[x R ,y R ], the intelligent reflective surface is assumed to adopt a semi-passive structure and has randomly distributed Active reflective elements with RF links, and the remaining positions are distributed passive reflective elements, the intelligent reflective surface can work in channel sensing mode and reflection mode respectively; wherein, Num≥4, n=1,...,N, i=1,...,Num, H R <H U ,

[0009] Step S2: Construct a channel model of the UAV data collection system assisted by the smart reflective surface: the ground-to-air channel between each sensor node and the UAV includes a direct link from the sensor node to the UAV and a cascade link from the sensor node to the UAV after the signal is reflected by the smart reflective surface;

[0010] For direct links: In a typical scattering scenario, the channel between the UAV and the sensor node is a high-probability line-of-sight link. The channel gain of the Rice fading channel between the i-th sensor node and the UAV in the t-th time slot is recorded as Among them, wp means with probability, β0 represents the channel gain at the reference distance, α1 represents the path loss index between the sensor node and the drone, represents the distance between the i-th sensor node and the UAV at the t-th time slot, The symbol “|| ||” is the Euclidean distance symbol, e is the natural cardinality, and j is the imaginary part of the complex number. represents the phase shift generated when the signal in the tth time slot is sent from the i-th sensor node through the line-of-sight link to the drone, v represents the additional attenuation factor caused by the non-line-of-sight link loss, v < 1, k1 is the Rice factor, represents the random non-line-of-sight component from the i-th sensor node to the UAV in the t-th time slot, follows a complex Gaussian distribution with zero mean and unit variance, The probability of generating a line-of-sight environment when the i-th sensor node communicates with the drone in the t-th time slot is expressed as: is modeled as a logistic function of elevation angle and is expressed as represents the elevation angle between the i-th sensor node and the UAV at the t-th time slot, represents the horizontal distance between the i-th sensor node and the UAV at the t-th time slot, η1 and η2 are environmental parameters that depend on the building density in the rectangular target area on the ground, and exp() represents an exponential function with the natural base e as the base;

[0011] For the cascade link: The channel gain of the Rice fading channel between the i-th sensor node and the smart reflective surface at the t-th time slot is denoted as The channel gain of the air-to-air channel between the smart reflective surface and the UAV at the tth time slot is denoted as in, The dimension is M×1, M=M x ×M y , α2 represents the path loss index between the sensor node and the smart reflective surface, d i,R represents the distance between the i-th sensor node and the smart reflective surface, k2 is the Rice factor, and The dimensions of are all M×1, represents the line-of-sight component from the i-th sensor node to the smart reflective surface, , d r represents the distance between two adjacent reflective elements in the smart reflective surface, λ represents the carrier length, d r =λ / 2, the superscript "H" indicates conjugate transposition, represents the Krone product, θ i,R represents the elevation angle from the i-th sensor node to the smart reflective surface, ξ i,R represents the azimuth from the i-th sensor node to the smart reflective surface, The symbol “||” is the absolute value symbol. represents the random non-line-of-sight component from the i-th sensor node to the smart reflective surface at the t-th time slot, follows a complex Gaussian distribution with zero mean and unit variance, The dimension is M×1, represents the distance between the smart reflective surface and the drone at the tth time slot, The dimension is M×1, represents the line-of-sight component from the smart reflective surface to the UAV at the tth time slot, represents the elevation angle from the smart reflective surface to the UAV at the tth time slot, represents the azimuth from the smart reflective surface to the UAV at the tth time slot,

[0012] The received signal-to-noise ratio of the drone when the signal is sent from the i-th sensor node to the drone in the t-th time slot is recorded as like Less than the threshold η th , it means that in the tth time slot, the drone cannot successfully decode the signal transmitted by the i-th sensor node. Greater than or equal to the threshold η th , it means that in the tth time slot, the drone can successfully decode the signal transmitted by the i-th sensor node; where P represents the transmission power of the sensor node, represents the reflection phase shift vector of the smart reflection surface at the tth time slot, The mth element in is represented by represents the phase shift of the mth reflective element in the smart reflective surface at the tth time slot, σ 2 represents the power of Gaussian white noise, “⊙” represents the Hadamard product, and the superscript “T” represents the transpose of a matrix or vector;

[0013] Step S3: Each sensor node samples the environmental status information fixedly and uses it to establish a queuing model: the sampling information of size ω bit is packaged into a data packet with a sampling timestamp, and then stored in the buffer of the sensor node for queuing, waiting for the drone to collect; the sampling action of the i-th sensor node in the t-th time slot is defined as Indicates that in the tth time slot, the i-th sensor node samples the environmental status information and generates a data packet. Indicates that the i-th sensor node does not sample the environmental status information in the t-th time slot; the survival time of the latest data packet stored in the buffer of the i-th sensor node in the t-th time slot Recorded and updated to When t = 1 When t>1 represents the lifetime of the most recently stored data packet in the buffer of the ith sensor node in the t-1th time slot; the scheduling decision of the ith sensor node in the tth time slot is defined as It means that in the tth time slot, the i-th sensor node is associated with the drone and is ready to send a data packet to the drone. Indicates that the i-th sensor node is not associated with the drone in the t-th time slot; the service state of the i-th sensor node in the t-th time slot is defined as If a data packet in the buffer of the i-th sensor node in the t-th time slot is successfully transmitted to the drone with the assistance of the smart reflective surface, the i-th sensor node is said to be served in the t-th time slot, then set If a data packet in the buffer of the ith sensor node in the tth time slot is not successfully transmitted to the drone or no data packet transmission occurs in the ith sensor node in the tth time slot, the ith sensor node is said to be unserved in the tth time slot, then set

[0014] The information age of the i-th sensor node at the beginning of the t-th time slot is Recorded and updated as Then the average information age of all sensor nodes in the tth time slot is Defined as Among them, when t = 1 When t>1 represents the information age of the i-th sensor node at the t-1-th time slot, A i represents the information age set of the i-th sensor node, represents the maximum information age of the i-th sensor node;

[0015] The power consumption of the UAV mainly comes from propulsion energy, that is, the energy consumption during flight or hovering. The flight speed of the UAV in the tth time slot is set to The propulsion power of the drone at the tth time slot consists of three parts: the first part is the power required to overcome the profile resistance of the blade, that is, the blade profile power, which increases with the quadratic increase of the flight speed. Expressed as Among them, δ u represents the profile drag coefficient of the UAV, ρ a represents the air density, z represents the hardness of the drone's rotor, represents the rotor disk area of ​​the UAV, b s represents the angular velocity of the drone blades, R u Indicates the rotor radius of the UAV, U r represents the tip speed of the UAV's rotor blades; the second part is the power required to overcome the induced drag of the blades, that is, the induced power, which decreases with the increase of flight speed. Expressed as Where w is the correction factor, G is the weight of the drone, and V h represents the average induced speed of the rotor when the drone is hovering; the third part is the power required to overcome the drone's fuselage resistance, that is, the parasitic power, which increases with the quadratic increase of the flight speed and will be the parasitic power of the drone in the tth time slot Expressed as Among them, d0 represents the main drag ratio; then according to the blade profile power of the drone at the tth time slot The induced power of the UAV at the tth time slot The parasitic power of the drone at the tth time slot The propulsion power of the UAV at the tth time slot is Modeled as At the same time, the hovering power P of the UAV in the tth time slot is t ho Expressed as Among them, P t f (0) When it is equal to 0, the drone is hovering Value, P t s represents the blade profile power when the drone is hovering at the tth time slot, P t m represents the induced power when the UAV is hovering at the tth time slot;

[0016] The energy consumption of the drone in the tth time slot Expressed as in, represents the time that the drone hovers to collect data in each time slot; then the remaining energy of the drone in the tth time slot is Expressed as Among them, when t = 1 The value of is the maximum energy of the drone, when t>1 represents the remaining energy of the drone at the t-1th time slot;

[0017] Step S4: In the tth time slot, the UAV hovers and the i-th sensor node is associated with the UAV, that is, given and In the case of , before data transmission, the i-th sensor node and the UAV simultaneously send orthogonal uplink pilot signals to the smart reflective surface. When the smart reflective surface is in the channel sensing mode, the active reflective element will receive these orthogonal uplink pilot signals and estimate the sampling channel vectors from the i-th sensor node to the smart reflective surface and from the UAV to the smart reflective surface in the t-th time slot. The sampling channel vector from the i-th sensor node to the smart reflective surface in the t-th time slot It is defined as the time from the i-th sensor node to The channel vector of the active reflective element is expressed as The sampling channel vector from the drone to the smart reflective surface in the tth time slot It is defined as the time from the drone to The channel vector of the active reflective element is expressed as Then the sampling channel vector of the smart reflection surface in the tth time slot is Expressed as in, and The dimensions of G IRS is a dimension of The binary selection matrix, G IRS From the identity matrix of dimension M×M and The rows corresponding to the active reflective elements are composed;

[0018] Get the sampling channel vector from the i-th sensor node to the smart reflective surface in the t-th time slot Estimated value of And obtain the sampling channel vector from the drone to the smart reflective surface in the tth time slot Estimated value of Get the sampled channel vector of the smart reflection surface at the tth time slot Estimated value of Among them, w 1,tIndicates that when the i-th sensor node reaches the smart reflective surface in the t-th time slot The received noise vector on the active reflective element, w 2,t Indicates that when the drone reaches the smart reflective surface in the tth time slot The received noise vector on the active reflective element, w 1,t ,w 2,t All have mean 0 and variance σ 2 Gaussian distribution of I, I represents the identity matrix, and vec() is a vectorized operation;

[0019] According to the codebook in the tth time slot Determine the reflection phase shift vector of the smart reflection surface at the tth time slot

[0020] Step S5: Construct a non-convex target problem P1. The non-convex target problem P1 is to minimize the average information age of all sensor nodes by jointly optimizing the flight trajectory of the drone, the scheduling of the sensor nodes, and the reflection phase shift vector of the smart reflective surface. The non-convex target problem P1 is described as:

[0021]

[0022] Where C represents the scheduling of all sensor nodes, Q represents the flight trajectory of the drone. Φ represents the reflection phase shift matrix of the smart reflection surface, δ represents the relative importance factor. When t=1 The projection coordinate position of the initial position of the UAV is in the upper area defined by the rectangular target area on the ground. When t>1 represents the projection coordinate position of the drone in the t-1th time slot, Indicates the maximum flight speed of the drone;

[0023] Step S6: Solve the non-convex target problem P1. The specific process is as follows:

[0024] Step S6.1: Model the non-convex target problem P1 as an MDP model for intelligent reflective surface-assisted UAV data collection: The state s at the tth time slot t Includes the information age of all sensor nodes The lifetime of the latest data packet stored in the buffer of all sensor nodes Sampling channel vectors for smart reflective surfaces Estimated value of Scheduling decisions for all sensor nodes Projected coordinate position of the drone and the remaining energy of the drone in, Action a in time slot t t Including the discretized flight actions of the drone The associated actions o of all sensor nodes t ,according to Sure Action in, North means flying north, South means flying south, West means flying west, East means flying east, and Hover means hovering. In the tth time slot, the drone can only hover or fly to the adjacent square grid. t ∈{0,1,2,...,Num},o t =0 means that the UAV is not associated with any sensor node in the tth time slot, o t =1 means that the UAV is associated with the first sensor node in the tth time slot, o t =Num means that the UAV is associated with the Num-th sensor node in the t-th time slot, Then take action a t , the drone flies to the projection coordinate position of the drone in the t+1th time slot for At the same time, the information age A of all sensor nodes in the t+1th time slot is obtained t+1 , in, represents the information age of the ith sensor node at the beginning of the t+1th time slot, represents the service status of the i-th sensor node in the t+1-th time slot, Given a state s t and action a t , the reward function r at the tth time slot t (s t ,a t ) is expressed as Among them, p t is the penalty function, The goal of the MDP model is to find the optimal policy π that maximizes the long-term reward function * , long-term reward function C π Expressed as represents the mathematical expectation under strategy π, γ represents the discount factor, (γ) t-1 is γ to the power of t-1, r t (s t ,a ts0) represents the reward function at the tth time slot given the initial state s0;

[0025] Step S6.2: An enhanced proximal strategy optimization method for intelligent reflective surface assisted UAV data collection is proposed, called the PPO-RAC algorithm. The framework of the PPO-RAC algorithm is a dual-Actor-single Critic network framework. The dual actors are Actor 1 network and Actor 2 network. Actor 1 network is responsible for determining the phase shift control strategy of the intelligent reflective surface. The network parameters of Actor 1 network are The Actor 2 network is responsible for the data collection strategy of the drone. The network parameters of the Actor 2 network are The network parameter of the Critic network is φ;

[0026] Step S6.3: Let iter represent the training round, the initial value of iter is 1, let itermax represent the maximum training round; initialize the state, let the initial state be s0; initialize the network parameters of Actor1 network Initialize the network parameters of the Actor 2 network Initialize the network parameters φ of the Critic network;

[0027] Step S6.4: During the iter-th round of training, the state of the smart reflective surface at the t-th time slot is Input to Actor 1 network, Actor 1 network outputs the action of the smart reflective surface at time slot t At the same time, the state of the drone at the tth time slot Input to Actor 2 network, Actor 2 network outputs the action of the drone at the tth time slot The state at the tth time slot Input into the Critic network, the Critic network outputs s t The value function under Then according to Adjustment Take Action The drone flies to the projection coordinate position of the drone in the t+1th time slot At; then update the average information age of all sensor nodes in the tth time slot and the energy consumption of the drone in the tth time slot Then update the reward function r at the tth time slot t (s t ,a t );

[0028] During the iterth round of training, the network parameters of Actor1 network are updated every Tu consecutive time slots. Network parameters for the Actor 2 network The network parameter φ of the Critic network;

[0029] Step S6.5: Determine whether iter reaches itermax. If so, end the training process, obtain the PPO-RAC algorithm under the optimal network parameters, and determine the optimal strategy π * ; Otherwise, let iter = iter + 1, and then return to step S6.4 to continue execution; where the "=" in iter = iter + 1 is the assignment symbol.

[0030] In step S1, the projection coordinate position of the UAV refers to the coordinate position of the UAV projected onto the ground rectangular target area, the projection coordinate position of the smart reflective surface refers to the coordinate position of the smart reflective surface projected onto the ground rectangular target area, the center coordinate position of the square grid, the coordinate position of the sensor node, the projection coordinate position of the UAV, and the projection coordinate position of the smart reflective surface are all determined based on the two-dimensional coordinate system of the ground rectangular target area, and the two-dimensional coordinate system of the ground rectangular target area takes a corner of the ground rectangular target area as the origin, and the two adjacent sides of the corner are the x-axis, i.e., the horizontal axis, and the y-axis, i.e., the vertical axis.

[0031] In step S2, the reference distance refers to 1 meter away from the i-th sensor node, that is, β0 is the channel gain of the Rice fading channel between the i-th sensor node and the drone when the drone is 1 meter away from the i-th sensor node.

[0032] In step S2, k1=10, k2=30.

[0033] In step S4, the codebook in the tth time slot The design process is as follows: Construct a discrete Fourier transform codebook in, represents the discrete Fourier transform codebook for the horizontal direction, The dimension is M x ×M x , The mth x Listed as represents the discrete Fourier transform codebook for the vertical direction, The dimension is M y ×M y , The mth y Listed as Then in the codebook of the tth time slot Designed for in, represents the estimated value of the channel discrete phase shift between the UAV and its associated i-th sensor node at the t-th time slot.

[0034] In step S6.4, the network parameters of Actor1 network are updated every Tu consecutive time slots. Network parameters for the Actor 2 network The specific process of the network parameter φ of the Critic network is:

[0035] Step S6.4.1: Use the general advantage estimator to perform a stable low-variance estimate of the advantage function and obtain the estimated value of the advantage function Where γ represents the discount factor, represents the smoothing factor, δ t' When indicating

[0036] Difference error, δ t' =r t' +γV φ (s t'+1 )-V φ (s t' ), r t' represents the reward function at the t'th time slot, V φ (s t' ) represents the state s in the t'th time slot t' The value function under φ (s t'+1 ) represents the state s at the t'+1th time slot t'+1 The value function under

[0037] Step S6.4.2: By minimizing the objective value and the predicted value V φ (s t ) between the mean square error loss L C (φ) to update the network parameter φ of the Critic network, and L C (φ) is expressed as in, To find the mathematical expectation, And update the network parameters of Actor 1 network by maximizing the clipping function of Actor 1 network and Actor 2 network through the proximal strategy and the network parameters of the Actor 2 network The clipping function Expressed as in, ζ represents the hyperparameter that controls the clipping range, clip() represents the clipping operation, Indicates that the network parameters are updated in the state Next select action The probability of Indicates that the network parameters are updated in the state Next select action The probability of Indicates that the network parameters are in the state before updating Next select action The probability of Indicates that the network parameters are in the state before updating Next select action probability.

[0038] Compared with the prior art, the advantages of the present invention are:

[0039] The method of the present invention proposes an enhanced proximal strategy optimization method for intelligent reflective surface-assisted unmanned aerial vehicle data collection, which is called the PPO-RAC algorithm. The PPO-RAC algorithm integrates a redesigned Actor-Critic framework, namely a dual Actor-single Critic network framework, including two Actor networks and one Critic network. One Actor network is used to estimate the reflection phase shift vector of the intelligent reflective surface, and the other Actor network is used to optimize the flight trajectory of the unmanned aerial vehicle and its association with the sensor node. A small number of active reflective elements carried in the deployed intelligent reflective surface sample the cascade channel state to form a partial estimated channel state, and adjust the phase shift of the intelligent reflective surface according to these sampled channel states. In this way, the intelligent reflective surface improves the communication quality of the unmanned aerial vehicle, while ensuring the communication data quality of each sensor node, minimizing the average information age of all sensor nodes, so that the data remains fresh. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A scene diagram of a drone data collection system assisted by a smart reflective surface constructed in the method of the present invention;

[0041] Figure 2 A schematic diagram of the structure of the PPO-RAC algorithm based on a dual-Actor-single-Critic network framework proposed by the method of the present invention;

[0042] Figure 3 The diagram is a diagram showing the relationship between the number of active reflective elements equipped in the IRS and the average information age under the estimation of the reflection phase shift vector of four different IRSs in the simulation;

[0043] Figure 4 It is a schematic diagram of the relationship between the data packet size generated by the sensor node and the average information age under the four comparison algorithms in the simulation;

[0044] Figure 5 Schematic diagram of the relationship between the transmission power and average information age of the sensor node under different numbers of IRS reflective elements in the simulation. DETAILED DESCRIPTION

[0045] The present invention will be further described below with reference to the accompanying drawings and in combination with specific embodiments, so that those skilled in the art can implement the invention with reference to the description. The protection scope of the present invention is not limited to the specific embodiments.

[0046] The present invention proposes a data collection design method for an intelligent reflective surface-assisted unmanned aerial vehicle, which comprises the following steps:

[0047] Step S1: Constructing a UAV data collection system assisted by intelligent reflective surfaces: Figure 1 As shown in the figure, the system includes Num sensor nodes, a drone, and an intelligent reflective surface. The total working time of the system is time slots, and set the fixed length of each time slot to seconds; Num sensor nodes are randomly distributed in a And the width is The ground rectangular target area is divided into N square grids of the same size, and the side length of the square grid is l s , the center coordinate position of the nth square grid is recorded as Ω n ,Ω n =[X n ,Y n ], the coordinate position of the i-th sensor node is recorded as q i ,q i =[x i ,y i ]; The maximum battery capacity of the drone is The drone is set to fly only in the airspace above the rectangular target area on the ground and along the center of the square grid. The drone is set to always fly at a height of H above the rectangular target area on the ground. U The drone is set to collect data in the sky above the , and it is set that in any time slot, the drone is associated with at most one sensor node and collects data. The drone is set to collect data when the remaining energy is less than the energy threshold e th Stop collecting data at t, and record the projection coordinate position of the drone at the tth time slot as Smart reflective surface is an M x ×M y The uniform plane array is set to deploy the intelligent reflective surface at a height of H above the rectangular target area on the ground. R On the building, the projection coordinate position (generally the center) of the smart reflective surface is recorded as q R ,q R=[x R ,y R ], the intelligent reflective surface is assumed to adopt a semi-passive structure and has randomly distributed Active reflective elements with RF links, and the remaining positions are distributed passive reflective elements, the intelligent reflective surface can work in channel sensing mode and reflection mode respectively; where Num≥4, such as taking Num=5, n=1,...,N, X n and Y n Corresponding to Ω n The horizontal and vertical coordinates of i and i Corresponding to q i The horizontal and vertical coordinates of and Corresponding to The horizontal and vertical coordinates, M x Represents the number of rows of a uniform planar array, such as M x =10,M y Represents the number of columns of a uniform planar array, such as M y =10,

[0048] H R <H U , x R and R Corresponding to q R The horizontal and vertical coordinates of If you take N, l s , H U 、e th 、M x 、M y , H R , The specific values ​​of each are preset values.

[0049] In this specific embodiment, in step S1, the projection coordinate position of the UAV refers to the coordinate position of the UAV projected onto the ground rectangular target area, the projection coordinate position of the smart reflective surface refers to the coordinate position of the smart reflective surface projected onto the ground rectangular target area, the center coordinate position of the square grid, the coordinate position of the sensor node, the projection coordinate position of the UAV, and the projection coordinate position of the smart reflective surface are all determined based on the two-dimensional coordinate system of the ground rectangular target area. The two-dimensional coordinate system of the ground rectangular target area takes a corner of the ground rectangular target area as the origin, and the two adjacent sides of the corner are the x-axis, i.e., the horizontal axis, and the y-axis, i.e., the vertical axis.

[0050] Step S2: Construct a channel model of the UAV data collection system assisted by the intelligent reflective surface: the ground-to-air channel between each sensor node and the UAV includes a direct link from the sensor node to the UAV and a cascade link from the sensor node to the UAV after the signal is reflected by the intelligent reflective surface. That is, the direct link refers to the communication link between the sensor node and the UAV, and the cascade link refers to the signal sent by the sensor node to the intelligent reflective surface, which is then reflected by the intelligent reflective surface to the UAV, thereby achieving the communication link for the purpose of communication between the sensor node and the UAV.

[0051] For direct links: In UAV communication systems, signal propagation involves two main environments: line-of-sight (LoS) and non-line-of-sight (NLoS). In a line-of-sight environment, the signal propagates in a straight line between the transmitter (such as a UAV) and the receiver (such as a sensor node on the ground) without obstruction. In contrast, in a non-line-of-sight environment, the signal propagation path is disturbed due to the obstruction of complex terrain or objects such as buildings and trees. These two environments appear alternately with a certain probability during the UAV communication process, but usually only one of them exists at the same time. When in a line-of-sight environment, the communication link between the sensor node and the UAV is called a line-of-sight link (LoS Link); when in a non-line-of-sight environment, the corresponding communication link is called a non-line-of-sight link (NLoS Link). In a typical scattering scenario, the channel between the UAV and the sensor node is a line-of-sight link with a high probability. The channel gain of the Rice fading channel between the i-th sensor node and the UAV in the t-th time slot is recorded as Among them, wp means with probability, β0 represents the channel gain at the reference distance, which refers to the distance of 1 meter from the i-th sensor node, that is, β0 is the channel gain of the Rice fading channel between the i-th sensor node and the drone when the drone is 1 meter away from the i-th sensor node, α1 represents the path loss index between the sensor node and the drone, and the specific value of α1 is a preset value, such as α1=3.2, represents the distance between the i-th sensor node and the UAV at the t-th time slot, The symbol “||||” is the Euclidean distance symbol, e is the natural cardinality, e=2.71..., j is the imaginary part of the complex number, It represents the phase shift generated when the signal in the tth time slot is sent from the i-th sensor node through the line-of-sight link to the drone. v represents the additional attenuation factor caused by the non-line-of-sight link loss. v < 1. For example, v = 0.1. k1 is the Rice factor, k1 = 10. represents the random non-line-of-sight component from the i-th sensor node to the UAV in the t-th time slot, follows a complex Gaussian distribution with zero mean and unit variance, The probability of generating a line-of-sight environment when the i-th sensor node communicates with the drone in the t-th time slot is expressed as: is modeled as a logistic function of elevation angle and is expressed as represents the elevation angle between the i-th sensor node and the UAV at the t-th time slot, represents the horizontal distance between the i-th sensor node and the UAV in the t-th time slot, η1 and η2 are environmental parameters that depend on the building density in the rectangular target area on the ground, and the specific values ​​of η1 and η2 are preset values, such as η1 = 30.95, η2 = 0.12, and exp() represents an exponential function with the natural base e as the base.

[0052] For the cascade link: Since the positions of the sensor nodes and the smart reflective surface are fixed and the installation height of the smart reflective surface is H R Below the flight altitude of the drone, H U , so the channel between the sensor node and the smart reflective surface also adopts the Rice fading channel. Therefore, the channel gain of the Rice fading channel between the i-th sensor node and the smart reflective surface in the t-th time slot is recorded as Since the intelligent reflective surface is installed at a certain height, namely H R Therefore, the channel between the smart reflective surface and the UAV is an air-to-air channel dominated by the line-of-sight link. The channel gain of the air-to-air channel between the smart reflective surface and the UAV in the tth time slot is recorded as in, The dimension is M×1, M=M x ×M y , α2 represents the path loss index between the sensor node and the smart reflective surface. The specific value of α2 is a preset value, such as α2 = 2.1, d i,R represents the distance between the i-th sensor node and the smart reflective surface, k2 is the Rice factor, k2=30, and The dimensions of are all M×1, represents the line-of-sight component from the i-th sensor node to the smart reflective surface, , d r represents the distance between two adjacent reflective elements in the smart reflective surface, λ represents the carrier length, d r =λ / 2, the superscript "H" indicates conjugate transposition, represents the Krone product, θ i,R represents the elevation angle from the i-th sensor node to the smart reflective surface, ξ i,R represents the azimuth from the i-th sensor node to the smart reflective surface, The symbol “||” is the absolute value symbol. represents the random non-line-of-sight component from the i-th sensor node to the smart reflective surface at the t-th time slot, follows a complex Gaussian distribution with zero mean and unit variance, The dimension is M×1, represents the distance between the smart reflective surface and the drone at the tth time slot, The dimension is M×1, represents the line-of-sight component from the smart reflective surface to the UAV at the tth time slot, represents the elevation angle from the smart reflective surface to the UAV at the tth time slot, represents the azimuth from the smart reflective surface to the UAV at the tth time slot,

[0053] The received signal-to-noise ratio of the drone when the signal is sent from the i-th sensor node to the drone in the t-th time slot is recorded as like Less than the threshold η th , it means that in the tth time slot, the drone cannot successfully decode the signal transmitted by the i-th sensor node. Greater than or equal to the threshold η th , it means that in the tth time slot, the drone can successfully decode the signal transmitted by the i-th sensor node; where P represents the transmission power of the sensor node, and the transmission power of all sensor nodes is the same. represents the reflection phase shift vector of the smart reflection surface at the tth time slot, The mth element in is represented by represents the phase shift of the mth reflective element in the smart reflective surface at the tth time slot, In reality, due to hardware limitations, Usually set to discrete values, it can restrict the reflected phase shift vector to a predefined codebook Select from represents the phase shift of the first reflective element in the smart reflective surface at the tth time slot, represents the phase shift of the Mth reflective element in the smart reflective surface at the tth time slot, σ 2represents the power of Gaussian white noise, “⊙” represents the Hadamard product, η th The specific value of is a preset value, and the superscript “T” indicates the transpose of a matrix or vector.

[0054] Step S3: Each sensor node samples the environmental status information fixedly and uses it to establish a queuing model: the sampling information of size ω bit is packaged into a data packet with a sampling timestamp, and then stored in the buffer of the sensor node for queuing, waiting for the drone to collect; the sampling action of the i-th sensor node in the t-th time slot is defined as Indicates that in the tth time slot, the i-th sensor node samples the environmental status information and generates a data packet. Indicates that the i-th sensor node does not sample the environmental status information in the t-th time slot; the survival time of the latest data packet stored in the buffer of the i-th sensor node in the t-th time slot Recorded and updated to Once a new data packet is stored in the buffer of the i-th sensor node in the t-th time slot, the lifetime of this data packet is In other cases where no new data packet is stored in the buffer of the i-th sensor node in the t-th time slot, the lifetime of the latest stored data packet is When t = 1 When t>1 represents the lifetime of the most recently stored data packet in the buffer of the ith sensor node in the t-1th time slot; the scheduling decision of the ith sensor node in the tth time slot is defined as It means that in the tth time slot, the i-th sensor node is associated with the drone and is ready to send a data packet to the drone. Indicates that the i-th sensor node is not associated with the drone in the t-th time slot; the service state of the i-th sensor node in the t-th time slot is defined as If a data packet in the buffer of the i-th sensor node in the t-th time slot is successfully transmitted to the UAV with the assistance of the smart reflective surface (i.e. and successfully transmitted), the i-th sensor node is said to be served in the t-th time slot, then set If a data packet in the buffer of the ith sensor node in the tth time slot is not successfully transmitted to the UAV (i.e. and failed to transmit successfully) or no data packet transmission occurs at the i-th sensor node in the t-th time slot (i.e. In the case of t, the i-th sensor node is not served in the t-th time slot, so set

[0055] In the present invention, the information age of the sensor node is used to measure the performance. At the beginning of the tth time slot, the information age of the i-th sensor node is Recorded and updated as If a data packet in the buffer of the i-th sensor node in the t-th time slot is successfully transmitted to the UAV with the assistance of the smart reflective surface, that is, So according to Otherwise, the information age increases linearly by 1 after a time slot; then the average information age of all sensor nodes in the tth time slot is Defined as Among them, when t = 1 When t>1 represents the information age of the i-th sensor node at the t-1-th time slot, A i represents the information age set of the i-th sensor node, Indicates the maximum information age of the i-th sensor node. In this embodiment, Here the maximum information age is the same for all sensor nodes.

[0056] In the present invention, the power consumption of the drone mainly comes from the propulsion energy, that is, the energy consumption during flight or hovering. The flight speed of the drone in the tth time slot is set to The flight speed of the drone is the same in each time slot. The propulsion power of the drone in the tth time slot consists of three parts: the first part is the power required to overcome the profile resistance of the blade, that is, the blade profile power, which increases with the quadratic increase of the flight speed. Expressed as Among them, δ u represents the profile drag coefficient of the UAV, ρ a represents the air density, z represents the hardness of the drone's rotor, represents the rotor disk area of ​​the UAV, b s represents the angular velocity of the drone blades, R u Indicates the rotor radius of the UAV, U r represents the tip speed of the UAV’s rotor blade, δ u ,z, b s , R u , U r The specific values ​​of are all preset values; the second part is the power required to overcome the induced resistance of the blade, that is, the induced power, which decreases with the increase of flight speed. Expressed as Where w is the correction factor, G is the weight of the drone, and V h represents the average induced speed of the rotor when the drone is hovering, w, G, V h The specific values ​​are all preset values, such as w is 0.1, G is 20kg, V h The value is 4.03m / s; the third part is the power required to overcome the fuselage resistance of the drone, that is, the parasitic power, which increases with the quadratic increase of the flight speed. Expressed as Among them, d0 represents the main drag ratio, and the specific value of d0 is a preset value, such as d0 is 0.6; then according to the blade profile power of the drone at the tth time slot The induced power of the UAV at the tth time slot The parasitic power of the drone at the tth time slot The propulsion power of the UAV at the tth time slot is Modeled as At the same time, since the drone needs to receive data information from the sensor node when hovering, the hovering power of the drone in the tth time slot is Expressed as Among them, P t f (0) When it is equal to 0, the drone is hovering Value, P t s represents the blade profile power when the drone is hovering at the tth time slot, P t m represents the induced power when the UAV is hovering in the tth time slot.

[0057] If no sensor node is scheduled in the tth time slot, that is, all sensor nodes are not associated with the UAV, that is, The UAV does not collect data. The UAV flies throughout the time slot. If a sensor node is scheduled in the tth time slot, that is, a sensor node is associated with the UAV, that is, Then in the tth time slot, the drone is at time The system will hover to collect data regardless of whether the transmission is successful or not, and will hover for the rest of the time slot. Fly to the next designated location within t seconds, and the energy consumption of the drone in the tth time slot Expressed as in, represents the time that the drone hovers to collect data in each time slot; then the remaining energy of the drone in the tth time slot is Expressed as Among them, when t = 1 The value of is the maximum energy of the drone, when t>1 represents the remaining energy of the drone at the t-1th time slot.

[0058] Step S4: In the tth time slot, the UAV hovers and the i-th sensor node is scheduled, that is, the i-th sensor node is associated with the UAV, that is, given and In the case of , before data transmission, the i-th sensor node and the UAV simultaneously send orthogonal uplink pilot signals to the smart reflective surface. When the smart reflective surface is in the channel sensing mode, the active reflective element will receive these orthogonal uplink pilot signals and estimate the sampling channel vectors from the i-th sensor node to the smart reflective surface and from the UAV to the smart reflective surface in the t-th time slot. The sampling channel vector from the i-th sensor node to the smart reflective surface in the t-th time slot It is defined as the time from the i-th sensor node to The channel vector of the active reflective element is expressed as The sampling channel vector from the drone to the smart reflective surface in the tth time slot It is defined as the time from the drone to The channel vector of the active reflective element is expressed as Then the sampling channel vector of the smart reflection surface in the tth time slot is Expressed as in, and The dimensions of G IRS is a dimension of The binary selection matrix, G IRS From the identity matrix of dimension M×M and The active reflective elements correspond to the rows.

[0059] Get the sampling channel vector from the i-th sensor node to the smart reflective surface in the t-th time slot Estimated value of And obtain the sampling channel vector from the drone to the smart reflective surface in the tth time slot Estimated value of Get the sampled channel vector of the smart reflection surface at the tth time slot Estimated value of Among them, w 1,t Indicates that when the i-th sensor node reaches the smart reflective surface in the t-th time slot The received noise vector on the active reflective element, w 2,t Indicates that when the drone reaches the smart reflective surface in the tth time slot The received noise vector on the active reflective element, w1,t ,w 2,t All have mean 0 and variance σ 2 I represents the Gaussian distribution, I represents the identity matrix, and vec() represents a vectorized operation.

[0060] According to the codebook in the tth time slot Determine the reflection phase shift vector of the smart reflection surface at the tth time slot In this specific embodiment, the codebook in the tth time slot The design process is as follows: Considering the UPA (Uniform Planar Array) structure of the smart reflective surface, a Discrete Fourier Transform (DFT) codebook is constructed in, represents the discrete Fourier transform codebook for the horizontal direction, The dimension is M x ×M x , The mth x Listed as m x =1,...,M x , represents the discrete Fourier transform codebook for the vertical direction, The dimension is M y ×M y , The mth y Listed as Then in the codebook of the tth time slot Designed for in, represents the estimated value of the channel discrete phase shift between the UAV and its associated i-th sensor node at the t-th time slot.

[0061] Step S5: Construct a non-convex target problem P1. The non-convex target problem P1 is to minimize the average information age of all sensor nodes by jointly optimizing the flight trajectory of the drone, the scheduling of the sensor nodes, and the reflection phase shift vector of the smart reflective surface. The non-convex target problem P1 is described as:

[0062]

[0063] Among them, min is the minimum function, st means "constrained by...", C represents the scheduling of all sensor nodes, Q represents the flight trajectory of the drone. Φ represents the reflection phase shift matrix of the smart reflection surface, δ represents a relatively important factor. The specific value of δ is a preset value. For example, if δ is 0.0001, when t=1 The projection coordinate position representing the initial position of the drone, Given that the initial position of the drone is within the airspace defined by the rectangular target area on the ground, when t>1 represents the projection coordinate position of the drone in the t-1th time slot, Indicates the maximum flight speed of the drone. The specific value of is the preset value. The first constraint C1 in the non-convex objective problem P1 is expressed as from It is determined that the second constraint C2 means that the UAV can associate with at most one sensor node and collect data in each time slot, the third constraint C3 represents the range within which the UAV can fly, and the fourth constraint C4 represents the feasible set of the UAV's flight trajectory in each time slot.

[0064] Solving the non-convex target problem P1 is challenging. The third and fourth constraints of the non-convex target problem P1 are non-convex about the flight trajectory Q of the drone and the reflection phase shift matrix Φ of the smart reflective surface. It is a mixed integer non-convex optimization problem, which is very difficult to solve using traditional optimization algorithms. In order to solve this complex problem, the present invention proposes an enhanced proximal policy optimization (PPO) method for IRS-assisted drone data collection, called the PPO-RAC algorithm, to find a better solution.

[0065] Step S6: Solve the non-convex target problem P1. The specific process is as follows:

[0066] Step S6.1: Model the non-convex target problem P1 as an MDP (Markov decision process) model for intelligent reflective surface-assisted UAV data collection: The state s at the tth time slot t Includes the information age of all sensor nodes The lifetime of the latest data packet stored in the buffer of all sensor nodes Sampling channel vectors for smart reflective surfaces Estimated value of Scheduling decisions for all sensor nodes Projected coordinate position of the drone and the remaining energy of the drone in, Action a in time slot t t Including the discretized flight actions of the drone The associated actions o of all sensor nodes t ,according to Sure Action in, North means flying north, South means flying south, West means flying west, East means flying east, and Hover means hovering. In the tth time slot, the drone can only hover or fly to the adjacent square grid. t ∈{0,1,2,...,Num},o t =0 means that the UAV is not associated with any sensor node in the tth time slot, o t =1 means that the UAV is associated with the first sensor node in the tth time slot, o t =Num means that the UAV is associated with the Num-th sensor node in the t-th time slot, Then take action a t , the drone flies to the projection coordinate position of the drone in the t+1th time slot for At the same time, the information age A of all sensor nodes in the t+1th time slot is obtained t+1 , in, represents the information age of the ith sensor node at the beginning of the t+1th time slot, represents the service status of the i-th sensor node in the t+1-th time slot, Given a state s t and action a t , the reward function r at the tth time slot t (s t ,a t ) is expressed as Among them, p t is the penalty function, The goal of the MDP model is to find the optimal policy π that maximizes the long-term reward function * , long-term reward function C π Expressed as represents the mathematical expectation under strategy π, γ represents the discount factor, and in this embodiment, γ is 0.98, (γ) t-1 is γ to the power of t-1, r t (s t ,a t |s0) represents the reward function at the tth time slot given the initial state s0.

[0067] Step S6.2: An enhanced proximal policy optimization (PPO) method for smart reflective surface assisted UAV data collection is proposed, called the PPO-RAC algorithm, which jointly optimizes the phase shift of the smart reflective surface, the flight trajectory of the UAV, and the scheduling of sensor nodes based on incomplete channel estimation. It not only ensures the communication data quality of each sensor node, but also minimizes the average information age of the sensor nodes to keep the data fresh. The algorithm structure is as follows: Figure 2 As shown in Figure 1, the framework of the PPO-RAC algorithm is a dual-Actor-single Critic network framework. The dual Actors are Actor 1 network and Actor 2 network. Actor 1 network is responsible for determining the phase shift control strategy of the smart reflective surface. The network parameters of Actor 1 network are The Actor 2 network is responsible for the data collection strategy of the drone. The network parameters of the Actor 2 network are The network parameter of the Critic network is φ.

[0068] Step S6.3: Let iter represent the training round, the initial value of iter is 1, let itermax represent the maximum training round, in this embodiment, itermax=50000; initialize the state, let the initial state be s0; initialize the network parameters of Actor1 network Initialize the network parameters of the Actor 2 network Initialize the network parameters φ of the Critic network.

[0069] Step S6.4: During the iter-th round of training, the state of the smart reflective surface at the t-th time slot is Input to Actor 1 network, Actor 1 network outputs the action of the smart reflective surface at time slot t At the same time, the state of the drone at the tth time slot Input to Actor2 network, Actor 2 network outputs the action of the drone at the tth time slot The state at the tth time slot Input into the Critic network, the Critic network outputs s t The value function under Then according to Adjustment Take Action The drone flies to the projection coordinate position of the drone in the t+1th time slot At; then update the average information age of all sensor nodes in the tth time slot and the energy consumption of the drone in the tth time slot Then update the reward function r at the tth time slott (s t ,a t ).

[0070] During the iterth round of training, the network parameters of Actor1 network are updated every Tu consecutive time slots. Network parameters for the Actor 2 network The network parameter φ of the Critic network; wherein, in this embodiment, Tu=10 is taken, and the network parameter is updated every 10 consecutive time slots.

[0071] In this specific embodiment, the network parameters of Actor1 network are updated every Tu consecutive time slots. Network parameters for the Actor2 network The specific process of the network parameter φ of the Critic network is:

[0072] Step S6.4.1: Use General Advantage Estimation (GAE) to perform a stable low-variance estimate of the advantage function and obtain an estimate of the advantage function Wherein, γ represents the discount factor. In this embodiment, γ is 0.98. represents the smoothing factor, δ t' represents the temporal-difference (TD) error, δ t' =r t' +γV φ (s t'+1 )-V φ (s t' ), r t' represents the reward function at the t'th time slot, V φ (s t' ) represents the state s in the t'th time slot t' The value function under φ (s t'+1 ) represents the state s at the t'+1th time slot t'+1 The value function below.

[0073] Step S6.4.2: By minimizing the objective value and the predicted value V φ (s t ) between the mean square error loss L C (φ) to update the network parameter φ of the Critic network, and L C (φ) is expressed as in, To find the mathematical expectation, And update the network parameters of Actor 1 network by maximizing the pruning function of Actor 1 network and Actor 2 network through the existing proximal strategy and the network parameters of the Actor 2 network The clipping function Expressed as in, min is the minimum function, ζ is the hyperparameter that controls the clipping range, and clip() is the clipping operation, which takes the objective function Restricted to the range (1-ζ, 1+ζ) to ensure that the gap between the distribution of the new strategy (the strategy after the network parameters are updated) and the old strategy (the strategy before the network parameters are updated) is small, Indicates that the network parameters are updated in the state Next select action The probability of Indicates that the network parameters are updated in the state Next select action The probability of Indicates that the network parameters are in the state before updating Next select action The probability of Indicates that the network parameters are in the state before updating Next select action probability.

[0074] Step S6.5: Determine whether iter reaches itermax. If so, end the training process, obtain the PPO-RAC algorithm under the optimal network parameters, and determine the optimal strategy π * ; Otherwise, let iter = iter + 1, and then return to step S6.4 to continue execution; where the "=" in iter = iter + 1 is the assignment symbol.

[0075] In order to verify the feasibility and effectiveness of the method of the present invention, a simulation test is carried out on the method of the present invention, as follows:

[0076] Set a ground rectangular target area of ​​300m×400m and divide it into 300 square grids of 20m×20m. Take a drone to provide services for Num=5 sensor nodes. The maximum flight speed of the drone is The initial position of the drone is The altitude of the drone is fixed at H U = 60m, the maximum battery capacity it carries is The energy threshold is e th=8e3J; the coordinate positions of the sensor nodes are q1 = [60,140], q2 = [100,220], q3 = [160,180], q4 = [200,80] and q5 = [200,160]; an IRS with 10×10 reflective elements is deployed, equipped with There are active reflective elements, whose coordinate position is q R = [0,200], height is H R =30m, considering the structure of UPA, the reflection phase shift vector is set to be selected from the DFT-based codebook; the PPO-RAC algorithm adopts the Adam optimization algorithm, in which the discount factor for updating the strategy is γ = 0.98, and the learning rates of the two strategy networks, namely the Actor1 network and the Actor2 network, and the value network, namely the Critic network, are α A1 =1e-5,α A2 =1e-4 and α C = 1e-5, and the minimum update batch is 200, the hyperparameter controlling the clipping range is 0.3, and each optimization update is performed for 10 consecutive steps.

[0077] For performance comparison, the following four algorithms are evaluated: 1) The inventive method (PPO-RAC for short): The phase control strategy of IRS and the UAV data collection strategy are jointly optimized using an improved PPO framework; 2) PPO-Dual: This algorithm uses a separate PPO framework to independently find the UAV data collection scheme and the phase control strategy of IRS. The IRS-assisted UAV data collection scheme is optimized using PPO, where the phase shift of IRS is adjusted according to the trained Actor network; 3) Ideal IRS phase control: In the IRS-assisted UAV data collection scheme, the IRS phase shift is adjusted under the assumption of perfect channel state information, and PPO is used to determine the flight path of the UAV and the association of sensor nodes; 4) No IRS: This UAV-assisted data collection scheme does not require IRS and only relies on PPO for optimization.

[0078] like Figure 3 The figure shows the number of active reflective elements and the average information age of IRS under the four comparison algorithms. relationship. Figure 3It can be observed that the method of the present invention (PPO-RAC shown in the figure) is compared with the case where no IRS is deployed (no IRS shown in the figure). When 5 active reflective elements or more are configured, the average information age is smaller, and it decreases as the number of active reflective elements equipped with the IRS increases. When the number of active reflective elements is 8, the optimal value of the method of the present invention is reached, which is close to the value generated under the ideal phase of the IRS. Comparing the method of the present invention with the solution of estimating the reflection phase shift vector of the IRS through the trained PPO network (PPO-Dual shown in the figure), the average information age obtained by the method of the present invention is smaller and the overall performance is better.

[0079] like Figure 4 The figure shows the data packet size and average information age generated by sensor nodes under the four comparison algorithms. Consider the case where the size of the data packet generated by the sensor node increases from 100KB to 140KB. Figure 4 It can be seen that the larger the size of the data packet generated by the sensor node, the larger the average information age. This is because the size of the data packet is related to the minimum transmission threshold. The larger the data packet size, the larger the minimum transmission threshold required. The position where the drone can transmit is reduced, and the transmission range is reduced, resulting in an increase in the time it takes for the drone to find a suitable position, thereby increasing the average information age of the sensor node. Comparing these four schemes, under any data packet size, the average information age obtained by the method of the present invention (PPO-RAC shown in the figure) is significantly smaller than the scheme without deploying IRS (No IRS shown in the figure), and is not much different from the average information age obtained by the scheme that estimates the reflection phase shift vector of IRS through the trained PPO network (PPO-Dual shown in the figure). When the data packet size is 140KB, it is very close to the average information age obtained under the ideal IRS phase shift.

[0080] like Figure 5 As shown in the figure, the transmission power and average information age of sensor nodes under different numbers of IRS reflective elements are shown. relationship. Figure 5It can be seen that with the increase of the number of IRS reflective elements or the transmission power of the sensor node, the average information age is greatly reduced. Among them, when the transmission power is small, the average information age is significantly reduced by increasing the number of reflective elements. When the transmission power is large, the impact of the number of reflective elements on the average information age is reduced. This is because increasing the transmission power can also directly increase the signal-to-noise ratio from the sensor node to the drone, thereby increasing the channel capacity and reducing the average information age. However, the energy of the sensor nodes in the wireless sensor network is very limited. In order to extend the life of the wireless sensor network, the transmission power of the sensor nodes cannot be too large. In this case, the network performance can be improved by increasing the number of IRS reflective elements to ensure the freshness of the data.

Claims

1. A data collection design method for intelligent reflective surface-assisted drones, characterized in that The following steps are involved: Step S1: Construct a UAV data collection system assisted by a smart reflective surface: The system includes Num sensor nodes, a UAV, and a smart reflective surface. The total working time of the system is time slots, and set the fixed length of each time slot to seconds; Num sensor nodes are randomly distributed in a And the width is The ground rectangular target area is divided into N square grids of the same size, and the side length of the square grid is l s , the center coordinate position of the nth square grid is recorded as Ω n ,Ω n =[X n ,Y n ], the coordinate position of the i-th sensor node is recorded as q i ,q i =[x i ,y i ]; The maximum battery capacity of the drone is The drone is set to fly only in the airspace above the rectangular target area on the ground and along the center of the square grid. The drone is set to always fly at a height of H from the rectangular target area on the ground. U The drone is set to collect data in the air above the , and it is set that in any time slot, the drone is associated with at most one sensor node and collects data. The drone is set to collect data when the remaining energy is less than the energy threshold e th Stop collecting data at time t, and record the projection coordinate position of the drone at time slot t as Smart Reflective Surface is an M x ×M y The uniform plane array is set to deploy the smart reflective surface at a height of H above the rectangular target area on the ground. R On the building, the projection coordinate position of the smart reflective surface is recorded as q R ,q R =[x R ,y R ], the intelligent reflective surface is assumed to adopt a semi-passive structure and to have randomly distributed Active reflective elements with RF links, and the remaining positions are distributed passive reflective elements, the intelligent reflective surface can work in channel sensing mode and reflection mode respectively; wherein, Num≥4, n=1,...,N, i=1,...,Num, H R <H U , Step S2: Construct a channel model for the UAV data collection system assisted by the smart reflective surface: the ground-to-air channel between each sensor node and the UAV includes a direct link from the sensor node to the UAV and a cascade link from the sensor node to the UAV after the signal is reflected by the smart reflective surface. For direct links: In a typical scattering scenario, the channel between the UAV and the sensor node is a high-probability line-of-sight link. The channel gain of the Rice fading channel between the i-th sensor node and the UAV in the t-th time slot is recorded as Among them, wp refers to with probability, β0 represents the channel gain at the reference distance, α1 represents the path loss index between the sensor node and the drone, represents the distance between the i-th sensor node and the UAV at the t-th time slot, The symbol "|| ||" is the Euclidean distance symbol, e is the natural cardinality, and j is the imaginary part of the complex number. represents the phase shift generated when the signal in the tth time slot is sent from the i-th sensor node through the line-of-sight link to the drone, v represents the additional attenuation factor caused by the non-line-of-sight link loss, v < 1, k1 is the Rice factor, represents the random non-line-of-sight component from the i-th sensor node to the UAV in the t-th time slot, obeys a complex Gaussian distribution with zero mean and unit variance, The probability of generating a line-of-sight environment when the i-th sensor node communicates with the drone in the t-th time slot is expressed as: is modeled as a logistic function of elevation angle and expressed as represents the elevation angle between the i-th sensor node and the UAV at the t-th time slot, represents the horizontal distance between the i-th sensor node and the UAV at the t-th time slot, η1 and η2 are environmental parameters that depend on the building density within the rectangular target area on the ground, and exp() represents the exponential function with the natural base e as the base; For the cascade link: the channel gain of the Rice fading channel between the i-th sensor node and the smart reflective surface at the t-th time slot is denoted as The channel gain of the air-to-air channel between the smart reflective surface and the UAV at the tth time slot is denoted as in, The dimension is M×1, M=M x ×M y , α2 represents the path loss index between the sensor node and the smart reflective surface, d i,R represents the distance between the i-th sensor node and the smart reflective surface, k2 is the Rice factor, and The dimensions are all M×1, represents the line-of-sight component from the i-th sensor node to the smart reflective surface, , d r represents the distance between two adjacent reflective elements in the smart reflective surface, λ represents the carrier length, d r =λ / 2, the superscript "H" indicates conjugate transposition, represents the Krone product, θ i,R represents the elevation angle from the i-th sensor node to the smart reflective surface, ξ i,R represents the azimuth angle from the i-th sensor node to the smart reflective surface, The symbol "||" is the absolute value symbol. represents the random non-line-of-sight component from the i-th sensor node to the smart reflective surface at the t-th time slot, obeys a complex Gaussian distribution with zero mean and unit variance, The dimension is M×1, represents the distance between the smart reflective surface and the UAV at the tth time slot, The dimension is M×1, represents the line-of-sight component from the smart reflective surface to the UAV at the tth time slot, represents the elevation angle from the smart reflective surface to the UAV at the tth time slot, represents the azimuth from the smart reflective surface to the UAV at the tth time slot, The signal-to-noise ratio of the drone when the signal is sent from the i-th sensor node to the drone in the t-th time slot is recorded as like Less than the threshold η th , it means that in the tth time slot, the drone cannot successfully decode the signal transmitted by the i-th sensor node. If Greater than or equal to the threshold η th , it means that in the tth time slot, the drone can successfully decode the signal transmitted by the i-th sensor node; where P represents the transmission power of the sensor node, represents the reflection phase shift vector of the smart reflection surface in the tth time slot, The mth element in is represented as represents the phase shift of the mth reflective element in the smart reflective surface at the tth time slot, σ 2 represents the power of Gaussian white noise, "⊙" represents the Hadamard product, and the superscript "T" represents the transpose of a matrix or vector; Step S3: Each sensor node samples the environmental status information fixedly and uses it to establish a queuing model: the sampling information of size ω bits is packaged into a data packet with a sampling timestamp, and then stored in the buffer of the sensor node for queuing, waiting for the drone to collect it; the sampling action of the i-th sensor node in the t-th time slot is defined as Indicates that in the tth time slot, the i-th sensor node samples the environmental status information and generates a data packet. Indicates that the i-th sensor node does not sample the environmental status information in the t-th time slot; the survival time of a data packet stored in the buffer of the i-th sensor node in the t-th time slot Recorded and updated When t=1 When t>1 represents the lifetime of the most recently stored data packet in the buffer of the i-th sensor node in the t-1th time slot; the scheduling decision of the i-th sensor node in the t-th time slot is defined as Indicates that in the tth time slot, the i-th sensor node is associated with the drone and is ready to send a data packet to the drone. Indicates that the i-th sensor node is not associated with the UAV in the t-th time slot; the service state of the i-th sensor node in the t-th time slot is defined as If a data packet in the buffer of the i-th sensor node in the t-th time slot is successfully transmitted to the drone with the assistance of the smart reflective surface, the i-th sensor node is said to be served in the t-th time slot, then set If a data packet in the buffer of the i-th sensor node in the t-th time slot is not successfully transmitted to the UAV or no data packet transmission occurs in the i-th sensor node in the t-th time slot, the i-th sensor node is said to be unserved in the t-th time slot, then set The information age of the i-th sensor node at the beginning of the t-th time slot is Recorded and updated as Then the average information age of all sensor nodes in the tth time slot is Defined as Among them, when t=1 When t>1 represents the information age of the i-th sensor node at the t-1-th time slot, A i represents the information age set of the i-th sensor node, represents the maximum information age of the i-th sensor node; The power consumption of the UAV mainly comes from propulsion energy, that is, the energy consumption during flight or hovering. The flight speed of the UAV in the tth time slot is set to The propulsion power of the UAV at the tth time slot consists of three parts: the first part is the power required to overcome the profile resistance of the blade, that is, the blade profile power, which increases with the quadratic increase of the flight speed. Expressed as Among them, δ u represents the profile drag coefficient of the UAV, ρ a represents the air density, z represents the hardness of the drone's rotor, represents the rotor disk area of ​​the UAV, b s represents the angular velocity of the drone's blades, R u Indicates the rotor radius of the UAV, U r represents the tip speed of the UAV's rotor blades; the second part is the power required to overcome the induced drag of the blades, that is, the induced power, which decreases with the increase of flight speed. Expressed as Where w is the correction factor, G is the weight of the drone, and V h represents the average induced speed of the rotor when the drone is hovering; the third part is the power required to overcome the drone's fuselage resistance, that is, the parasitic power, which increases with the square of the flight speed and will be the parasitic power of the drone in the tth time slot. Expressed as Among them, d0 represents the main drag ratio; then according to the blade profile power of the UAV at the tth time slot The induced power of the UAV in the tth time slot The parasitic power of the drone at time slot t The propulsion power of the UAV at time slot t is Modeled as At the same time, the hovering power of the UAV in the tth time slot is Expressed as in, For the time When it is equal to 0, the drone is hovering value, represents the blade profile power when the UAV is hovering at the tth time slot, represents the induced power when the UAV is hovering at the tth time slot; The energy consumption of the drone in the tth time slot Expressed as in, Indicates the time the drone hovers to collect data in each time slot; then the remaining energy of the drone in the tth time slot is Expressed as Among them, when t=1 The value of is the maximum energy of the UAV, when t>1 represents the remaining energy of the UAV at the t-1th time slot; Step S4: In the tth time slot, the UAV is hovering and the i-th sensor node is associated with the UAV, i.e., given and In the case of , before data transmission, the i-th sensor node and the UAV simultaneously send orthogonal uplink pilot signals to the smart reflective surface. In the channel sensing mode of the smart reflective surface, the active reflective element will receive these orthogonal uplink pilot signals and estimate the sampling channel vectors from the i-th sensor node to the smart reflective surface and from the UAV to the smart reflective surface in the t-th time slot. The sampling channel vector from the i-th sensor node to the smart reflective surface in the t-th time slot is It is defined as the time from the i-th sensor node to the The channel vector of the active reflective element is expressed as The sampling channel vector from the drone to the smart reflective surface in the tth time slot is It is defined as the time from the drone to The channel vector of the active reflective element is expressed as Then the sampling channel vector of the smart reflection surface in the tth time slot is Expressed as in, and The dimensions of G IRS is a dimension of The binary selection matrix, G IRS From the identity matrix of dimension M×M The rows corresponding to the active reflective elements are composed; Get the sampling channel vector from the i-th sensor node to the smart reflective surface in the t-th time slot Estimated value of And obtain the sampling channel vector from the drone to the smart reflective surface in the tth time slot Estimated value of Get the sampling channel vector of the smart reflection surface in the tth time slot Estimated value of Among them, w 1,t Indicates that when the i-th sensor node reaches the smart reflective surface in the t-th time slot The received noise vector on the active reflective element, w 2,t Indicates that when the drone reaches the smart reflective surface in the tth time slot The received noise vector on the active reflective element, w 1,t ,w 2,t All have mean 0 and variance σ 2 Gaussian distribution of I, I represents the identity matrix, and vec() is a vectorized operation; According to the codebook in the tth time slot Determine the reflection phase shift vector of the smart reflective surface at time slot t Step S5: Construct a non-convex objective problem P1. The non-convex objective problem P1 is to minimize the average information age of all sensor nodes by jointly optimizing the flight trajectory of the UAV, the scheduling of the sensor nodes, and the reflection phase shift vector of the smart reflective surface. The non-convex objective problem P1 is described as: Among them, C represents the scheduling of all sensor nodes, Q represents the flight trajectory of the drone, Φ represents the reflection phase shift matrix of the smart reflective surface, δ represents the relative importance factor. When t=1 The projection coordinate position of the initial position of the UAV is in the upper area defined by the rectangular target area on the ground. When t>1 represents the projected coordinate position of the drone at the t-1th time slot, Indicates the maximum flight speed of the drone; Step S6: Solve the non-convex target problem P1. The specific process is as follows: Step S6.1: Model the non-convex target problem P1 as an MDP model for intelligent reflective surface-assisted UAV data collection: the state s at the tth time slot t Includes the information age of all sensor nodes The lifetime of the latest data packet stored in the buffer of all sensor nodes Sampling channel vector of smart reflective surface Estimated value of Scheduling decisions for all sensor nodes Projected coordinate position of the drone and the remaining energy of the drone in, Action a in time slot t t Including the discretized flight actions of the drone The associated actions o of all sensor nodes t ,according to Sure Action in, North means flying to the north, South means flying to the south, West means flying to the west, East means flying to the east, and Hover means hovering. In the tth time slot, the drone can only hover or fly to the adjacent square grid. t ∈{0,1,2,...,Num},o t =0 means that the UAV is not associated with any sensor node in the tth time slot, o t =1 means that the UAV is associated with the first sensor node in the tth time slot, o t =Num means that the UAV is associated with the Num-th sensor node in the t-th time slot, Then take action a t , the drone flies to the projection coordinate position of the drone in the t+1th time slot for At the same time, the information age A of all sensor nodes in the t+1 time slot is obtained t+1 , in, represents the information age of the i-th sensor node at the beginning of the t+1-th time slot, represents the service status of the i-th sensor node in the t+1-th time slot, Given state s t and action a t , the reward function r at the tth time slot t (s t ,a t ) is expressed as Among them, pt is the penalty function, The goal of the MDP model is to find the optimal policy π that maximizes the long-term reward function * , long-term reward function C π Expressed as represents the mathematical expectation under strategy π, γ represents the discount factor, (γ) t-1 is the t-1 power of γ, r t (s t ,a t |s0) represents the reward function at the tth time slot given the initial state s0; Step S6.2: An enhanced proximal strategy optimization method for intelligent reflective surface assisted UAV data collection is proposed, called the PPO-RAC algorithm. The framework of the PPO-RAC algorithm is a dual-actor-single-critic network framework. The dual actors are Actor 1 network and Actor 2 network. Actor 1 network is responsible for determining the phase shift control strategy of the intelligent reflective surface. The network parameters of Actor 1 network are The Actor 2 network is responsible for the data collection strategy of the UAV. The network parameters of the Actor 2 network are The network parameter of the Critic network is φ; Step S6.3: Let iter represent the training round, the initial value of iter is 1, and let itermax represent the maximum training round; initialize the state, let the initial state be s0; initialize the network parameters of Actor1 network Initialize the network parameters of the Actor 2 network Initialize the network parameters φ of the Critic network; Step S6.4: During the iter-th round of training, the state of the smart reflective surface at the t-th time slot is Input to Actor 1 network, Actor 1 network outputs the action of the smart reflective surface at time slot t At the same time, the state of the drone at time slot t Input to Actor 2 network, Actor 2 network outputs the action of the drone at time slot t The state at time slot t will be Input into the Critic network, the Critic network outputs s t The value function under Then according to Adjustment Take action The drone flies to the projection coordinate position of the drone in the t+1th time slot At; then update the average information age of all sensor nodes in the tth time slot and the energy consumption of the drone in the tth time slot Then update the reward function r at the tth time slot t (s t ,a t ); During the iter-th round of training, the network parameters of Actor1 network are updated every Tu consecutive time slots. Network parameters for the Actor 2 network The network parameter φ of the critic network; Step S6.5: Determine whether iter reaches itermax. If so, end the training process, obtain the PPO-RAC algorithm under the optimal network parameters, and determine the optimal strategy π * ; Otherwise, let iter = iter + 1, and then return to step S6.4 to continue execution; where the "=" in iter = iter + 1 is an assignment symbol.

2. The data collection design method of a smart reflective surface-assisted drone according to claim 1 is characterized in that In step S1, the projection coordinate position of the drone refers to the coordinate position of the drone projected onto the ground rectangular target area, and the projection coordinate position of the smart reflective surface refers to the coordinate position of the smart reflective surface projected onto the ground rectangular target area. The center coordinate position of the square grid, the coordinate position of the sensor node, the projection coordinate position of the drone, and the projection coordinate position of the smart reflective surface are all determined based on the two-dimensional coordinate system of the ground rectangular target area. The two-dimensional coordinate system of the ground rectangular target area takes a corner of the ground rectangular target area as the origin, and the two adjacent sides of the corner are the x-axis, i.e., the horizontal axis, and the y-axis, i.e., the vertical axis.

3. The data collection design method of a smart reflective surface-assisted drone according to claim 1 is characterized in that In step S2, the reference distance refers to 1 meter away from the i-th sensor node, that is, β0 is the channel gain of the Rice fading channel between the i-th sensor node and the drone when the drone is 1 meter away from the i-th sensor node.

4. The data collection design method for an intelligent reflective surface-assisted drone according to claim 1 is characterized in that In step S2, k1=10, k2=30.

5. The data collection design method for an intelligent reflective surface-assisted drone according to claim 1 is characterized in that In step S4, the codebook of the tth time slot The design process is as follows: Constructing a discrete Fourier transform codebook in, represents the discrete Fourier transform codebook for the horizontal direction, The dimension is M x ×M x , mth x Listed as represents the discrete Fourier transform codebook for the vertical direction, The dimension is M y ×M y , mth y Listed as Then in the codebook of time slot t Designed for in, represents the estimated value of the channel discrete phase shift between the UAV and its associated i-th sensor node at the t-th time slot.

6. The data collection design method for an intelligent reflective surface-assisted drone according to claim 1 is characterized in that In step S6.4, the network parameters of Actor1 network are updated every Tu consecutive time slots. Network parameters for the Actor 2 network The specific process of the network parameter φ of the Critic network is: Step S6.4.1: Use the general advantage estimator to perform a stable low-variance estimate of the advantage function and obtain the estimated value of the advantage function Where γ represents the discount factor, represents the smoothing factor, δ t' represents the time difference error, δ t' =r t' +γV φ (s t'+1 )-V φ (s t' ), rt' represents the reward function at the t'th time slot, V φ (s t' ) represents the value function in the state st' of the t'th time slot, V φ (s t'+1 ) represents the state st' in the t'+1th time slot +1 The value function under Step S6.4.2: By minimizing the objective value and the predicted value V φ (s t ) between the mean square error loss L C (φ) to update the network parameter φ of the Critic network, and L C (φ) is expressed as in, To find the mathematical expectation, And update the network parameters of Actor 1 network by maximizing the clipping function of Actor 1 network and Actor 2 network through the proximal strategy and the network parameters of the Actor 2 network The clipping function Expressed as in, ζ represents the hyperparameter that controls the clipping range, clip() represents the clipping operation, Indicates that the network parameters are updated in the state Select Action The probability of Indicates that the network parameters are updated in the state Select Action The probability of Indicates that the network parameters are in the state before updating Select Action The probability of Indicates that the network parameters are in the state before updating Select Action probability.