RIS-assisted unmanned aerial vehicle intelligent communication method based on deep reinforcement learning

Abstract

The invention belongs to the technical field of wireless communication, and discloses an RIS-assisted unmanned aerial vehicle intelligent communication method based on deep reinforcement learning, and the method comprises the steps: 1, collecting input data, and carrying out the initialization; 2, defining a state space; 3, constructing a deep reinforcement learning algorithm, converting the state space into an input tensor, inputting the input tensor into an Actor network, and then generating a continuous action and a discrete action; step 4, generating a mixed action and calculating an instant reward to obtain an experience tuple; step 5, storing the experience tuple into a priority experience playback pool to obtain a high TD error sample; step 6, updating the Critic network; step 7, a delay updating mechanism; and step 8, outputting an optimal track sequence, an optimal RIS phase matrix, an optimal beam forming matrix and a performance index. According to the invention, through deep reinforcement learning type joint optimization of unmanned aerial vehicle trajectory optimization, RIS phase regulation and beam forming, cooperative improvement of physical layer safety and energy efficiency is realized.

Description

Technical Field

[0001] This application belongs to the field of wireless communication technology, specifically relating to a RIS-assisted intelligent communication method for unmanned aerial vehicles based on deep reinforcement learning. Background Technology

[0002] Existing safety and energy-saving optimization schemes combining Unmanned Aerial Vehicle (UAV) assisted communication systems with Reconfigurable Intelligent Surfaces (RIS) are primarily applied in dynamic urban environments and areas with limited infrastructure (such as emergency rescue sites, remote rural areas, and disaster-damaged areas) to ensure wireless communication coverage and data transmission. The main approach is to jointly optimize the UAV's three-dimensional position and the RIS phase matrix to improve communication security and energy efficiency. Currently, typical technical solutions employ methods based on convex optimization (AO) or alternating optimization (AO) to jointly adjust the UAV's trajectory, transmit power, and RIS phase.

[0003] The secure communication scheme based on convex relaxation and alternation optimization proposed by Li et al. (Li S, Duo B, Di Renzo M, et al. Robust secure UAV communications with the aid of reconfigurable intelligent surfaces[J].IEEE Transactions on Wireless Communications, 2021, 20(10): 6402-6417.) transforms the RIS phase optimization problem into a semidefinite programming (SDP) form and achieves the convergence of the objective function by iteratively optimizing the UAV position and the RIS phase matrix. However, this type of scheme usually assumes that the Channel State Information (CSI) is perfectly known, ignoring the time-varying nature of the channel caused by UAV movement and environmental changes. Therefore, the optimization results are not robust in dynamic scenarios. At the same time, the algorithm requires multiple iterations, and each optimization involves complex matrix inversion and gradient calculation, which is computationally intensive and difficult to meet the requirements of online real-time optimization for UAVs.

[0004] Xu et al. proposed a RIS-aided secure mobile edge computing framework based on the Deep Deterministic Policy Gradient (DDPG) algorithm (Xu J, Xu A, Chen L, et al. Deep pre-inforcement learning for RIS-aided secure mobile edge computing in industrial Internet of Things[J].IEEE Transactions on Industrial Informatics,2023,20(2):2455-2464.), which achieves adaptive control of UAV trajectory and RIS phase through Deep Reinforcement Learning (DRL). This method achieves secure offloading optimization in an eavesdropping scenario, verifying the effectiveness of DRL under complex channel conditions. However, the DDPG algorithm is originally designed for continuous action space, while the RIS phase offset is actually limited by discrete quantization control. This continuous modeling assumption will cause policy approximation error, making it impossible to accurately reflect the discrete hardware constraints of RIS, thus leading to a mismatch between the optimization results and the physical device.

[0005] The DRL hybrid beamforming method for multi-hop RIS-empowered terahertz communications proposed by Huang et al. (Huang C, Yang Z, Alexandropoulos GC, et al. Multi-hop RIS-empowered terahertz communications: ADRL-based hybrid beamforming design[J]. IEEE Journal on Selected Areas in Communications, 2021, 39(6): 1663-1677) reduces propagation loss and improves system coverage and spectral efficiency by simultaneously optimizing digital and analog beamformers. This scheme achieves good performance in static scenarios, but its learning strategy is only applicable to static optimization under fixed channel conditions and lacks real-time response capabilities to UAV motion and channel dynamic changes. When there are mobile eavesdroppers or obstructions causing line-of-sight (LoS) and non-line-of-sight (NLoS) state switching, the algorithm's stability and security performance significantly decrease.

[0006] Furthermore, the multi-UAV-enabled load-balance mobile-edge computing system proposed by Yang et al. (Yang L, Yao H, Wang J, et al. Multi-UAV-enabled load-balance mobile-edge computing for IoT networks[J].IEEE Internet of Things Journal,2020,7(8):6898-6908.) achieves load balancing and task allocation optimization by combining differential evolution algorithm with DRL task scheduling. Although the applicability of multi-agent reinforcement learning in complex networks has been verified, this type of method is mainly aimed at task offloading and energy balance scenarios, without designing for physical layer security. Its optimization objective fails to simultaneously consider both confidentiality and energy efficiency. The UAV-RIS coordinated energy efficiency optimization algorithm proposed by Zhang et al. (Zhang C, Lin B, Li C, et al. Energy Efficiency Maximization for Multi-UAV-IRS-Assisted Marine Vehicle Systems[J]. Journal of Marine Science and Engineering, 2024, 12(10): 1761.) combines convex optimization with deep reinforcement learning to maximize the energy efficiency of UAV-RIS coordination. This method has achieved good results in marine vehicle communication systems, but it also assumes that the RIS phase is continuously adjustable and does not consider practical quantization limitations, leading to significant performance deviations in hardware deployment.

[0007] In summary, while existing UAV-intelligent reflector cooperative optimization technologies have some theoretical reference value, they suffer from significant shortcomings in areas such as algorithm hybridity, target coordination, real-time performance, and hardware feasibility. These limitations fail to meet the comprehensive requirements of high security, high energy efficiency, and low latency in complex dynamic environments. Therefore, there is an urgent need to propose a unified optimization framework capable of simultaneously handling continuous and discrete actions, possessing dynamic adaptive learning capabilities and engineering deployability, to achieve a UAV intelligent communication system with high security, low energy consumption, and high robustness in real-world scenarios. Summary of the Invention

[0008] To address the aforementioned technical challenges, this application proposes a RIS-assisted intelligent communication method for unmanned aerial vehicles (UAVs) based on deep reinforcement learning. This method achieves a synergistic improvement in physical layer security and energy efficiency through joint optimization of UAV trajectory, RIS phase modulation, and beamforming using deep reinforcement learning. It successfully overcomes the optimization bottleneck of existing technologies in a continuous-discrete hybrid space, significantly improving the security, energy efficiency, and robustness of the UAV-RIS collaborative communication system. This provides a key technology path that can be widely applied to the construction of next-generation intelligent secure communication networks.

[0009] To achieve the above objectives, this application employs the following technical solution:

[0010] This application discloses a RIS-assisted intelligent communication method for unmanned aerial vehicles (UAVs) based on deep reinforcement learning. The RIS-assisted UAV intelligent communication method is implemented through a communication system, which includes a UAV serving as a multi-antenna transmission platform, a legitimate user equipment (UE), a potential eavesdropper (EAV), a ground-based intelligent reflector (RIS), and a central control module. The ground-based intelligent reflector (RIS) is composed of M... r ×M c Composed of N reflective units, the UAV is equipped with N a A transmission antenna transmitter and attitude control unit serve as an airborne base station to provide signal transmission services to ground users. The UAV connects to the legitimate user equipment (UE) via a direct link or an indirect link reflected by a ground-based intelligent reflector (RIS) and sends data signals to the UE. A potential eavesdropper (EAV) attempts to intercept the transmitted data signals. Specifically, during system operation, the RIS-assisted UAV intelligent communication method includes the following steps.

[0011] Step 1: Collect and initialize input data: Obtain the initial positions of the UAV and the ground intelligent reflector RIS, the energy consumption of the UAV, establish a time discretization model, and initialize the channel parameters;

[0012] Step 2: Based on the input data collected in Step 1 and the initialized channel parameters, define the state space S;

[0013] Step 3: Construct the MADPO deep reinforcement learning algorithm based on hybrid action delay policy optimization. The MADPO deep reinforcement learning algorithm includes a dual Actor network and a dual Critic network. The state space constructed in Step 2 is transformed into an input tensor X that the network can process. s (n) will be the input tensor X s (n) After inputting into the Actor network, a series of actions are generated. c (n) and discrete action a d (n);

[0014] Step 4: Based on the neural network, the continuous action a c (n) and discrete action a d (n) Generate hybrid actions and calculate immediate rewards to obtain an experience tuple (s(n), a) c (n),a d (n),r(n),s(n+1));

[0015] Step 5: Store the empirical tuples obtained in Step 4 into the priority empirical replay pool to obtain high TD error samples.

[0016] Step 6: Update the Critic network using samples with high TD error;

[0017] Step 7, Delayed Update Mechanism: Perform an Actor network update every two rounds of Critic network updates to obtain the optimized policy parameters. θ i Target network parameters θ i ';

[0018] Step 8: When episode reaches the maximum iteration training period T max Then, output the optimal trajectory sequence. Optimal RIS phase matrix Φ * (n), Optimal beamforming matrix W * And performance metrics, including average security level, energy efficiency, and convergence curve.

[0019] A further improvement to this application is that step 1 specifically includes the following steps:

[0020] Step 1.1: Obtain the initial position Q of the unmanned aerial vehicle (UAV) u (0) Initial position Q of the reconfigurable ground smart reflector RIS r All users Q k ,k∈K and the eavesdropper Q e The three-dimensional coordinates of (0) are given, and the three-dimensional position of the UAV at time n is denoted as Q. u ={x u [n],y u [n],h u [n]}:

[0021]

[0022] Among them, v h For horizontal velocity, v v The vertical velocity is represented by θ, the yaw angle is ω, and the pitch angle is x. u [n] represents the x-coordinate of the UAV at time n, vh [n] represents the horizontal velocity of the UAV at time n, y u [n] represents the ordinate of the UAV at time n, h u [n] represents the altitude coordinates of the UAV at time n, x u [n+1] is the x-coordinate of the UAV at time n+1, y u [n+1] is the ordinate of the UAV at time n+1, h u [n+1] represents the altitude coordinates of the drone at time n+1, v v [n] is the vertical velocity of the UAV at time n;

[0023] Step 1.2: The energy consumption of the UAV includes hovering power, rotor induced power, and maneuvering power. The total energy consumption e of the UAV is... u :

[0024]

[0025] Among them, P a Power P configured for a fixed rotor b For the induced energy requirement during hovering, P d For the power required for vertical maneuvering, U tip v is the rotor tip velocity. c denoted as the average downwash velocity during hovering, c0 as the airframe drag coefficient, s as the rotor compaction ratio, ρ as the atmospheric density, and G as the rotor disk surface area.

[0026] Step 1.3: Establish a time discretization model: Let the total communication duration be... The number of time slots is The length of a single slot is

[0027] Step 1.4: Initialize channel parameters:

[0028] H ur (n),H rk (n),H re (n)~Rayleigh(R)

[0029]

[0030] in, For the UAV to ground-based intelligent reflector RIS channel at time n, H re [n] represents the channel from the ground-based intelligent reflector RIS to the potential eavesdropper EAV at time n, H rk [n] represents the channel from the ground intelligent reflector RIS to the legitimate user equipment (UE) at time n, and d ur[n] represents the distance between the UAV and the ground-based intelligent reflector RIS, ζ represents the reference path loss when the distance d0 = 1m, α1 is the air-to-ground (A2G) path loss exponent, and d ur γ represents the Euclidean distance between the drone and the ground-based intelligent reflector RIS. r Rayleigh factor, The NLoS scattering component from the UAV to the ground-based intelligent reflector RIS is distributed as follows: d re The Euclidean distance between the ground-based intelligent reflector RIS and the potential eavesdropper EAV. The Loss component between the ground-based intelligent reflector RIS and the potential eavesdropper EAV. For the NLoS scattering component between the ground-based intelligent reflector RIS and the potential eavesdropper EAV, d rk The Euclidean distance between the ground-based intelligent reflector RIS and the legitimate user equipment (UE) is given. The Loss component between the ground-based intelligent reflector RIS and the legitimate user equipment (UE). The NLoS scattering components between the ground-based intelligent reflector RIS and the legitimate user equipment (UE), and the LoS components between the UAV and the ground-based intelligent reflector RIS are:

[0031]

[0032] Where λ represents wavelength, d i It is the distance between the reflective units in the ground-based intelligent reflective surface RIS.

[0033] and Let these represent the cosine and sine components of the angle reached on the horizontal plane, respectively. Corresponding to the sinusoidal component in the vertical plane, j is the imaginary unit, y r M represents the ordinate of the ground-based intelligent reflective surface RIS. c M is the number of rows in the RIS. r h is the number of columns in the RIS. r For the height coordinates of the ground-based intelligent reflector RIS, x r The x-axis represents the ground-based intelligent reflective surface RIS.

[0034] A further improvement in this application is that, in step 2, the state space is defined as follows:

[0035] S={Q u (n),Φ(n),H ur (n),H rk (n),D k (n)}

[0036] Among them, D kLet be the amount of remaining task data for user k, and Φ(n) be the phase matrix of the ground-based intelligent reflector RIS. Each element This matrix represents the phase shift applied by the corresponding reflecting unit and is located in space. middle.

[0037] A further improvement in this application is that step 3 specifically includes the following steps:

[0038] Step 3.1, Normalization of the state space S and initialized channel parameters: Normalize the UAV position:

[0039]

[0040] Among them, Q max Q represents the maximum flight range of the drone. min Q represents the minimum flight range of the drone. u (n) represents the actual position;

[0041] Calculate the normalized channel gain for direct or indirect links:

[0042] |H ur | 2 ,|H rk | 2 ,|H re | 2

[0043] Map the normalized channel gain to the [0,1] interval;

[0044] Step 3.2: Construct the system state s(n) at time n as the input tensor X. s (n) is used as the state vector, which is input into the Actor network to generate the action space, which includes continuous actions a. c (n) and discrete action a d (n), continuous action a c (n) includes the trajectory, pitch angle θ, yaw angle ω, and horizontal acceleration a of the UAV. h Vertical acceleration a v Beamforming vector w; Discrete action a d (n) includes the phase matrix Φ(n) controlling the ground intelligent reflector RIS, and the set of phase quantization values ​​for the ground intelligent reflector RIS is as follows: L represents the number of discrete phase states.

[0045] A further improvement of this application is that step 4 specifically includes the following steps:

[0046] Step 4.1, Continuous motion sampling: Continuous motion a c(n) Use the tanh activation function to output control signals:

[0047]

[0048] Among them, noise item n b It is a threshold used to control the range of noise term fluctuation. To output a strategy for continuous actions in the Actor network, a min and a max It is a threshold used to control the range of output actions. The output range of the tanh activation function is [-1, 1], which is linearly mapped to the speed and angle constraint range of the UAV.

[0049] Step 4.2, Discrete Action Sampling: Discrete Action a d (n) Output the phase probability distribution of the ground-based intelligent reflector RIS using softmax:

[0050]

[0051] According to probability distribution sampling:

[0052] a d (n)~π φ,d (a|s)

[0053] The sampling results are then mapped to a phase matrix Φ[n].

[0054] Among them, z a For discrete action logit values, z b Let L be the logit value of the discrete-phase action candidate output of the Actor network RIS, where L is the number of discrete-phase states, b is the discrete action index, and π is the logit value. φ,d Output strategy for discrete actions;

[0055] Step 4.3, Environment Execution and State Transition: Execute hybrid actions (a c (n),a d (n)), update the system state to s(n+1), and calculate the immediate reward r(n), obtaining the experience tuple (s(n), a c (n),a d (n), r(n), s(n+1)), where the instant reward is:

[0056] r(n) = c k [n]R s (k,n)-λ e E tu (k,n)-μV(k,n)

[0057] Among them, ck [n] is a binary variable indicating whether the user has completed the service they requested. k [n] = 0 indicates that the service has been satisfied; otherwise, c k [n] = 1, V(k,n) represents the penalties for exceeding limits and rate violations, λ e μ represents the energy consumption and constraint penalty weights, and E tu [n] represents the total energy consumption of the UAV's signal transmission and motion, E tu [n] = e t [n]+e u [n], where e t [n] represents the total power consumption of the UAV used for signal transmission: It is a pre-encoding vector for user k, N a For the drone's transmitting antenna, e u [n] represents the total power consumed by the drone, R s [n] represents the confidentiality rate, R s [n] = max(R) k [n]-R e [n],0),R k [n] and R e [n] represents the achievable rates of legitimate User Equipment (UE) and potential eavesdropper EAV, respectively.

[0058] R k [n] = log2(1+λ) k [n])

[0059] R e [n] = log2(1+λ) e [n])

[0060] Where, λ k [n] represents the signal-to-noise ratio of a legitimate user equipment (UE), and λ e [n] represents the signal-to-noise ratio of the potential eavesdropper's EAV:

[0061]

[0062] Among them, w k It is a pre-encoded vector for the user, w e It is a pre-coded vector for eavesdroppers, σ 2 It is additive white Gaussian noise, G k [n] = H uk [n]+(H rk [n]) H Φ[n]H ur [n] represents the total effective channel gain from the UAV to the legitimate user equipment (UE), including direct and reflected channels, Ge [n] = H ue [n]+(H re [n]) H Φ[n]H ur [n] represents the total effective channel gain from the UAV to the potential eavesdropper EAV, including both direct and reflected channels, where H uk [n] represents the direct channel from the UAV to the legitimate user equipment (UE). H ue [n] represents the direct transmission channel from the drone to the potential eavesdropper EAV. U uk [n] and U ue [n] is a circularly symmetric complex Gaussian variable with zero mean and unit variance, and its distribution is [n]. α2 is the ground-to-ground path loss exponent, d uk and d ue These are the Euclidean distances from the drone to the legitimate user equipment (UE) and to the potential eavesdropper (EAV), respectively.

[0063] A further improvement of this application is that step 5 specifically includes the following steps:

[0064] Step 5.1: Convert the empirical tuple (s(n), a) c (n),a d (n), r(n), s(n+1)) are stored in the priority experience replay pool B. e Its priority is defined according to TD-error:

[0065] p i =|δ i |+ε

[0066] δ i =Q θ (s,a)-y

[0067] Where, p i For the priority of empirical tuples, |δ i | represents the absolute value of the timing difference error, ε is a small constant, and δ i Let y be the time-series difference error, and Q be the target Q value. θ To predict the Q value;

[0068] Step 5.2: Store the data from Step 5.1 into the finite experience replay pool B. e The empirical tuples are sampled, with the sampling probability being:

[0069]

[0070] Among them, samples with high TD error were selected more frequently, resulting in sample batches. N mThis refers to the batch size.

[0071] A further improvement in this application is that step 6 specifically includes the following steps:

[0072] Step 6.1: Calculate the target Q value using high TD error samples:

[0073]

[0074] in, To initialize the target network, s′ represents the system state at the next moment after s. for The strategy for consecutive actions after an update round for The discrete action policy after one round of updates, θ i θ represents the Critic network parameter. i The parameters obtained after one round of updates, where γ is the discount factor;

[0075] Step 6.2, Optimize using mean squared error loss:

[0076]

[0077] in, The Q-value, s, is the output of the Critic network. j Let a be the system state at time j. j Let y be the action output by the Actor network at time j. j Let θ be the target Q-value output by the Critic network at time j. i Let α be the parameter of one of the two Critic networks. c The learning rate of the Critic network is used to control the magnitude of its updates. The gradient of the Critic network is used to reduce overestimation through a dual-Critic structure.

[0078] A further improvement in this application is that step 7 specifically includes the following steps:

[0079] Step 7.1: Perform an Actor network update every two rounds of Critic network updates.

[0080]

[0081] Perform a soft update synchronously:

[0082] θ i ′←τθ i +(1-τ)θ i ′,i∈{1,2}

[0083]

[0084] Where, α a τ is the learning rate of the Actor network, and τ∈(0,1) is used as a weight coefficient to control the update speed of the target network.

[0085] The beneficial effects of this application are:

[0086] This application constructs a Hybrid Action Deep Reinforcement Learning (MADPO) algorithm framework, which realizes the joint optimization of continuous motion control of UAVs and discrete phase control of intelligent reflectors. It successfully solves the long-standing problems in existing technologies, such as the difficulty of multi-objective coupled optimization, unstable training, and mismatch of hardware constraints. It has verifiable performance advantages and potential for universal application.

[0087] First, this application achieves a breakthrough in improving communication security. It directly implements dynamic anti-eavesdropping optimization at the physical layer, significantly enhancing the signal gain of legitimate user links through coordinated adjustment of RIS phase programmable reflection and the UAV's three-dimensional trajectory, while simultaneously weakening the interference of the reflection path of eavesdropping nodes.

[0088] Secondly, regarding energy efficiency and endurance optimization, the algorithm proposed in this application achieves joint scheduling of UAV motion power consumption and signal transmission power through a reinforcement learning framework, effectively reducing overall energy consumption while maintaining a high safety rate. Based on the joint modeling of aerodynamics and energy cost functions, this application enables the algorithm to dynamically and adaptively adjust the energy allocation strategy according to flight attitude and environmental channels, thereby achieving a system energy efficiency improvement of approximately 29.2% and a corresponding extension of UAV endurance of over 30%, providing a technical foundation for resource-constrained emergency communication and long-term missions.

[0089] Furthermore, regarding algorithm stability and convergence performance, this application effectively overcomes the problems of discontinuous policy updates and large gradient fluctuations in the mixed action space through the design of "Delayed Policy Update" and Priority Experience Replay (PER) mechanisms. Delayed updates suppress overestimation of the Critic network, and the double-Q minimum policy improves the accuracy and stability of policy convergence; the Priority Experience Replay mechanism ensures high-weight replay of key state-action pairs during the learning process, thereby significantly improving the utilization rate of training samples.

[0090] Furthermore, this application is innovative in its hybrid action optimization framework and hardware constraint adaptation. In its structural design, this application explicitly models continuous and discrete actions through branching, outputting continuous UAV control quantities and discrete RIS phase strategies respectively via a dual-Actor network structure. This allows the optimization results to be directly mapped to the actual controllable hardware-level phase state, thus ensuring both high-precision optimization and hardware feasibility. This hybrid spatial collaborative optimization method breaks through the action type limitations of traditional reinforcement learning algorithms, providing a general algorithmic framework for collaborative communication between UAVs and intelligent reflectors.

[0091] Finally, this application has significant advantages in terms of system applicability and engineering scalability. The solution is not only applicable to single UAV-single RIS-multi-user scenarios, but can also be extended to multi-UAV collaborative networks to achieve joint security optimization and energy management among multiple nodes. Since the RIS is a passive reflective device, the overall power consumption of this system is extremely low, and the deployment cost is far lower than traditional relay systems, making it particularly suitable for critical scenarios with limited infrastructure, such as emergency rescue, post-disaster communication recovery, and network coverage in remote areas. The algorithm's decision latency is low, providing real-time response capabilities and enabling continuous high-efficiency operation in complex time-varying channel environments. Attached Figure Description

[0092] Figure 1 This is a schematic diagram of the system structure of this application.

[0093] Figure 2 This is a schematic diagram of the MADPO algorithm architecture in this application.

[0094] Figure 3 The average reward curves during the training process of different algorithms (DDPG, PPO, MADPO) in this application are shown.

[0095] Figure 4 This is a schematic diagram of the ablation experiment results of this application.

[0096] Figure 5 This is a performance comparison chart showing the impact of different numbers of RIS reflection units on the average reward of the system.

[0097] Figure 6 This is a comparison chart of system performance under different maximum transmit powers.

[0098] Figure 7 This is a flowchart of this application.

[0099] Figure 8 This is a comparison diagram of the complex cases involving multiple eavesdroppers with different numbers of reflective surfaces. Detailed Implementation

[0100] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential.

[0101] like Figure 1 As shown, this application discloses a RIS-assisted intelligent communication method for unmanned aerial vehicles (UAVs) based on deep reinforcement learning. This RIS-assisted UAV intelligent communication method is implemented through a communication system. The communication system includes an airborne communication node (UAV) serving as a multi-antenna transmission platform, a legitimate user equipment (UE), a potential eavesdropper (EAV), a ground-based intelligent reflector (RIS), and a central control module. The ground-based intelligent reflector (RIS) is composed of M... r ×M c Composed of N reflective units, the UAV is equipped with N a A transmission antenna transmitter and attitude control unit act as an airborne base station to provide signal transmission services to ground users. The UAV connects to the legitimate user equipment (UE) via a direct link or an indirect link reflected by a ground-based intelligent reflector (RIS) and sends data signals to the UE. Potential eavesdroppers (EAVs) attempt to intercept the transmitted data signals. Figure 1 The drone and the legitimate user equipment (UE) in the system simultaneously have a direct link and an indirect link reflected by the RIS array. The RIS array uses a phase matrix... The incident signal can be programmably controlled to achieve the dual effect of enhancing legitimate links and suppressing eavesdropping links. During system operation, the UAV and the ground-based intelligent reflector RIS receive decision commands from the central control module and dynamically adjust their position and phase configuration to adapt to channel changes, achieving efficient and secure communication.

[0102] like Figure 7 As shown, during system operation, the RIS-assisted UAV intelligent communication method includes the following steps;

[0103] Step 1: Collect and initialize input data: Obtain the initial positions of the UAV and the ground-based intelligent reflector RIS, the energy consumption of the UAV, establish a time-discrete model, and initialize channel parameters. This includes the following steps:

[0104] Step 1.1: Obtain the initial position Q of the unmanned aerial vehicle (UAV) u (0) Initial position Q of the reconfigurable ground smart reflector RIS r All users Q k ,k∈K and the eavesdropper Q eThe three-dimensional coordinates of (0) are given, and the three-dimensional position of the UAV at time n is denoted as Q. u ={x u [n],y u [n],h u [n]}:

[0105]

[0106] Among them, v h For horizontal velocity, v v The vertical velocity is represented by θ, the yaw angle is ω, and the pitch angle is x. u [n] represents the x-coordinate of the UAV at time n, v h [n] represents the horizontal velocity of the UAV at time n, y u [n] represents the ordinate of the UAV at time n, h u [n] represents the altitude coordinates of the UAV at time n, x u [n+1] is the x-coordinate of the UAV at time n+1, y u [n+1] is the ordinate of the UAV at time n+1, h u [n+1] represents the altitude coordinates of the drone at time n+1, v v [n] is the vertical velocity of the UAV at time n;

[0107] Step 1.2: The energy consumption of the UAV includes hovering power, rotor induced power, and maneuvering power. The total energy consumption e of the UAV is... u Given by the rotor aerodynamic model:

[0108]

[0109] Among them, P a Power P configured for a fixed rotor b For the induced energy requirement during hovering, P d For the power required for vertical maneuvering, U tip v is the rotor tip velocity. c denoted as the average downwash velocity during hovering, c0 as the drag coefficient, s as the rotor density ratio, ρ as the atmospheric density, and G as the rotor disk surface area. The numbers 2, 3, and 4 represent the 2nd, 3rd, and 4th powers, respectively.

[0110] Step 1.3: Establish a time discretization model: Let the total communication duration be... The number of time slots is The length of a single slot is

[0111] Step 1.4: Initialize channel parameters:

[0112] H ur (n),Hrk (n),H re (n)~Rayleigh(R)

[0113]

[0114] in, For the UAV to ground-based intelligent reflector RIS channel at time n, H re [n] represents the channel from the ground-based intelligent reflector RIS to the potential eavesdropper EAV at time n, H rk [n] represents the channel from the ground intelligent reflector RIS to the legitimate user equipment (UE) at time n, and d ur [n] represents the distance between the UAV and the ground-based intelligent reflector RIS, ζ represents the reference path loss when the distance d0 = 1m, α1 is the air-to-ground (A2G) path loss exponent, and d ur γ represents the Euclidean distance between the drone and the ground-based intelligent reflector RIS. r Rayleigh factor, The NLoS scattering component from the UAV to the ground-based intelligent reflector RIS is modeled as a circularly symmetric complex Gaussian random variable with the following distribution: d re The Euclidean distance between the ground-based intelligent reflector RIS and the potential eavesdropper EAV. The Loss component between the ground-based intelligent reflector RIS and the potential eavesdropper EAV. For the NLoS scattering component between the ground-based intelligent reflector RIS and the potential eavesdropper EAV, d rk The Euclidean distance between the ground-based intelligent reflector RIS and the legitimate user equipment (UE) is given. The Loss component between the ground-based intelligent reflector RIS and the legitimate user equipment (UE). The NLoS scattering components between the ground-based intelligent reflector RIS and the legitimate user equipment (UE), and the LoS components between the UAV and the ground-based intelligent reflector RIS are:

[0115]

[0116] Where λ represents wavelength, d i It is the distance between the reflective units in the ground-based intelligent reflective surface RIS.

[0117] and Let these represent the cosine and sine components of the angle reached on the horizontal plane, respectively. Corresponding to the sinusoidal component in the vertical plane, j is the imaginary unit, y r M represents the ordinate of the ground-based intelligent reflective surface RIS. c M is the number of rows in the RIS.r h is the number of columns in the RIS. r For the height coordinates of the ground-based intelligent reflector RIS, x r The x-axis represents the ground-based intelligent reflective surface RIS.

[0118] Step 2: Based on the input data collected in Step 1 (i.e., the initial positions of the UAV and the ground intelligent reflector RIS, and the energy consumption of the UAV), establish a time discretization model and initialize the channel parameters, and define the state space S:

[0119] S={Q u (n),Φ(n),H ur (n),H rk (n),D k (n)}

[0120] Among them, D k Let be the amount of remaining task data for user k, and Φ(n) be the phase matrix of the ground-based intelligent reflector RIS. Each element This matrix represents the phase shift applied by the corresponding reflecting unit and is located in space. middle.

[0121] Step 3: Construct the MADPO deep reinforcement learning algorithm based on hybrid action delay policy optimization, such as... Figure 2 As shown, the deep reinforcement learning algorithm MADPO includes a dual Actor network and a dual Critic network, transforming the state space constructed in step 2 into an input tensor X that the network can process. s (n) will be the input tensor X s (n) After inputting into the Actor network, a series of actions are generated. c (n) and discrete action a d (n). Overestimation of the Q-value is suppressed by minimizing the minimum value of the two Critic outputs. The algorithm employs Delayed Update and Prioritized Experience Replay (PER) mechanisms. Delayed Update ensures the smooth convergence of the policy network, while the PER mechanism improves learning efficiency by prioritizing high-value samples. The reward function R[n] comprehensively considers the confidentiality rate R. s [n], Energy consumption E tu The constraint term V[n] and the constraint penalty term V[n] ensure that the algorithm is stably optimized under multiple objective trade-offs. The algorithm structure can be implemented in software or deployed in a modular form in embedded controllers or cloud servers.

[0122] Specifically, step 3 includes the following steps:

[0123] Step 3.1, Normalization of the state space S and initialized channel parameters: Normalize the UAV position:

[0124]

[0125] Among them, Q max Q represents the maximum flight range of the drone. min Q represents the minimum flight range of the drone. u (n) represents the actual position;

[0126] Calculate the normalized channel gain for direct or indirect links:

[0127] |H ur | 2 ,|H rk | 2 ,|H re | 2

[0128] Map the normalized channel gain to the [0,1] interval;

[0129] Step 3.2: Construct the system state s(n) at time n as the input tensor X. s (n) is used as the state vector, which is input into the Actor network to generate the action space, which includes continuous actions a. c (n) and discrete action a d (n), continuous action a c (n) includes the trajectory, pitch angle θ, yaw angle ω, and horizontal acceleration a of the UAV. h Vertical acceleration a v Beamforming vector w; Discrete action a d (n) includes the phase matrix Φ(n) controlling the ground intelligent reflector RIS, and the set of phase quantization values ​​for the ground intelligent reflector RIS is as follows: L represents the number of discrete phase states.

[0130] Figure 3 The average reward curves of different algorithms (DDPG, PPO, MADPO) during the training process are shown. MADPO has the fastest convergence speed, the highest reward in the steady state and the smallest fluctuation, indicating that the algorithm in this application has superior convergence characteristics in the mixed action space. Figure 3 Simulation results show that, under the same number of training iterations, MADPO converges faster than PPO, and maintains a low-fluctuation reward curve under the high-dimensional RIS configuration (M=64), demonstrating good stability and generalization ability.

[0131] Figure 4The ablation experiment results verified the synergistic effect of delayed policy update and PER mechanism in improving algorithm performance: when either mechanism is removed, both training stability and final convergence effect decrease significantly.

[0132] Step 4: Based on the neural network, the continuous action a c (n) and discrete action a d (n) Generate hybrid actions and calculate immediate rewards to obtain an experience tuple (s(n), a) c (n),a d (n),r(n),s(n+1)).

[0133] Specifically, the steps include the following:

[0134] Step 4.1, Continuous motion sampling: Continuous motion a c (n) Use the tanh activation function to output control signals:

[0135]

[0136] Among them, noise item n b It is a threshold used to control the range of noise fluctuation, ensuring smoothness and exploratory nature of the action. To output a strategy for continuous actions in the Actor network, a min and a max It is a threshold used to control the range of output actions. The output range of the tanh activation function is [-1, 1], which is linearly mapped to the speed and angle constraint range of the UAV.

[0137] Step 4.2, Discrete Action Sampling: Discrete Action a d (n) Output the phase probability distribution of the ground-based intelligent reflector RIS using softmax:

[0138]

[0139] According to probability distribution sampling:

[0140] a d (n)~π φ,d (a|s)

[0141] The sampling results are then mapped to a phase matrix Φ[n].

[0142] Among them, z a For discrete action logit values, z b Let L be the logit value of the discrete-phase action candidate output of the Actor network RIS, where L is the number of discrete-phase states, b is the discrete action index, and π is the logit value. φ,dOutput strategy for discrete actions;

[0143] Step 4.3, Environment Execution and State Transition: Execute hybrid actions (a c (n),a d (n)), update the system state to s(n+1), and calculate the immediate reward r(n), obtaining the experience tuple (s(n), a c (n),a d (n), r(n), s(n+1)), where the instant reward is:

[0144] r(n) = c k [n]R s (k,n)-λ e E tu (k,n)-μV(k,n)

[0145] Among them, c k [n] is a binary variable indicating whether the user has completed the service they requested. k [n] = 0 indicates that the service has been satisfied; otherwise, c k [n] = 1, V(k,n) represents the penalties for exceeding limits and rate violations, λ e μ represents the energy consumption and constraint penalty weights, and E tu [n] represents the total energy consumption of the UAV's signal transmission and motion, E tu [n] = e t [n]+e u [n], where e t [n] represents the total power consumption of the UAV used for signal transmission: It is a pre-encoding vector for user k, N a For drones, the transmitting antenna is designed to optimize signal reception and minimize interference. u [n] represents the total power consumed by the drone, R s [n] represents the confidentiality rate, R s [n] = max(R) k [n]-R e [n],0),R k [n] and R e [n] represents the achievable rates of the legitimate user equipment (UE) and the potential eavesdropper (EAV), respectively. The goal is to maximize the average security and energy efficiency throughout the entire communication cycle.

[0146] R k [n] = log2(1+λ) k [n])

[0147] R e [n] = log2(1+λ) e [n])

[0148] Where, λ k [n] represents the signal-to-noise ratio of a legitimate user equipment (UE), and λ e [n] represents the signal-to-noise ratio of the potential eavesdropper's EAV:

[0149]

[0150] Among them, w k It is a pre-encoded vector for the user, w e It is a pre-coded vector for eavesdroppers, σ 2 It is additive white Gaussian noise, G k [n] = H uk [n]+(H rk [n]) H Φ[n]H ur [n] represents the total effective channel gain from the UAV to the legitimate user equipment (UE), including direct and reflected channels, G e [n] = H ue [n]+(H re [n]) H Φ[n]H ur [n] represents the total effective channel gain from the UAV to the potential eavesdropper EAV, including both direct and reflected channels, where H uk [n] represents the direct channel from the UAV to the legitimate user equipment (UE). H ue [n] represents the direct transmission channel from the drone to the potential eavesdropper EAV. U uk [n] and U ue [n] is a circularly symmetric complex Gaussian variable with zero mean and unit variance, and its distribution is [n]. α2 is the ground-to-ground path loss exponent, d uk and d ue H represents the Euclidean distance from the drone to the legitimate user equipment (UE) and to the potential eavesdropper (EAV), respectively. ur and H rk This refers to the channel from the drone to the RIS (Reflection Channel), and from the RIS to the user, which is the reflection channel from the drone to the user; H ur and H re This represents the channel from the drone to the RIS, and then from the RIS to the eavesdropper, i.e., the reflection channel from the drone to the eavesdropper.

[0151] Figure 5 The impact of different numbers of RIS reflection units M on the average reward of the system is described. Results show that MADPO maintains the highest performance stability as M increases. Experimental results demonstrate that, in typical urban scenarios, the algorithm proposed in this invention achieves a higher average security rate R0. sIt improves upon baseline algorithms (such as DDPG and PPO) by approximately 38.7%, and maintains stable security performance even in complex environments with multiple eavesdroppers cooperating in their interference, significantly enhancing the system's anti-interference and anti-eavesdropping capabilities.

[0152] Figure 6 It shows different maximum transmit power p max Despite changes in system performance, MADPO maintains an optimal balance between energy efficiency and security during the power boost phase, demonstrating the algorithm's energy adaptive characteristics.

[0153] Step 5: Store the empirical tuples obtained in Step 4 into the priority empirical replay pool to obtain high TD error samples. This includes the following steps:

[0154] Step 5.1: Convert the empirical tuple (s(n), a) c (n),a d (n), r(n), s(n+1)) are stored in the priority experience replay pool B. e Its priority is defined according to TD-error:

[0155] p i =|δ i |+ε

[0156] δ i =Q θ (s,a)-y

[0157] Where, p i For the priority of empirical tuples, |δ i | represents the absolute value of the timing difference error, ε is a small constant used to prevent empirical tuples from being unsampled when the timing difference error is 0, and δ i Let y be the time-series difference error, and Q be the target Q value. θ To predict the Q value;

[0158] Step 5.2: Store the data from Step 5.1 into the finite experience replay pool B. e The empirical tuples are sampled, and here the priority from the previous step is used to frequently sample samples with high time-series difference errors. The purpose of sampling is to obtain a batch of empirical tuples with high time-series difference errors, and the sampling probability is:

[0159]

[0160] Among them, samples with high TD error are selected more frequently to improve learning efficiency, resulting in sample batches. N m This refers to the batch size.

[0161] Step 6: Update the Critic network using high TD error samples, which includes the following steps:

[0162] Step 6.1: Calculate the target Q value using high TD error samples:

[0163]

[0164] in, To initialize the target network, s′ represents the system state at the next moment after s. for The strategy for consecutive actions after an update round for The discrete action policy after one round of updates, θ i θ represents the Critic network parameter. i The parameters obtained after one round of updates, where γ is the discount factor;

[0165] Step 6.2, Optimize using mean squared error loss:

[0166]

[0167] in, The Q-value, s, is the output of the Critic network. j Let a be the system state at time j. j Let y be the action output by the Actor network at time j. j Let θ be the target Q-value output by the Critic network at time j. i Let α be the parameter of one of the two Critic networks. c The learning rate of the Critic network is used to control the magnitude of its updates. The gradient of the Critic network is used to reduce overestimation through a dual-Critic structure.

[0168] Step 7, Delayed Update Mechanism: Perform an Actor network update every two rounds of Critic network updates to obtain the optimized policy parameters. θ i Target network parameters θ i Specifically, the steps include the following:

[0169] Step 7.1: Perform an Actor network update every two rounds of Critic network updates.

[0170]

[0171] Perform a soft update synchronously:

[0172] θ i ′←τθ i +(1-τ)θ i′,i∈{1,2}

[0173]

[0174] Where, α a τ is the learning rate of the Actor network, and τ∈(0,1) is used as a weight coefficient to control the update speed of the target network.

[0175] Step 8: When episode reaches the maximum iteration training period T max Then, output the optimal trajectory sequence. Optimal RIS phase matrix Φ * (n), Optimal beamforming matrix W * In addition, performance metrics include average security, energy efficiency, and convergence curve. Soft update means that for Actor and Critic networks, each update only updates a part of their respective networks, rather than replacing the entire network.

[0176] This application uses Hybrid Action Deep Reinforcement Learning (MADPO) as its core to achieve coordinated optimization of UAV trajectory, beamforming, and intelligent reflector (RIS) phase control. To ensure the applicability and engineering flexibility of the solution, this application provides the following alternative and extended implementation schemes for different scenarios, system scales, and hardware conditions. All of these schemes achieve the same or similar technical effects without changing the core idea of ​​the invention. First, at the algorithm structure level, the dual-scaffold Actor-Critic network of the MADPO algorithm can be replaced with other reinforcement learning architectures capable of handling hybrid action spaces. For example, the continuous part can use Deep Deterministic Policy Gradient (DDPG) or Soft Actor-Critic (SAC) algorithms to achieve continuous trajectory control of the UAV, while the discrete part can use Deep Q-Network (DQN) or Categorical Distributed Reinforcement Learning (CQN) for decision generation of discrete phase of the intelligent reflector. In addition, hierarchical reinforcement learning or multi-agent reinforcement learning can be used to replace the current single agent structure. Hierarchical decomposition can decouple and optimize the UAV flight strategy and RIS phase configuration, thereby further reducing algorithm complexity and training time in large-scale network or multi-node environments.

[0177] Secondly, regarding the reward function and optimization objective, the multi-objective joint optimization strategy of this application can be equivalently transformed according to the application scenario. The currently designed reward function...

[0178] r(n) = c k [n]R s (k,n)-λ e Etu (k,n)-μV(k,n),

[0179] With confidentiality rate, energy consumption, and quality of service (QoS) as the core indicators, the weighting parameter λ can be adjusted for use in other task scenarios. e λ can be dynamically adjusted, or additional objective function terms such as network latency, throughput, or user fairness can be introduced to achieve multi-dimensional adaptive trade-offs. For example, in emergency communication scenarios, λ can be reduced. e Prioritize secure transmission; in energy-constrained scenarios, λ can be increased. e To enhance energy efficiency, a fairness term can be added in multi-user, high-concurrency scenarios to prevent resource skew. This alternative solution, while maintaining the overall optimization framework, ensures system stability under varying performance requirements by flexibly adjusting weights and the objective function structure.

[0180] At the system architecture level, the single UAV and single RIS collaborative architecture of this application can be expanded into a multi-UAV and multi-RIS collaborative network to further improve communication coverage and system redundancy. In multi-UAV scenarios, collaborative optimization can be achieved through centralized or distributed reinforcement learning frameworks: the centralized scheme uses a central control module to uniformly schedule the action strategies of all UAVs and RIS nodes; the distributed scheme allows each UAV to make independent decisions and achieve collaborative learning through parameter sharing, thereby improving the scalability and real-time performance of the algorithm. In multi-RIS deployment scenarios, different RIS arrays can be assigned different reflection sub-regions according to terrain partitioning or user density to achieve parallel signal enhancement and interference suppression, which is particularly suitable for communication in densely populated urban high-rise building areas or disaster sites with complex terrain.

[0181] Regarding channel modeling and propagation environment substitution, the current system employs a Rayleigh fading model to simultaneously describe line-of-sight (LoS) and non-line-of-sight (NLoS) signal propagation. For applications in other frequency bands or specific environments, this can be replaced with a Rician fading, Nakagami-m fading, or millimeter-wave (mmWave) propagation model to match different signal characteristics. Correspondingly, the RIS phase adjustment mechanism can be extended to multi-band reflection control to support multi-carrier communication. If significant obstruction effects exist in complex environments, a joint channel estimation and learning mechanism can be introduced to ensure the MADPO algorithm maintains stable performance even under imperfect CSI conditions.

[0182] At the hardware adaptation level, the quantization precision L of the RIS discrete phase unit can be adjusted according to the actual hardware design, for example, extended from 4 levels (L=4) to 8 or 16 levels to improve the accuracy of reflected beam control. For different hardware platforms, the RIS control signal can also be generated programmatically through a software-defined radio frequency (SDR) module to enable coordinated operation with the UAV control system. In addition, the UAV dynamics model of this invention can be adjusted according to the differences in aircraft models. For example, on a fixed-wing UAV platform, an energy consumption-lift balance model can be used to replace the current rotor aerodynamic power model; if a hybrid UAV is used, a solar or fuel cell power supply model can be further introduced to enable the algorithm to maintain optimal energy efficiency decision-making under variable energy acquisition conditions.

[0183] Finally, regarding algorithm implementation and deployment, the reinforcement learning training process in this application can be executed offline or online. Offline training is suitable for long-term deployments in relatively stable scenarios. By pre-training the model on a high-performance server and exporting the policy network parameters, it can be deployed to the UAV and RIS control terminals to achieve rapid decision-making. Online training is suitable for task scenarios with significant dynamic environmental changes. Through a reinforcement learning process that updates as it executes, it adaptively responds to time-varying channels and sudden interference. In addition, the system can be implemented on different computing platforms: lightweight network structures (such as the Tiny-Actor model) can be deployed on embedded edge devices to reduce latency, or a complete MADPO model can be deployed in a centralized cloud, remotely controlling the UAV and RIS actions via downlink commands over a wireless link, thus balancing real-time performance and computing power requirements.

[0184] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A RIS-assisted intelligent communication method for unmanned aerial vehicles based on deep reinforcement learning, characterized in that: The RIS-assisted UAV intelligent communication method is implemented through a communication system, which includes a UAV serving as a multi-antenna transmission platform, a legitimate user equipment (UE), a potential eavesdropper (EAV), a ground-based intelligent reflector (RIS), and a central control module. The ground-based intelligent reflector (RIS) is composed of M... r ×M c Composed of a reflective unit, the UAV connects to the legitimate user equipment (UE) via a direct link or an indirect link reflected by the ground-based intelligent reflective surface (RIS), and sends data signals to the legitimate user equipment (UE). The potential eavesdropper (EAV) attempts to intercept the transmitted data signals. Specifically, during system operation, the RIS-assisted UAV intelligent communication method includes the following steps. Step 1: Collect and initialize input data: Obtain the initial positions of the UAV and the ground intelligent reflector RIS, the energy consumption of the UAV, establish a time discretization model, and initialize the channel parameters; Step 2: Based on the input data collected in Step 1 and the initialized channel parameters, define the state space S; Step 3: Construct the MADPO deep reinforcement learning algorithm based on hybrid action delay policy optimization. The MADPO deep reinforcement learning algorithm includes a dual Actor network and a dual Critic network, and transforms the state space constructed in Step 2 into the input tensor X. s (n) will be the input tensor X s (n) After inputting into the Actor network, a series of actions are generated. c (n) and discrete action a d (n); Step 4: Based on the neural network, the continuous action a c (n) and discrete action a d (n) Generate hybrid actions and calculate immediate rewards to obtain an experience tuple (s(n), a) c (n),a d (n),r(n),s(n+1)); Step 5: Store the empirical tuples obtained in Step 4 into the priority empirical replay pool to obtain high TD error samples. Step 6: Update the Critic network using samples with high TD error; Step 7, Delayed Update Mechanism: Perform an Actor network update every two rounds of Critic network updates to obtain the optimized policy parameters. θ i Target network parameters θ i '; Step 8: When episode reaches the maximum iteration training period T max Then, output the optimal trajectory sequence. Optimal RIS phase matrix Φ * (n), Optimal beamforming matrix W * And performance metrics, including average security level, energy efficiency, and convergence curve.

2. The RIS-assisted intelligent communication method for unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that: Step 1 specifically includes the following steps: Step 1.1: Obtain the initial position Q of the unmanned aerial vehicle (UAV) u (0) Initial position Q of the reconfigurable ground smart reflector RIS r All users Q k ,k∈K and the eavesdropper Q e The three-dimensional coordinates of (0) are given, and the three-dimensional position of the UAV at time n is denoted as Q. u ={x u [n],y u [n],h u [n]}: Among them, v h For horizontal velocity, v v The vertical velocity is represented by θ, the yaw angle is ω, and the pitch angle is x. u [n] represents the x-coordinate of the UAV at time n, v h [n] represents the horizontal velocity of the UAV at time n, y u [n] represents the ordinate of the UAV at time n, h u [n] represents the altitude coordinates of the UAV at time n, x u [n+1] is the x-coordinate of the UAV at time n+1, y u [n+1] is the ordinate of the UAV at time n+1, h u [n+1] represents the altitude coordinates of the drone at time n+1, v v [n] is the vertical velocity of the UAV at time n; Step 1.2: The energy consumption of the UAV includes hovering power, rotor induced power, and maneuvering power. The total energy consumption e of the UAV is... u for: Among them, P a Power P configured for a fixed rotor b For the induced energy requirement during hovering, P d For the power required for vertical maneuvering, U tip v is the rotor tip velocity. c denoted as the average downwash velocity during hovering, c0 as the airframe drag coefficient, s as the rotor compaction ratio, ρ as the atmospheric density, and G as the rotor disk surface area. Step 1.3: Establish a time discretization model: Let the total communication duration be... The number of time slots is The length of a single slot is Step 1.4: Initialize channel parameters: H ur (n),H rk (n),H re (n)~Rayleigh(R) in, For the UAV to ground-based intelligent reflector RIS channel at time n, H re [n] represents the channel from the ground-based intelligent reflector RIS to the potential eavesdropper EAV at time n, H rk [n] represents the channel from the ground intelligent reflector RIS to the legitimate user equipment (UE) at time n, and d ur [n] represents the distance between the UAV and the ground-based intelligent reflector RIS, ζ represents the reference path loss when the distance d0 = 1m, α1 is the air-to-ground (A2G) path loss exponent, and d ur γ represents the Euclidean distance between the drone and the ground-based intelligent reflector RIS. r Rayleigh factor, The NLoS scattering component from the UAV to the ground-based intelligent reflector RIS is distributed as follows: d re The Euclidean distance between the ground-based intelligent reflector RIS and the potential eavesdropper EAV. The Loss component between the ground-based intelligent reflector RIS and the potential eavesdropper EAV. For the NLoS scattering component between the ground-based intelligent reflector RIS and the potential eavesdropper EAV, d rk The Euclidean distance between the ground-based intelligent reflector RIS and the legitimate user equipment (UE) is given. The Loss component between the ground-based intelligent reflector RIS and the legitimate user equipment (UE). The NLoS scattering components between the ground-based intelligent reflector RIS and the legitimate user equipment (UE), and the LoS components between the UAV and the ground-based intelligent reflector RIS are: Where λ represents wavelength, d i It is the distance between the reflective units in the ground-based intelligent reflective surface RIS. and Let these represent the cosine and sine components of the angle reached on the horizontal plane, respectively. Corresponding to the sinusoidal component in the vertical plane, j is the imaginary unit, y r M represents the ordinate of the ground-based intelligent reflective surface RIS. c M is the number of rows in the RIS. r h is the number of columns in the RIS. r For the height coordinates of the ground-based intelligent reflector RIS, x r The x-axis represents the ground-based intelligent reflective surface RIS.

3. The RIS-assisted intelligent communication method for unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that: In step 2, the state space is defined: S={Q u (n),Φ(n),H ur (n),H rk (n),D k (n)} Among them, D k Let represent the remaining task data volume for user k, and Φ(n) be the phase matrix of the ground-based intelligent reflector RIS. Each element This matrix represents the phase shift applied by the corresponding reflecting unit and is located in space. middle.

4. The RIS-assisted intelligent communication method for unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that: Step 3 specifically includes the following steps: Step 3.1, Normalization of the state space S and initialized channel parameters: Normalize the UAV position: Among them, Q max Q represents the maximum flight range of the drone. min Q represents the minimum flight range of the drone. u (n) represents the actual position; Calculate the normalized channel gain for direct or indirect links: |H ur | 2 ,|H rk | 2 ,|H re | 2 Map the normalized channel gain to the [0,1] interval; Step 3.2: Construct the system state s(n) at time n as the input tensor X. s (n) is used as the state vector, which is input into the Actor network to generate the action space, which includes continuous actions a. c (n) and discrete action a d (n), continuous action a c (n) includes the trajectory, pitch angle θ, yaw angle ω, and horizontal acceleration a of the UAV. h Vertical acceleration a v Beamforming vector w; Discrete action a d (n) includes the phase matrix Φ(n) controlling the ground intelligent reflector RIS, and the set of phase quantization values ​​for the ground intelligent reflector RIS is as follows: L represents the number of discrete phase states.

5. The RIS-assisted intelligent communication method for unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that: Step 4 specifically includes the following steps: Step 4.1, Continuous motion sampling: Continuous motion a c (n) Use the tanh activation function to output control signals: Among them, noise item n b It is a threshold used to control the range of noise term fluctuation. To output a strategy for continuous actions in the Actor network, a min and a max It is a threshold used to control the range of output actions. The output range of the tanh activation function is [-1, 1], which is linearly mapped to the speed and angle constraint range of the UAV. Step 4.2, Discrete Action Sampling: Discrete Action a d (n) Output the phase probability distribution of the ground-based intelligent reflector RIS using softmax: According to probability distribution sampling: a d (n)~π φ,d (a|s) The sampling results are then mapped to a phase matrix Φ[n]. Among them, z a For discrete action logit values, z b Let L be the logit value of the discrete-phase action candidate output of the Actor network RIS, where L is the number of discrete-phase states, b is the discrete action index, and π is the logit value. φ,d Output strategy for discrete actions; Step 4.3, Environment Execution and State Transition: Execute hybrid actions (a c (n),a d (n)), update the system state to s(n+1), and calculate the immediate reward r(n), obtaining the experience tuple (s(n), a c (n),a d (n), r(n), s(n+1)), where the instant reward is: r(n)=c k [n]R s (k,n)-λ e E tu (k,n)-μV(k,n) Among them, c k [n] is a binary variable indicating whether the user has completed the service they requested. k [n] = 0 indicates that the service has been satisfied; otherwise, c k [n] = 1, V(k,n) represents the penalties for exceeding limits and rate violations, λ e μ represents the energy consumption and constraint penalty weights, and E tu [n] represents the total energy consumption of the UAV's signal transmission and motion, E tu [n] = e t [n]+e u [n], where e t [n] represents the total power consumption of the UAV used for signal transmission: It is a pre-encoding vector for user k, N a For the drone's transmitting antenna, e u [n] represents the total power consumed by the drone, R s [n] represents the confidentiality rate, R s [n] = max(R) k [n]-R e [n],0),R k [n] and R e [n] represents the achievable rates of legitimate User Equipment (UE) and potential eavesdropper EAV, respectively. R k [n]=log2(1+λ k [n]) R e [n]=log2(1+λ e [n]) Where, λ k [n] represents the signal-to-noise ratio of a legitimate user equipment (UE), and λ e [n] represents the signal-to-noise ratio of the potential eavesdropper's EAV: Among them, w k It is a pre-encoded vector for the user, w e It is a pre-coded vector for eavesdroppers, σ 2 It is additive white Gaussian noise, G k [n] = H uk [n]+(H rk [n]) H Φ[n]H ur [n] represents the total effective channel gain from the UAV to the legitimate user equipment (UE), including direct and reflected channels, G e [n] = H ue [n]+(H re [n]) H Φ[n]H ur [n] represents the total effective channel gain from the UAV to the potential eavesdropper EAV, including both direct and reflected channels, where H uk [n] represents the direct channel from the UAV to the legitimate user equipment (UE). H ue [n] represents the direct transmission channel from the drone to the potential eavesdropper EAV. U uk [n] and U ue [n] is a circularly symmetric complex Gaussian variable with zero mean and unit variance, and its distribution is [n]. α2 is the ground-to-ground path loss exponent, d uk and d ue These are the Euclidean distances from the drone to the legitimate user equipment (UE) and to the potential eavesdropper (EAV), respectively.

6. The RIS-assisted intelligent communication method for unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that: Step 5 specifically includes the following steps: Step 5.1: Convert the empirical tuple (s(n), a) c (n),a d (n), r(n), s(n+1)) are stored in the priority experience replay pool B. e Its priority is defined according to TD-error: p i =|δ i |+e δ i =Q θ (s,a)-y Where, p i For the priority of empirical tuples, |δ i | represents the absolute value of the timing difference error, ε is a small constant, and δ i Let y be the time-series difference error, and Q be the target Q value. θ To predict the Q value; Step 5.2: Store the data from Step 5.1 into the finite experience replay pool B. e The empirical tuples are sampled, with the sampling probability being: Among them, samples with high TD error were selected more frequently, resulting in sample batches. N m This refers to the batch size.

7. The RIS-assisted intelligent communication method for unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that: Step 6 specifically includes the following steps: Step 6.1: Calculate the target Q value using high TD error samples: in, To initialize the target network, s′ represents the system state at the next moment after s. for The strategy for consecutive actions after an update round for The discrete action policy after one round of updates, θ i ′ Critic network parameters θ i The parameters obtained after one round of updates, where γ is the discount factor; Step 6.2, Optimize using mean squared error loss: in, The Q-value, s, is the output of the Critic network. j Let a be the system state at time j. j Let y be the action output by the Actor network at time j. j Let θ be the target Q-value output by the Critic network at time j. i Let α be the parameter of one of the two Critic networks. c The learning rate of the Critic network is used to control the magnitude of its updates. The gradient of the Critic network is used to reduce overestimation through a dual-Critic structure.

8. The RIS-assisted intelligent communication method for unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that: Step 7 specifically includes the following steps: Step 7.1: Perform an Actor network update every two rounds of Critic network updates. Perform a soft update synchronously: i i ′ ←tth i +(1-τ)θ i ′ ,i∈{1,2} Where, α a τ is the learning rate of the Actor network, and τ∈(0,1) is used as a weight coefficient to control the update speed of the target network.

Images