Unmanned aerial vehicle secrecy rate optimization method based on evolutionary reinforcement learning

Through the evolution reinforcement learning algorithm, optimize the physical layer security of the UAV-assisted communication system, combined with the HTS model and resource allocation, the problem of eavesdropping and attack risks in the UAV-assisted communication system is solved, and efficient confidentiality rate and communication security are achieved. .

CN118921654APending Publication Date: 2024-11-08NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410854561.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-28
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

There are risks of eavesdropping and attack in drone-assisted communication systems. Traditional cryptography mechanisms face cracking risks in the era of quantum computing. Physical layer security technology operates in resource-constrained environments and consumes high energy, making it difficult to effectively optimize the confidentiality rate.

Method used

Adopting the drone confidentiality rate optimization method based on evolution reinforcement learning, the UAV-RIS system optimizes time slot allocation, RIS unit reflection, drone position and energy balance from the physical layer security angle, combined with the HTS model and resource allocation model to achieve Wireless charging and signal transmission reduces eavesdropper signals.

Benefits of technology

It improves the communication security of legitimate users, enhances the battery life and confidentiality rate of the drone, reduces eavesdropper interference, and is suitable for resource-constrained environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure FT_1
    Figure FT_1
  • Figure FT_2
    Figure FT_2
  • Figure FT_3
    Figure FT_3
Patent Text Reader

Abstract

The invention belongs to the field of secrecy rate optimization in a physical layer security technology, and discloses an unmanned aerial vehicle secrecy rate optimization method based on evolutionary reinforcement learning. The method comprises the following specific steps: deploying the proposed method in a secrecy rate optimization system; model parameters of an EGG-RL algorithm are initialized; the UAV-RIS receives a signal from the AP, and alternately performs energy capture and signal transmission in time and space of the RIS unit; in the energy capturing stage, an airborne battery of the unmanned aerial vehicle is wirelessly charged through the HTS model and the resource allocation model; in the signal transmission stage, signals are transmitted to legal users through the signal model, and communication signals of eavesdroppers are reduced through movement of the unmanned aerial vehicle. By introducing the EGG-RL algorithm, the unmanned aerial vehicle and the RIS, a more stable and higher-quality user secrecy rate can be provided. A large number of experimental results show that the method provided by the invention not only can ensure the electric quantity balance of the airborne battery of the unmanned aerial vehicle platform, but also can effectively reduce the influence of an eavesdropper and improve the overall secrecy rate of the system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0003] The invention relates to an unmanned aerial vehicle confidentiality rate optimization method based on evolutionary reinforcement learning, and belongs to the field of physical layer security. Background Art

[0004] The transmission of information in wireless communication networks faces unique security challenges. Since wireless networks lack the boundaries of physical media, data packets transmitted through wireless media can be captured by legitimate receivers, and there is also a risk of being intercepted by unauthorized third parties. This openness makes wireless communications vulnerable to passive monitoring and active interference, thereby increasing the possibility of information leakage and data tampering. In the field of secure information transmission, there are mainly two core solutions: one is a security strategy based on traditional cryptographic mechanisms, which protects the network layer and higher levels in the wireless communication system through encryption, identity authentication, and data integrity protection to ensure the security of information. This strategy relies on key encryption to transmit signals, making it difficult for potential eavesdroppers to crack within a limited time. However, with the advancement of quantum computing technology, existing computing power has been significantly improved, which not only shortens the potential cracking time, but also puts the originally complex key system at risk of being quickly cracked. In addition, since this technology requires complex key management and generation, its operation complexity and energy consumption are high, and it may not be suitable for use in resource-constrained environments. The other type is physical layer security technology based on information theory. This technology uses the characteristics of wireless channels, such as channel fading, noise, and multipath effects, to enhance the channel difference between legitimate receivers and potential eavesdroppers, thereby achieving wireless communication security without keys. Compared with traditional encryption technology, physical layer security technology has the following three main advantages: first, it does not require key generation and management, so it is not limited by the computing power of hardware devices; second, even when potential eavesdroppers have strong computing power, physical layer security technology can still provide excellent confidentiality performance; finally, physical layer security technology can adjust system parameters according to the real-time wireless channel status to optimize the security of communication. When the channel conditions of the legitimate receiver are better than those of the eavesdropper, physical layer security technology can achieve highly secure communication confidentiality. Summary of the invention

[0005] Aiming at the problem that UAV-assisted communication system is vulnerable to eavesdropping and attack, a model based on UAV-RIS to optimize confidentiality rate is constructed. The model includes: 1) eavesdroppers intercept data packets transmitted through wireless media; 2) there are multiple legitimate users with communication needs; 3) high-rise buildings block the direct signal transmitted by the base station; 4) UAVs carry RIS for auxiliary communication; 5) UAVs have delayed channels. By jointly optimizing time slot allocation, RIS unit reflection, RIS passive beamforming, UAV position and energy balance, the user total confidentiality rate maximization problem is established from the perspective of physical layer security.

[0006] The present invention discloses a method for optimizing the confidentiality rate of a UAV based on evolutionary reinforcement learning, the method comprising the following steps:

[0007] Step 1, deploying the proposed invention method in the optimized confidentiality rate system;

[0008] Step 2: Initialize the model parameters of the EGG-RL algorithm;

[0009] Step 3: UAV-RIS receives the signal from AP and performs energy capture and signal transmission in an interleaved manner in the time and space of the RIS unit, i.e., in the same time slot, the unit can perform both energy capture and signal transmission operations;

[0010] Step 4: In the energy capture phase, the UAV onboard battery is wirelessly charged through the HTS model and resource allocation model;

[0011] Step 5: In the signal transmission phase, the signal model is used to transmit signals to legitimate users, and the movement of drones reduces the communication signals of eavesdroppers.

[0012] The further improvement of the present invention is that: a UAV optimization confidentiality rate scenario model is constructed in which there is an eavesdropper and the signal transmission is blocked by buildings. The UAV-SSR-RIS system is deployed to assist the signal transmission from the wireless access point (Access Point, AP) to K single antennas. UT is expressed as The eavesdropper is represented by The antenna position of the kth UT at time slot t is denoted as A k (t) = (x k (t),y k (t),h k (t)). k (t) is the height of the kth UT antenna in the Cartesian coordinate system with the AP at the origin, x k (t),y k (t) is the horizontal position of the kth UT. The position of the pth eavesdropper can be expressed as A p (t) = (xp (t),y p (t),h p (t)),h p (t) is the height of the pth ED antenna in the Cartesian coordinate system with the AP at the origin, x p (t),y p (t) is the horizontal position of the pth ED. In this work, an AP with Z antennas sends a signal to the RIS system, which consists of The RIS surface element is represented by R. i,j The position i,j of the RIS surface element E in each time slot t is represented by and are the height and horizontal position of the RIS surface element R, respectively. In addition, the position of the RIS surface element is associated with the position of the drone. Without loss of generality, the RIS metasurface cell array is represented as In addition, RIS can exchange channel status information with the AP through a connected intelligent controller.

[0013] A further improvement of the present invention is that a model based on HTS is constructed.

[0014] To improve the endurance of the UAV by harvesting energy in the time domain, the UAV-SSR-RIS system is equipped with a rechargeable battery to store the harvested energy and convert it into electricity. For the simplicity of implementation, it is assumed that linear transmit precoding is used at the AP. For EH and reflected signals, the entire time period is divided into T equal time slots, expressed as Each time slot contains two phases: the Energy Harvest (EH) phase and the information transmission phase, considering the normalized unit time slot in the sequel. In the t-th time slot, the length of the EH phase is denoted as τ(t). Then, in the t-th time slot it is (1-τ(t)). In the EH phase, all reflection units only harvest energy. After the EH phase, the information transmission phase starts immediately, and all RIS surfaces are used to reflect signals in this phase. The AP's transmitted signal can be expressed as follows:

[0015]

[0016] in and S k is the precoding vector and the signal of the kth UT, S k is a circularly symmetric complex Gaussian random variable with zero mean and unit variance, that is, Therefore, the total transmit power at the AP is given by

[0017]

[0018] where ‖·‖ represents the Euclidean norm of the vector, and p max It is the upper limit of the AP's transmit power. k =‖V k ‖ 2 is the transmission power of the kth UT. Therefore, the RIS in the tth time slot can be expressed as follows:

[0019]

[0020] in is the Z antenna of the AP and the element surface element R i,j The channel vector between , and follows the path loss of the air-to-ground propagation model. In addition, the small-scale channel fading in the channel matrix Assume Rayleigh fading distribution. η∈(0,1) is the EH efficiency, is the transmission power of the AP. From the AP to each reflective element R i,j In the channel vector g i,j The path loss PL i,j , which can be expressed as:

[0021]

[0022] Where α is R i,j Path loss exponent to AP, is the additional attenuation factor caused by the NLoS connection, and P i,j (LoS) is the AP and the surface element R i,j The LoS probability between i,j (LoS) can be calculated according to the following formula:

[0023]

[0024] in and is a constant that depends on the environment. AP and the surface element R i,j The elevation angle between is given by:

[0025]

[0026] A further improvement of the present invention is that a resource allocation model is constructed.

[0027] In order to further improve the endurance of the UAV, a resource allocation model is designed to harvest energy on the dual domain. The UAV-SSR-RIS often operates in the communication interruption area. Unlike the HTS model, part of the metasurface in RIS is used to reflect the signal during the information transmission phase, while the rest of the metasurface in the system is used to harvest energy. At each time slot t, the energy harvested by the UAV-SSR-RIS can be redefined as:

[0028]

[0029] in, Represents the element R i,j Used to reflect the signal to the kth UT, otherwise Therefore, the energy harvesting efficiency of the RIS system at each time slot t can be defined as:

[0030]

[0031] in, is the total received energy from the incident RF signal in each time slot t.

[0032] A further improvement of the present invention is that a channel model is constructed

[0033] In the channel model, the passive reflection beamforming of the UAV-SSR-RIS system is considered. In the information transmission phase t, h r,k =[h 1,1 (k),...,h 1,N (k),...,h M,N (k)] and They represent the baseband equivalent channel from RIS to the kth UT and the baseband equivalent channel from AP to RIS respectively. A diagonal matrix Φ is defined as the reflection coefficient matrix of RIS, as shown below:

[0034]

[0035] in is an imaginary unit, represents the phase shift of the lth reflection unit, represents the amplitude reflection coefficient. Ideally set to unity, for simplicity, the antenna of each metasurface element can be independently controlled to maximize signal reflection efficiency. Based on the formula The kth UT passing through the channel can be expressed as follows:

[0036]

[0037] in, The additive white Gaussian noise at the kth UT has a noise power of is the channel matrix from RIS to the kth UT with RIS scheduling and can be expressed as:

[0038]

[0039] The present invention takes into account the path loss and small-scale fading h r,k , the path loss between RIS and UT is given by in represents the path loss exponent of the RIS-UT link, is the reflective element R i,j The distance between the kth UT and ‖·‖ 2 is the Euclidean norm. Corresponding to the path loss index at the reference distance d′=1m. Small-scale channel fading h in the channel r,k Assume that the Rician factor K rician =10, expressed as:

[0040]

[0041] in, They represent deterministic LoS and NLoS (Rayleigh fading) components respectively. Assume that each UT decodes the desired signal S k The interference from other RIS-UT links can be completely eliminated. Therefore, the received signal to noise ratio (SNR) at the kth UT is given by:

[0042]

[0043] According to Shannon's capacity formula, the throughput (in bits / second / Hz) of the kth UT during time slot t is given by:

[0044]

[0045] Where B is the bandwidth. The average throughput in each UT must be greater than or equal to the given Γ min To maintain service quality within a limited time frame.

[0046]

[0047] The throughput of the pth ED during time slot t is given by:

[0048]

[0049] The confidentiality rate from a specific UAV to the kth UT can be expressed as:

[0050]

[0051] Among them, [j] + =max(0,j). Therefore, the total confidentiality rate of the system is expressed as:

[0052]

[0053] In practical systems, perfect CSI is not available at the UAV due to transmission delay and processing delay as well as the mobility of the UAV and the user. When the RIS on the UAV transmits data stream to the user, the CSI may be outdated. Once such outdated CSI is used for transmission, it will lead to inevitable performance loss. Let T d The delay between the outdated CSI and the real-time CSI is called the delay between the outdated channel vector h(t) and the real-time channel vector h(t+T d ) can be expressed as:

[0054]

[0055] where Δh is the value of h(t+T d )and The independent and identically distributed error term is the autocorrelation function of the channel gain h(t), which is given by the zero-order Bessel function of the first kind: f D is the Doppler spread, denoted as f D =vf c / c, while v, f c , c represents the speed of the transceiver, the carrier frequency and the speed of light.

[0056] The formula for calculating the speed of the drone is:

[0057]

[0058] in, is the horizontal position of the RIS or UAV at time t.

[0059] Therefore, the energy consumption of drone flight is:

[0060]

[0061] Constant P i and P 0 represent the induced power and blade profile power in the hovering state, δ t represents each individual time slot, is the rotor blade tip speed, v 0is the average rotor-induced velocity in hover, d 0 is the fuselage drag ratio, s is the rotor density, ρ is the air density, A r is the rotor disk area.

[0062] A further improvement of the present invention is that a UAV trajectory method is introduced

[0063] In order to be as realistic as possible and to provide a stable communication environment, we will set a speed limit for the drone to keep it in a level flight state. Therefore, the maximum flight speed of the drone is v max =15m / s.

[0064] v(t)≤v max

[0065] The present invention attempts two methods of UAV deployment movement, namely, density-aware deployment method and Fermat point-based method:

[0066] 1) Density-aware deployment method

[0067] For the density-aware deployment method, UAVs are deployed at points that minimize the squared Euclidean distance and satisfy the following conditions:

[0068]

[0069] are the horizontal positions of the RIS and the kth UT, respectively. The value of can be obtained using the standard K-means algorithm.

[0070] 2) Method based on Fermat point

[0071] The trajectory of the UAV can be obtained by finding the horizontal Fermat points between all UTs in the Cartesian coordinate system. Unlike the K-means algorithm that minimizes the squared Euclidean distance, the Fermat point aims to minimize the sum of the Euclidean distances from a point to each vertex. Therefore, the deployment points obtained by the Fermat point-based method can be expressed as:

[0072]

[0073] The further improvement of the present invention is that the state and observation space, action space and reward design are designed according to the actual scenario.

[0074] State space: At each time step t, the observation is constructed from the current state of the environment s t , which includes the baseband equivalent channel G from AP to RIS and the channel gain from RIS to the kth UT Each RIS surface element R i,jThe distance between the kth UT The location of each RIS surface element The antenna position P of each UT k , each eavesdropper's antenna position C p And each RIS surface element R i,j The distance between the pth eavesdropper Therefore, observations on the proposed algorithm can be expressed as follows:

[0075]

[0076] Action space: At the tth time step, action a t It consists of three main components, namely, the length of the EH phase τ(t)∈[0,1], the transmission power p of each UT k ∈[0,p′ max ], and the phase shift for each reflective element l Reflection element dispatch variables In addition, τ(t), p k ,and is defined in the continuous feasible region, and are converted into discrete variables.

[0077]

[0078] Reward Design: The positive reward represents the goal of the proposed model, which is to maximize the overall secrecy rate of the UAV-SSR-RIS system. At each time step t, the instantaneous reward is positively correlated with the secrecy rate SSR. The proposed model must also take into account the minimum capability requirements of the users defined in the constraints. Therefore, the reward r t It can be described as follows:

[0079] r t =SSR×ρ

[0080] v is used to achieve the desired Γ min The number of UTs is defined as:

[0081]

[0082] Among them, ρ k (t) is:

[0083]

[0084] The cumulative reward is maxJ = ∑ t γ t r t Given.

[0085] The described UAV confidentiality rate optimization method based on evolutionary reinforcement learning can be used in communication scenarios where there are eavesdroppers and communications are blocked by buildings.

[0086] The beneficial effect of the present invention is that the present invention ensures the communication security of legitimate users by optimizing the confidentiality rate of the drone. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 Schematic diagram of the UAV confidentiality rate optimization system model based on evolutionary reinforcement learning

[0088] Figure 2 Algorithm framework process for optimizing the confidentiality rate of drones based on evolutionary reinforcement learning

[0089] Figure 3 Comparison results of the optimization and exhaustive search method for drone confidentiality rate based on evolutionary reinforcement learning, the traditional reinforcement learning algorithm TD3, the improved algorithm SD3 based on TD3, and the classic evolutionary reinforcement learning algorithm CEM-RL DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific implementation methods.

[0091] It should be noted here that in order to avoid unnecessary details that would obscure important parts of the present invention, the drawings only show structures or processing steps that are closely related to the present invention, while ignoring other details that are not very relevant to the present invention.

[0092] In addition, it should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0093] like Figure 1 The system model schematic diagram shown in the figure, in order to facilitate the understanding of the technical solution of the embodiment of the present application, firstly gives the specific execution process of the implementation of the present application:

[0094] Step 1, deploying the proposed invention method in the optimized confidentiality rate system;

[0095] Step 2: Initialize the model parameters of the EGG-RL algorithm, set the maximum step size, set the policy network parameters and population initial parameters in EGG-RL through random parameters, set the capacity of the historical gradient pool, and set the replay buffer capacity to 1e6;

[0096] Step 3: UAV-RIS receives the signal from AP and performs energy capture and signal transmission in an interleaved manner in the time and space of the RIS unit, i.e., in the same time slot, the unit can perform both energy capture and signal transmission operations;

[0097] Step 4: In the energy capture phase, the drone’s onboard battery is wirelessly charged through the HTS model and resource allocation model;

[0098] Step 5: During the signal transmission phase, the signal model is used to transmit signals to legitimate users, and the movement of drones reduces the communication signals of eavesdroppers.

[0099] Regarding step 1 of deploying the inventive method in the system for optimizing the confidentiality rate, the specific steps are as follows.

[0100] Step 11: Design the state space. At each time step t, the observation is constructed from the current environment state s t , which includes the baseband equivalent channel G from AP to RIS and the channel gain from RIS to the kth UT Each RIS surface element R i,j The distance between the kth UT The location of each RIS surface element The antenna position P of each UT k , each eavesdropper's antenna position C p And each RIS surface element R i,j The distance between the pth eavesdropper Therefore, observations on the proposed algorithm can be expressed as follows:

[0101]

[0102] Step 12: Design the action space. At the tth time step, action a t It consists of three main components, namely, the length of the EH phase τ(t)∈[0,1], the transmission power p of each UT k ∈[0,p′ max ], and the phase shift for each reflective element l Reflection element dispatch variables In addition, τ(t), p k ,and is defined in the continuous feasible region, and are converted into discrete variables.

[0103]

[0104] Step 13, reward design, the positive reward represents the goal of the proposed model, which is to maximize the overall confidentiality rate of the UAV-SSR-RIS system. At each time step t, the instantaneous reward is positively correlated with the confidentiality rate SSR. The proposed model must also take into account the minimum ability requirements of the users defined in the constraints. Therefore, the reward r t It can be described as follows:

[0105] r t =SSR×ρ

[0106] ρ is used to achieve the desired Γ min The number of UTs is defined as:

[0107]

[0108] Among them, ρ k (t) is:

[0109]

[0110] The cumulative reward is maxJ = ∑ t γ t r t Given.

[0111] The EGG-RL algorithm in step 2 will be introduced below.

[0112] Step 21: Initialize the evaluation network Q π and target evaluation network Initialize the replay buffer R and define its capacity as max_mem; initialize an actor network π with random parameters μ , whose network parameters can be regarded as an individual in the population; through known hyperparameters: capacity k, adaptive coefficient α and π μ The number of parameters is used to initialize the gradient buffer grad_buffer and noise generator noise_generator required in subsequent steps.

[0113] Step 22: Generate pop_size individuals through the cross entropy method (CEM), all of which can become parameters of an actor network.

[0114] Step 23: Use the TD3 method to perform reinforcement learning training on the first five individuals to optimize the strategy distribution.

[0115] Step 24: Evaluate each individual through the evaluation method and obtain the corresponding fitness value. At the same time, add the experience gained during the evaluation process to the replay buffer, record the step length, calculate the fitness value ranking list, and extract the one with the best fitness value based on the outstanding strategy.

[0116] Step 25: Determine whether it is in the warm-up stage by judging the conditions. If it is in the warm-up stage, sample the search direction through normal distribution, otherwise use noise_generator to sample the search direction. Add the sampled search direction to the outstanding ones, and evaluate all the generated individuals. Compare the obtained fitness value with the result (threshold) of multiplying the optimal fitness value obtained in the previous round of evaluation by the proportional coefficient. Individuals greater than the threshold will be recruited and enter the elite pool based on population iteration.

[0117] Step 26: Based on the calculated gradient of this round and putting it into grad_buffer, update the value of α according to formula (2), and use the parameters of the top 5 individuals in the elite pool to update the matrix in CEM. The hyperparameter α is updated as

[0118]

[0119] Among them, the hyperparameter α represents the proportional coefficient of using historical gradients, and Represent the two subspaces in the current iteration and The excellence of The number of sampling times in The number of sampling times;

[0120]

[0121] Among them, α t represents the value of α at time t, δ is a proportional factor greater than 1, and k 1 and k 2 are the lower and upper bounds of α.

[0122] Regarding the signal transmission of the UAV-SSR-RIS system in step 3, the specific steps are as follows.

[0123] Step 31: Determine the locations of legitimate users and eavesdroppers. The UAV-SSR-RIS system is deployed to assist the signal transmission from the wireless access point (AP) to K single antennas. UT is represented as The eavesdropper is represented by The antenna position of the kth UT at time slot t is denoted as A k (t) = (x k (t),y k (t),h k (t)). k (t) is the height of the kth UT antenna in the Cartesian coordinate system with the AP at the origin, x k (t),yk (t) is the horizontal position of the kth UT. The position of the pth eavesdropper can be expressed as A p (t) = (x p (t),y p (t),h p (t)),h p (t) is the height of the pth ED antenna in the Cartesian coordinate system with the AP at the origin, x p (t),y p (t) is the horizontal position of the pth ED.

[0124] Step 32, determine the AP antenna and RIS location. In this work, the AP with Z antennas sends a signal to the RIS system, which consists of The RIS surface element is represented by R. i,j The position i,j of the RIS surface element R in each time slot t is represented by and are the height and horizontal position of the RIS surface element R, respectively. In addition, the position of the RIS surface element is associated with the position of the drone. Without loss of generality, the RIS metasurface cell array is represented as

[0125] Step 33: Execute different steps according to different states. When the UAV-RIS is performing energy capture, execute step 4; when the UAV-RIS is performing signal transmission, execute step 5;

[0126] For the energy capture phase in step 4, the specific steps are as follows.

[0127] Step 41, split time slots. For simplicity of implementation, it is assumed that linear transmit precoding is used at the AP. For EH and reflected signals, the entire time period is divided into T equal time slots, expressed as Each time slot contains two phases: the energy harvesting (EH) phase and the information transmission phase, considering the normalized unit time slot in the sequel. At the t-th time slot, the length of the EH phase is denoted as τ(t). Then, at the t-th time slot is (1-τ(t)).

[0128] Step 42: In the EH phase, all reflection units only collect energy. After the EH phase, the information transmission phase begins immediately, and all RIS surfaces are used to reflect signals in this phase. The AP's transmission signal can be expressed as follows:

[0129]

[0130] in and S k is the precoding vector and the signal of the kth UT, S k is a circularly symmetric complex Gaussian random variable with zero mean and unit variance, that is,

[0131] Step 43: Calculate the total system transmit power. The total transmit power at the AP is given by:

[0132]

[0133] where ‖·‖ represents the Euclidean norm of the vector, and p max It is the upper limit of the AP's transmit power. k =‖V k ‖ 2 is the transmission power of the kth UT. The RIS in the tth time slot can be expressed as follows:

[0134]

[0135] in is the Z antenna of the AP and the element surface element R i,j The channel vector between , and follows the path loss of the air-to-ground propagation model. In addition, the small-scale channel fading in the channel matrix Assume Rayleigh fading distribution. η∈(0,1) is the EH efficiency, is the AP's transmission power.

[0136] Step 44: Calculate the path loss from the AP to each reflective element R i,j In the channel vector g i,j The path loss PL i,j , which can be expressed as:

[0137]

[0138] Where α is R i,j Path loss exponent to AP, is the additional attenuation factor caused by the NLoS connection, and P i,j (LoS) is the AP and the surface element R i,j The LoS probability between i,j (LoS) can be calculated according to the following formula:

[0139]

[0140] in and is a constant that depends on the environment. AP and the surface element R i,j The elevation angle between is given by:

[0141]

[0142] At each time slot t, the energy captured by UAV-SSR-RIS can be redefined as:

[0143]

[0144]

[0145] in, Represents the element R i,j Used to reflect the signal to the kth UT, otherwise

[0146] Step 45, calculate the system energy collection efficiency. The energy collection efficiency of the RIS system in each time slot t can be defined as:

[0147]

[0148] in, is the total received energy from the incident RF signal in each time slot t.

[0149] Regarding the signal transmission acquisition phase in step 5, the specific steps are as follows.

[0150] Step 51: Construct the channel. In the channel model, the passive reflection beamforming of the UAV-SSR-RIS system is considered. In the information transmission phase t, h in the time slot r,k =[h 1,1 (k),...,h 1,N (k),...,h M,N (k)] and They represent the baseband equivalent channel from RIS to the kth UT and the baseband equivalent channel from AP to RIS respectively. A diagonal matrix Φ is defined as the reflection coefficient matrix of RIS, as shown below:

[0151]

[0152] in is an imaginary unit, represents the phase shift of the lth reflection unit, represents the amplitude reflection coefficient. Ideally set to unity, for simplicity, the antenna of each metasurface element can be independently controlled to maximize signal reflection efficiency. Based on the formula The kth UT passing through the channel can be expressed as follows:

[0153]

[0154] in, The additive white Gaussian noise at the kth UT has a noise power of is the channel matrix from RIS to the kth UT with RIS scheduling and can be expressed as:

[0155]

[0156] The present invention takes into account the path loss and small-scale fading h r,k , the path loss between RIS and UT is given by in represents the path loss exponent of the RIS-UT link, is the reflective element R i,j The distance between the kth UT and ‖·‖ 2 is the Euclidean norm. Corresponding to the path loss index at the reference distance d′=1m. Small-scale channel fading h in the channel r,k Assume that the Rician factor K rician =10, expressed as:

[0157]

[0158] in, They represent deterministic LoS and NLoS (Rayleigh fading) components respectively. Assume that each UT decodes the desired signal S k Previously, interference from other RIS-UT links could be completely eliminated.

[0159] Step 52: Calculate the signal-to-noise ratio using the channel gain. The received signal-to-noise ratio (SNR) at the kth UT is given by:

[0160]

[0161] According to Shannon's capacity formula, the throughput (in bits / second / Hz) of the kth UT during time slot t is given by:

[0162]

[0163] Where B is the bandwidth. The average throughput in each UT must be greater than or equal to the given Γ min To maintain service quality within a limited time frame.

[0164]

[0165] Step 53: Calculate the data throughput. The throughput of the pth ED during the t time slot is given by:

[0166]

[0167] Step 54: Calculate the confidentiality rate. The confidentiality rate from a specific UAV to the kth UT can be expressed as:

[0168]

[0169] Among them, [j] + =max(0,j). Therefore, the total confidentiality rate of the system is expressed as:

[0170]

[0171] Step 55: Construct a time-delay channel. In the actual system, due to transmission delay and processing delay as well as the mobility of the UAV and the user, perfect CSI is not available at the UAV. When the RIS on the UAV transmits data stream to the user, the CSI may be outdated. Once such outdated CSI is used for transmission, it will lead to inevitable performance loss. Assume T d The delay between the outdated CSI and the real-time CSI is called the delay between the outdated channel vector h(t) and the real-time channel vector h(t+T d ) can be expressed as:

[0172]

[0173] where Δh is the value of h(t+T d )and The independent and identically distributed error term is the autocorrelation function of the channel gain h(t), which is given by the zero-order Bessel function of the first kind: f D is the Doppler spread, denoted as f D =vf c / c, while v, f c , c represents the speed of the transceiver, the carrier frequency and the speed of light.

[0174] Step 56: Calculate the speed of the drone. The speed calculation formula is:

[0175]

[0176] in, is the horizontal position of the RIS or UAV at time t.

[0177] Step 57, calculate the energy consumption of the drone. The energy consumption of the drone flight is:

[0178]

[0179] Constant P i and P 0 represent the induced power and blade profile power in the hovering state, δ t represents each individual time slot, is the rotor blade tip speed, v 0 is the average rotor-induced velocity in hover, d 0 is the fuselage drag ratio, s is the rotor density, ρ is the air density, A r is the rotor disk area.

[0180] Step 58: Construct the UAV international method. In order to be as realistic as possible and to provide a stable communication environment, we will set a speed limit for the UAV to keep it in a level flight state. Therefore, the maximum flight speed of the UAV is v max =15m / s.

[0181] v(t)≤v max

[0182] The present invention attempts two methods of UAV deployment movement, namely, density-aware deployment method and Fermat point-based method:

[0183] 1) Density-aware deployment method

[0184] For the density-aware deployment method, UAVs are deployed at points that minimize the squared Euclidean distance and satisfy the following conditions:

[0185]

[0186] are the horizontal positions of the RIS and the kth UT, respectively. The value of can be obtained using the standard K-means algorithm.

[0187] 2) Method based on Fermat point

[0188] The trajectory of the UAV can be obtained by finding the horizontal Fermat points between all UTs in the Cartesian coordinate system. Unlike the K-means algorithm that minimizes the squared Euclidean distance, the Fermat point aims to minimize the sum of the Euclidean distances from a point to each vertex. Therefore, the deployment points obtained by the Fermat point-based method can be expressed as:

[0189]

[0190] In summary, the present invention proposes a UAV confidentiality rate optimization method based on evolutionary reinforcement learning for confidentiality rate optimization, which fully considers the joint optimization of key factors such as hardware computing power, non-line-of-sight communication, UAV onboard energy, and beamforming of UAV or its auxiliary equipment, improves the robustness of the algorithm when deployed in actual scenarios, solves the problem of reduced confidentiality rate in multi-user and multi-eavesdropper scenarios, and enables UAV-RIS to use the characteristics of communication itself with the assistance of the algorithm to physically reduce the eavesdropper's signal, thereby achieving the purpose of improving the confidentiality rate.

[0191] The present invention proposes a method for optimizing the confidentiality rate of unmanned aerial vehicles based on evolutionary reinforcement learning. The method optimizes the confidentiality rate of users through an evolutionary reinforcement learning algorithm, and utilizes the characteristics of the communication itself to physically reduce the signal of the eavesdropper, so as to achieve the effect of improving the confidentiality rate.

[0192] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for optimizing the confidentiality rate of unmanned aerial vehicles based on evolutionary reinforcement learning, the method comprising the following steps: Step 1, deploying the proposed invention method in the optimized confidentiality rate system; Step 2: Initialize the model parameters of the EGG-RL algorithm, set the maximum step size, set the policy network parameters and population initial parameters in EGG-RL through random parameters, set the capacity of the historical gradient pool, and set the replay buffer capacity to 1e6; Step 3: UAV-RIS receives the signal from AP and performs energy capture and signal transmission in an interleaved manner in the time and space of the RIS unit, i.e., in the same time slot, the unit can perform both energy capture and signal transmission operations; Step 4: In the energy capture phase, the UAV onboard battery is wirelessly charged through the HTS model and resource allocation model; Step 5: In the signal transmission phase, the signal model is used to transmit signals to legitimate users, and the movement of drones reduces the communication signals of eavesdroppers.

2. The method for optimizing the confidentiality rate of unmanned aerial vehicles based on evolutionary reinforcement learning according to claim 1, characterized in that: Step 11: Design the state space. At each time step t, the observation is constructed from the current environment state s t , which includes the baseband equivalent channel G from AP to RIS and the channel gain from RIS to the kth UT Each RIS surface element R i,j The distance between the kth UT The location of each RIS surface element The antenna position P of each UT k , each eavesdropper's antenna position C p And each RIS surface element R i,j The distance between the pth eavesdropper Therefore, observations on the proposed algorithm can be expressed as follows: Step 12: Design the action space. At the tth time step, action a t It consists of three main components, namely, the length of the EH phase τ(t)∈[0,1], the transmission power p of each UT k ∈[0,p′ max ], and the phase shift for each reflective element l Reflection element dispatch variables j∈[0,N], In addition, τ(t), p k ,and is defined in the continuous feasible region, and are converted into discrete variables. Step 13, reward design, the positive reward represents the goal of the proposed model, which is to maximize the overall confidentiality rate of the UAV-SSR-RIS system. At each time step t, the instantaneous reward is positively correlated with the confidentiality rate SSR. The proposed model must also take into account the minimum ability requirements of the users defined in the constraints. Therefore, the reward r t It can be described as follows: r t =SSR×ρ ρ is used to achieve the desired Γ min The number of UTs is defined as: Among them, ρ k (t) is: The cumulative reward is maxJ = ∑ t γ t r t Given.

3. The method for optimizing the confidentiality rate of unmanned aerial vehicles based on evolutionary reinforcement learning according to claim 1, characterized in that: Step 21: Initialize the evaluation network Q π and target evaluation network Initialize the replay buffer R and define its capacity as max_mem; initialize an actor network π with random parameters μ , whose network parameters can be regarded as an individual in the population; through known hyperparameters: capacity k, adaptive coefficient α and π μ The number of parameters is used to initialize the gradient buffer grad_buffer and noise generator noise_generator required in subsequent steps. Step 22: Generate pop_size individuals through the cross entropy method (CEM), all of which can become parameters of an actor network. Step 23: Use the TD3 method to conduct reinforcement learning training on the first five individuals to optimize the strategy distribution. Step 24: Evaluate each individual through the evaluation method and obtain the corresponding fitness value. At the same time, add the experience gained during the evaluation process to the replay buffer, record the step length, calculate the fitness value ranking list, and extract the one with the best fitness value based on the outstanding strategy. Step 25: Determine whether it is in the warm-up stage by judging the conditions. If it is in the warm-up stage, sample the search direction through normal distribution, otherwise use noise_generator to sample the search direction. Add the sampled search direction to the outstanding ones, and evaluate all the generated individuals. Compare the obtained fitness value with the result (threshold) of multiplying the optimal fitness value obtained in the previous round of evaluation by the proportional coefficient. Individuals greater than the threshold will be recruited and enter the elite pool based on population iteration. Step 26: Based on the calculated gradient of this round and putting it into grad_buffer, update the value of α according to formula (2), and use the parameters of the top 5 individuals in the elite pool to update the matrix in CEM. The hyperparameter α is updated as Among them, the hyperparameter α represents the proportional coefficient of using historical gradients, and Represent the two subspaces in the current iteration and The excellence of The number of sampling times in The number of sampling times; Among them, α t represents the value of α at time t, δ is a proportional factor greater than 1, and k1 and k2 are the lower and upper limits of α.

4. The method for optimizing the confidentiality rate of unmanned aerial vehicles based on evolutionary reinforcement learning according to claim 1, characterized in that: Step 31: Determine the locations of legitimate users and eavesdroppers. The UAV-SSR-RIS system is deployed to assist the signal transmission from the wireless access point (AP) to K single antennas. UT is represented as The eavesdropper is represented by The antenna position of the kth UT at time slot t is denoted as A k (t) = (x k (t),y k (t),h k (t)). k (t) is the height of the kth UT antenna in the Cartesian coordinate system with the AP at the origin, x k (t),y k (t) is the horizontal position of the kth UT. The position of the pth eavesdropper can be expressed as A p (t) = (x p (t),y p (t),h p (t)),h p (t) is the height of the pth ED antenna in the Cartesian coordinate system with the AP at the origin, x p (t),y p (t) is the horizontal position of the pth ED. Step 32, determine the AP antenna and RIS location. In this work, the AP with Z antennas sends a signal to the RIS system, which consists of The RIS surface element is represented by R. i,j The position i,j of the RIS surface element R in each time slot t is represented by and are the height and horizontal position of the RIS surface element R, respectively. In addition, the position of the RIS surface element is associated with the position of the drone. Without loss of generality, the RIS metasurface cell array is represented as Step 33: Execute different steps according to different states. When the UAV-RIS is performing energy capture, execute step 4; when the UAV-RIS is performing signal transmission, execute step 5.

5. The method for optimizing the confidentiality rate of unmanned aerial vehicles based on evolutionary reinforcement learning according to claim 4 is characterized in that: Step 41, split time slots. For simplicity of implementation, it is assumed that linear transmit precoding is used at the AP. For EH and reflected signals, the entire time period is divided into T equal time slots, expressed as Each time slot contains two phases: the energy harvesting (EH) phase and the information transmission phase, considering the normalized unit time slot in the sequel. At the t-th time slot, the length of the EH phase is denoted as τ(t). Then, at the t-th time slot is (1-τ(t)). Step 42: In the EH phase, all reflection units only collect energy. After the EH phase, the information transmission phase begins immediately, and all RIS surfaces are used to reflect signals in this phase. The AP's transmission signal can be expressed as follows: in and S k is the precoding vector and the signal of the kth UT, S k is a circularly symmetric complex Gaussian random variable with zero mean and unit variance, that is, Step 43: Calculate the total system transmit power. The total transmit power at the AP is given by: where ‖·‖ represents the Euclidean norm of the vector, and p max It is the upper limit of AP’s transmit power. k =‖V k ‖ 2 is the transmission power of the kth UT. The RIS in the tth time slot can be expressed as follows: in is the Z antenna of the AP and the element surface element R i,j The channel vector between , and follows the path loss of the air-to-ground propagation model. In addition, the small-scale channel fading in the channel matrix Assume Rayleigh fading distribution. η∈(0,1) is the EH efficiency, is the AP's transmission power. Step 44: Calculate the path loss from the AP to each reflective element R i,j In the channel vector g i,j The path loss PL i,j , which can be expressed as: Where α is R i,j Path loss exponent to AP, is the additional attenuation factor caused by the NLoS connection, and P i,j (LoS) is the AP and the surface element R i,j The LoS probability between i,j (LoS) can be calculated according to the following formula: in and is a constant that depends on the environment. AP and the surface element R i,j The elevation angle between is given by: At each time slot t, the energy captured by UAV-SSR-RIS can be redefined as: in, Represents the element R i,j Used to reflect the signal to the kth UT, otherwise Step 45, calculate the system energy collection efficiency. The energy collection efficiency of the RIS system in each time slot t can be defined as: in, is the total received energy from the incident RF signal in each time slot t.

6. The method for optimizing the confidentiality rate of unmanned aerial vehicles based on evolutionary reinforcement learning according to claim 4 is characterized in that: Step 51: Construct the channel. In the channel model, the passive reflection beamforming of the UAV-SSR-RIS system is considered. In the information transmission phase t, h in the time slot r,k =[h 1,1 (k),...,h 1,N (k),...,h M,N (k)] and They represent the baseband equivalent channel from RIS to the kth UT and the baseband equivalent channel from AP to RIS respectively. A diagonal matrix Φ is defined as the reflection coefficient matrix of RIS, as shown below: in is an imaginary unit, represents the phase shift of the lth reflection unit, represents the amplitude reflection coefficient. Ideally set to unity, for simplicity, the antenna of each metasurface element can be independently controlled to maximize signal reflection efficiency. Based on the formula The kth UT passing through the channel can be expressed as follows: in, The additive white Gaussian noise at the kth UT has a noise power of is the channel matrix from RIS to the kth UT with RIS scheduling and can be expressed as: The present invention takes into account the path loss and small-scale fading h r,k , the path loss between RIS and UT is given by in represents the path loss exponent of the RIS-UT link, is the reflective element R i,j and the distance between the kth UT, ‖·‖2 is the Euclidean norm. Corresponding to the path loss index at the reference distance d′=1m. Small-scale channel fading h in the channel r,k Assume that the Rician factor K rician =10, expressed as: in, They represent deterministic LoS and NLoS (Rayleigh fading) components respectively. Assume that each UT decodes the desired signal S k Previously, interference from other RIS-UT links could be completely eliminated. Step 52: Calculate the signal-to-noise ratio using the channel gain. The received signal-to-noise ratio (SNR) at the kth UT is given by: According to Shannon's capacity formula, the throughput (in bits / second / Hz) of the kth UT during time slot t is given by: Where B is the bandwidth. The average throughput in each UT must be greater than or equal to the given Γ min To maintain service quality within a limited time frame. Step 53: Calculate the data throughput. The throughput of the pth ED during the t time slot is given by: Step 54: Calculate the confidentiality rate. The confidentiality rate from a specific UAV to the kth UT can be expressed as: Among them, [j] + =max(0,j). Therefore, the total confidentiality rate of the system is expressed as: Step 55: Construct a time-delay channel. In the actual system, due to transmission delay and processing delay as well as the mobility of the UAV and the user, perfect CSI is not available at the UAV. When the RIS on the UAV transmits data stream to the user, the CSI may be outdated. Once such outdated CSI is used for transmission, it will lead to inevitable performance loss. Assume T d The delay between the outdated CSI and the real-time CSI is called the delay between the outdated channel vector h(t) and the real-time channel vector h(t+T d ) can be expressed as: where Δh is the value of h(t+T d )and The independent and identically distributed error term is the autocorrelation function of the channel gain h(t), which is given by the zero-order Bessel function of the first kind: f D is the Doppler spread, denoted as f D =vf c / c, while v, f c , c represents the speed of the transceiver, the carrier frequency and the speed of light. Step 56: Calculate the speed of the drone. The speed calculation formula is: in, is the horizontal position of the RIS or UAV at time t. Step 57, calculate the energy consumption of the drone. The energy consumption of the drone flight is: Constant P i and P0 represent the induced power and blade profile power in the hovering state, respectively, δ t represents each individual time slot, is the tip speed of the rotor blade, v0 represents the average rotor-induced velocity in hover, d0 is the fuselage drag ratio, s is the rotor density, ρ is the air density, and A r is the rotor disk area. Step 58: Construct the UAV international method. In order to be as realistic as possible and to provide a stable communication environment, we will set a speed limit for the UAV to keep it in a level flight state. Therefore, the maximum flight speed of the UAV is v max =15m / s. v(t)≤v max The present invention attempts two methods of UAV deployment movement, namely, density-aware deployment method and Fermat point-based method: 1) Density-aware deployment method For the density-aware deployment method, UAVs are deployed at points that minimize the squared Euclidean distance and satisfy the following conditions: are the horizontal positions of the RIS and the kth UT, respectively. The value of can be obtained using the standard K-means algorithm. 2) Method based on Fermat point The trajectory of the UAV can be obtained by finding the horizontal Fermat points between all UTs in the Cartesian coordinate system. Unlike the K-means algorithm that minimizes the squared Euclidean distance, the Fermat point aims to minimize the sum of the Euclidean distances from a point to each vertex. Therefore, the deployment points obtained by the Fermat point-based method can be expressed as:

Citation Information

Cited By

  • Anti-interference unmanned aerial vehicle-IOS auxiliary wireless communication energy collection method

    CN119743782A

  • TD3 algorithm-based RIS-assisted mobile industrial robot simultaneous wireless information and power transfer optimization method

    CN120302316A

  • RIS-aided mobile industrial robot wireless power transfer optimization method based on td3 algorithm

    CN120302316B