Safe transmission method of intelligent reflective surface of unmanned aerial vehicle based on deep reinforcement learning
Through deep reinforcement learning, the parameters of drones and intelligent reflective surfaces are optimized, and the complexity problem of traditional methods is solved when optimizing the parameters of drones and intelligent reflective surfaces is realized, efficient and secure transmission of millimeter wave communication networks is achieved, and the confidentiality rate is improved and network security performance.
Patent Information
- Application Number
- CN202210477440.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-04
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-05-04
AI Technical Summary
The existing technology is difficult to effectively optimize the joint parameters of the drone and the intelligent reflection surface to improve the physical layer security performance of millimeter wave communication networks, especially when decoupling variables, the accuracy and algorithm complexity of modern communication networks are difficult to meet the high level of modern communication networks. Confidentiality and real-time requirements.
Using a deep reinforcement learning method, the beamforming vector of the drone and the phase shift matrix of the intelligent reflection surface are combined to optimize the average confidentiality rate by using the Proximal Policy Optimization algorithm to maximize the average confidentiality rate, deal with non-convex optimization problems, and reduce the complexity of the algorithm. .
It significantly improves the security performance of the intelligent reflective surface communication network of the drone, reduces optimization time, is suitable for online deployment, and has the ability to migrate and apply, and can provide efficient and secure transmission solutions when traditional methods cannot be effectively solved.
Smart Images

Figure CN115052285B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of wireless communications, and in particular relates to a parameter optimization method for secure transmission of an unmanned aerial vehicle intelligent reflective surface communication network based on deep reinforcement learning. Background Art
[0002] The fifth-generation (5G) communication network has higher requirements for coverage and transmission rate. Unmanned aerial vehicles (UAVs) can use their strong mobility to provide ubiquitous and reliable connections for ground nodes, and reconfigurable intelligence surfaces (IRS) have also attracted more and more attention due to their ability to effectively reconstruct the electromagnetic wave propagation environment. They are considered to be a technology with great potential in the next generation of communication networks. The combination of UAVs and IRSs can bring higher capacity and energy efficiency to communication networks. In addition, the strong line-of-sight (LoS) link properties of air-to-ground channels and the three-dimensional beamforming capabilities of IRSs can also compensate for the severe path loss of millimeter-wave transmission and alleviate its sensitive blocking effect.
[0003] However, despite the many advantages that UAV and IRS bring to communication networks, security threats in communication networks have become an urgent problem to be solved due to the broadcast characteristics of wireless channels. Physical layer security technology uses the randomness and reciprocity of communication media to achieve the purpose of secure transmission through signal processing, modulation and coding methods, which is a powerful supplement to the upper layer based on cryptography encryption technology. The application of physical layer security technology in UAV communication networks can seek the best compromise between confidential transmission and energy efficiency, which is also consistent with the characteristics of IRS that no additional energy is required. The key to enhancing the physical layer security performance of IRS-assisted UAV millimeter wave communication networks lies in the joint optimization of the UAV beamforming vector and trajectory and the IRS phase shift matrix. Traditional methods generally decouple these three variables and then optimize them separately. At the same time, the optimization of the IRS phase shift matrix often requires approximations such as semidefinite relaxation (SDR) and minimization (MM). In terms of solution accuracy and algorithm complexity, it is difficult to meet the high confidentiality and real-time requirements of modern communication networks. Therefore, it is urgent to invent a secure transmission method that can quickly and effectively optimize communication network parameters to enhance the security performance of the physical layer. Summary of the invention
[0004] Technical issues to be solved
[0005] In order to avoid the shortcomings of the prior art, the present invention proposes a safe transmission method for an intelligent reflective surface of a UAV based on deep reinforcement learning.
[0006] Technical Solution
[0007] A method for secure transmission of an intelligent reflective surface of a drone based on deep reinforcement learning, characterized by the following steps:
[0008] Step 1: Assume that the UAV acts as an aerial base station to transmit millimeter waves to legitimate communication nodes on the ground. At the same time, there is a malicious eavesdropper on the ground to eavesdrop on the confidential link. The UAV is equipped with multiple antennas, and the ground nodes are equipped with single antennas. In order to enhance the security performance of the communication network, an intelligent transmitting surface is further deployed in the network to maximize the performance difference between the confidential channel and the eavesdropping channel through flexible regulation of its phase. When the UAV performs its mission, that is, to provide services to legitimate receivers on the ground, it will fly at a fixed altitude within a certain period of time T. In order to improve the security performance of the communication network, it is necessary to jointly optimize the beamforming vector and trajectory of the UAV and the phase shift matrix of the IRS.
[0009] Step 2: Since continuous time will make optimization difficult, first discretize the flight time T, that is, T = NΔ t , where N is the total number of time slots, Δ t is the length of a single time slot; thus, the problem of enhancing the security performance of the communication network is reduced to the problem of maximizing the average confidentiality rate of the legitimate receiver, and the optimization variables include the beamforming vector and position of the UAV at each time slot n, and the phase shift matrix of the IRS; however, even if the time is discretized, the optimization problem is still non-convex and difficult to solve directly;
[0010] Step 3: This optimization problem is difficult to solve directly, so a DRL-based algorithm is proposed to optimize the variables simultaneously; DRL is good at solving dynamic optimization problems and can obtain a strategy π for taking a certain action under a specific state. θ ; First, the average confidentiality rate maximization problem is reduced to a Markov decision process, where the state s at time slot n n is the CSI and UAV position of the legitimate user and the eavesdropper, action a n is the beamforming vector and position of the UAV and the phase shift matrix of the IRS, and reports r n is the confidentiality rate R s[n]; This setting enables the agent to maximize the average confidentiality rate within the flight time; after that, the proximal policy optimization (PPO) algorithm in DRL is used to solve it, where the policy network is used to output the action to be performed according to the current state, and the update gradient is:
[0011]
[0012] in Used to measure the difference between the new and old strategies, is the advantage function ∈ and the clipping parameter; the value network is used to evaluate the difference between the predicted value and the actual value of the action, and its updated objective function is:
[0013]
[0014] in Indicates state s n The value of the strategy network; through continuous trial and error and training of the strategy network and the value network, the optimal strategy can be obtained, and a safe transmission method for the UAV intelligent reflective surface communication network can be given accordingly.
[0015] A computer system, characterized in that it comprises: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.
[0016] A computer-readable storage medium, characterized in that it stores computer-executable instructions, which are used to implement the above method when executed.
[0017] A computer program, characterized by comprising computer executable instructions, which are used to implement the above method when executed.
[0018] Beneficial Effects
[0019] The UAV intelligent reflective surface security transmission method based on deep reinforcement learning proposed in this invention has the following advantages compared with the traditional transmission method without introducing IRS and performing parameter optimization:
[0020] (1) Improved the security performance of transmission.
[0021] By introducing an additional IRS and flexibly adjusting the phase of its reflection unit, the reception quality at the legitimate user is maximized, and the transmission beamforming vector and trajectory of the drone are optimized to minimize the leakage of confidential signals to eavesdroppers, so that the drone can be as close to the legitimate user as possible and away from the eavesdropper during flight, thereby improving the average confidentiality rate of transmission. Simulation experiments show that the present invention can effectively improve the security performance of the drone intelligent reflection surface communication network.
[0022] (2) Suitable for online deployment and migration applications.
[0023] Deep reinforcement learning originates from dynamic programming and is good at handling sequence optimization problems. It does not require sub-problem decomposition and complex convex approximation to handle the proposed average confidentiality rate maximization problem. The algorithm complexity is lower, and the advantages are more obvious when the number of UAV antennas, the number of IRS reflection units, and the number of time slots N increase. Therefore, it will take less time to optimize the network parameters. In addition, the trained policy network can also be used as the initial model for similar optimization problems, and has a certain migration application capability. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.
[0025] Figure 1 This is a model diagram of the UAV millimeter wave communication network assisted by the intelligent reflective surface proposed in the present invention;
[0026] Figure 2 This is a comparison chart of the confidentiality rate of each time slot between the secure transmission method based on deep reinforcement learning proposed in the present invention and the traditional method based on sequence convex approximation optimization under the condition that the number of time slots N=80;
[0027] Figure 3 This is a comparison chart of the UAV trajectory between the deep reinforcement learning-based secure transmission method proposed in the present invention and the traditional sequence convex approximation optimization method under the condition of time slot number N=80. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0029] Reference Figure 1, the UAV acts as an aerial base station to serve the legitimate receiver (Bob) on the ground. At the same time, there is an eavesdropper (Eve) on the ground. In order to further enhance the security of transmission, an IRS is introduced. A three-dimensional Cartesian coordinate system is established, and the coordinates of Bob, Eve and IRS are c b =[x b ,y b ,0] T , c e =[x e ,y e ,0] T and c R =[0,y R ,z R ] T , IRS deployed on the yoz plane. UAV equipped with N t The number of reflection units of IRS is M=M. y ×M z The trajectory sequence of the UAV can be expressed as c U [n] = [x U [n],y U [n],z U ] T ,n=0,1,...,N. In this way, the received signals of Bob and Eve in time slot n can be expressed as:
[0030]
[0031]
[0032] The time slot n is omitted for simplicity of expression. is the channel vector from UAV to Bob (Eve), is the channel vector from IRS to Bob (Eve). is the vectorized IRS channel matrix, is the channel matrix between UAV and IRS, and s are the transmit beamforming vector and transmit symbol at the UAV, respectively, and s satisfies n b and n e are the additive Gaussian white noises at Bob and Eve, with mean 0 and variances respectively. and Then the transmission rates of Bob and Eve in time slot n are:
[0033]
[0034]
[0035] The confidentiality rate is R sec [n]=[R b [n]-R e [n]] + , [·] + represents the maximum value between and 0. Next, the average confidentiality rate maximization problem can be expressed as:
[0036]
[0037] sttr(f[n]f H [n])≤P max ,
[0038] φ m [n]∈[0,2π),
[0039]
[0040] in P max is the maximum transmission power of UAV, c0 is the initial position of UAV, V max is the maximum speed of the UAV, δ x [n] and δ y [n] is the displacement of the UAV along the x-axis and y-axis respectively. However, the objective function of the optimization problem is non-convex with respect to the variables, and the constant modulus constraint of the IRS phase shift matrix is also non-convex, which makes it difficult to solve using traditional methods. Therefore, we propose a DRL-based solution method based on the PPO algorithm.
[0041] First, we summarize the average confidentiality rate maximization problem as an MDP. In general, an MDP can be composed of a five-tuple<S,A,P,R,γ> Represented as follows, where S is the state space, A is the action space, P is the state transition probability, R is the return, and γ is the discount factor. n The total discounted return at the beginning is The agent's goal is to maximize G by optimizing the strategy π(a|s) n expectations, that is
[0042]
[0043] For the average confidentiality rate maximization problem considered, the agent is the whole composed of UAV and IRS, and the other nodes of the network and the propagation environment together constitute the environment with which it interacts. Considering time slot n as the nth step of MDP, the MDP corresponding to the average confidentiality rate maximization problem is described in detail as follows:
[0044] 1) State space S. Since the basic basis for network parameter optimization is the channel state information (CSI) of Bob and Eve, the state space of MDP should include this information. In order to simplify S, the channel from UAV to IRS is further cascaded with the channel from IRS to each node, that is, Then the state of step n can be expressed as:
[0045] s n ={h UB [n],h UE [n],H CB [n],H CE [n],c U [n]}
[0046] Note that CSI has both real and imaginary parts, and for the convenience of inputting into the neural network, the real and imaginary parts need to be separated. n The total dimension is 4N t (M+1)+3.
[0047] 2) Action space A. The agent's action should include the UAV's velocity and beamforming vector f[n] and the IRS phase shift matrix Φ[n]. The real and imaginary parts of f[n] are output by the policy network and then merged, and Φ[n] is converted to phase to satisfy the constant modulus constraint, that is, Therefore, the action of step n can be expressed as
[0048]
[0049] Its total dimension is 2N t +M+2.
[0050] 3) State transition probability P. The state transition probability of MDP is defined as the relationship between the agent and the state s n Take action a n Then transfer to state s n+1 The probability of can be expressed as Obviously, we do not consider circuit and mechanical failures here, so we ignore the state transition probability P.
[0051] 4) Return R. Since our goal is to maximize the average confidentiality rate, the return R of the nth step n It can be directly equal to the confidentiality rate of time slot n, that is, r n =R sec [n].
[0052] At this point, the MDP transformation is complete. Since the Clip version of the PPO algorithm has better performance and is simpler in form, we solve the original optimization problem based on the Clip version of the PPO algorithm and describe the proposed DRL-based optimization algorithm as follows:
[0053] The agent's state at step n is s n , take action a n After reaching state s n+1 and get instant returns n In this process, s n ,a n ,r n ,s n+1 After the agent interacts with the environment a certain number of times, the training for round J begins: First, the policy network θ is updated by maximizing the following objective function i :
[0054]
[0055] Then maximize the following objective function to update the value network
[0056]
[0057] The training is stopped after reaching a certain number of iterations I.
[0058] The specific effects of the present invention can be further illustrated by the following simulation:
[0059] The coordinates of IRS, Bob and Eve are [0,30,10] T , [50,50,0] T and [30,50,0] T , the initial position of UAV is c0 = [0,0,20] T The flight altitude is fixed at 20m. The UAV is equipped with a 4-element uniform linear array, and the number of reflective units of the IRS is 4×4. The maximum transmit power of the UAV is P max 30dBm, maximum speed V max is 10m / s. The variance of the channel noise is -100dBm, and the discrete time interval Δ t is 0.25s. The total number of time slots N is 80, that is, the total flight time is 20s. The total number of iterations in the PPO algorithm is I=500, the number of training rounds is J=10, and the discount factor is γ=0.9. Experiments were conducted in a simulation environment to obtain the values of the confidentiality rate in each time slot and the specific trajectory of the UAV for the method proposed in the present invention and the traditional method, as shown in the figure. Figure 2 and Figure 3 shown.
[0060] from Figure 2It can be seen that the DRL-based optimization method proposed in the present invention has a slightly lower confidentiality rate than the traditional method only in a short period of time at the beginning of the flight. sec Compared with the traditional method, it has obvious advantages. From the actual trajectory of the UAV, both methods are to fly as fast as possible to Bob's vicinity. The difference is that the proposed method can actively find a position that is more conducive to safe transmission, giving full play to the advantages of UAV's high maneuverability. In summary, the DRL-based optimization method proposed in the present invention can effectively enhance the safe transmission of the IRS-assisted UAV millimeter wave communication network.
[0061] In summary, the proposed UAV intelligent reflective surface secure transmission method based on deep reinforcement learning can be specifically described as follows: in order to enhance the secure transmission of the UAV communication network, an additional IRS is introduced into the communication network, and the UAV's beamforming vector and trajectory and the IRS phase shift matrix are optimized during the flight of the UAV. The UAV first flies to the vicinity of the legitimate user as quickly as possible, and then actively seeks a position that is beneficial to safety to carry out transmission; in this process, the main lobe of the UAV's transmitting beam and the main lobe of the reflected beam of the IRS are aligned with the legitimate user.
[0062] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should be included in the protection scope of the present invention.
Claims
1. A method for secure transmission of intelligent reflective surfaces of unmanned aerial vehicles based on deep reinforcement learning, characterized in that Here are the steps: Step 1: Assume that the UAV acts as an aerial base station to transmit millimeter waves to legitimate communication nodes on the ground. At the same time, there is a malicious eavesdropper on the ground to eavesdrop on the confidential link. The UAV is equipped with multiple antennas, and the ground nodes are equipped with single antennas. In order to enhance the security performance of the communication network, an intelligent transmitting surface is further deployed in the network to maximize the performance difference between the confidential channel and the eavesdropping channel through flexible regulation of its phase. When the UAV performs its mission, that is, to provide services to legitimate receivers on the ground, it will fly at a fixed altitude within a certain period of time T. In order to improve the security performance of the communication network, it is necessary to jointly optimize the beamforming vector and trajectory of the UAV and the phase shift matrix of the IRS. Step 2: Since continuous time will make optimization difficult, first discretize the flight time T, that is, T = NΔ t , where N is the total number of time slots, Δ t is the length of a single time slot; thus, the problem of enhancing the security performance of the communication network is reduced to the problem of maximizing the average confidentiality rate of the legitimate receiver. The optimization variables include the beamforming vector and position of the UAV at each time slot n, and the phase shift matrix of the IRS; however, even if the time is discretized, the optimization problem is still non-convex and difficult to solve directly; Step 3: The optimization problem is difficult to solve directly, so a DRL-based algorithm is proposed to optimize the variables simultaneously; DRL is good at solving dynamic optimization problems and can obtain a strategy π for taking a certain action under a specific state. θ ; First, the average confidentiality rate maximization problem is reduced to a Markov decision process, where the state s at time slot n n is the CSI and UAV position of the legitimate user and the eavesdropper, action a n is the beamforming vector and position of the UAV and the phase shift matrix of the IRS, and reports r n is the confidentiality rate R s [n]; This setting enables the agent to maximize the average confidentiality rate during flight time; after that, the proximal policy optimization algorithm in DRL is used to solve it, where the policy network is used to output the action to be performed based on the current state, and the updated gradient is: in Used to measure the difference between the new and old strategies, is the advantage function, ∈ is the clipping parameter; the value network is used to evaluate the difference between the predicted value and the actual value of the action, and its updated objective function is: in Indicates state s n The value of the strategy network; through continuous trial and error and training of the strategy network and the value network, the optimal strategy can be obtained, and a safe transmission method for the UAV intelligent reflective surface communication network can be given accordingly.
2. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.
3. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.
4. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when being executed.
