Unmanned aerial vehicle communication secrecy energy efficiency optimization method and system based on deep reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle communication secrecy energy efficiency optimization method and system based on deep reinforcement learning. The method comprises the following steps: constructing an intelligent reflector-assisted NOMA unmanned aerial vehicle communication network security secrecy energy efficiency optimization model; performing iterative optimization solution on the intelligent reflector-assisted NOMA unmanned aerial vehicle communication network security secrecy energy efficiency optimization model to obtain unmanned aerial vehicle communication network security secrecy parameters; setting a state vector, an action space and a reward function of a deep reinforcement learning algorithm, and constructing a deep reinforcement learning optimization algorithm; and performing iterative optimization on the security secrecy parameters of the unmanned aerial vehicle communication network based on a deep reinforcement learning optimization algorithm to obtain the optimal unmanned aerial vehicle NOMA secrecy communication energy efficiency. According to the embodiment of the invention, deep reinforcement learning can be combined and the flight path energy loss of the unmanned aerial vehicle is considered, so that the confidential energy efficiency in an unmanned aerial vehicle communication system is improved. The method can be widely applied to the technical field of unmanned aerial vehicle network communication energy efficiency.

Description

Technical Field

[0001] The present application relates to the field of drone network communication energy efficiency technology, and in particular to a method and system for optimizing drone communication confidentiality energy efficiency based on deep reinforcement learning. Background Art

[0002] Compared with traditional ground communication systems, UAV communication systems have unique advantages. By utilizing the characteristics of UAV mobility, flexibility and line-of-sight connection, they can improve network coverage, connectivity and transmission rate in communication systems, while reducing transmission delay. Therefore, UAV communication systems bring new possibilities and solutions to existing communication networks. Related technical research includes the limited energy and communication security issues of UAVs and the combination of non-orthogonal multiple access (NOMA) technology and reconfigurable intelligent surfaces (RIS). However, in the field of wireless communication network optimization, traditional optimization techniques often require complex mathematical derivation and repeated iterations to obtain the optimal solution. Especially when it comes to mobile elements such as UAVs, the difficulty of problem solving is multiplied due to the environmental dynamics brought by them. Therefore, the related technology uses traditional convex optimization methods to deal with the problem of UAV confidentiality energy efficiency, which requires complex formula derivation and iteration to solve the problem. In the communication system of RIS-assisted UAVs, most of the research only stays on solving the problem of maximizing the confidentiality rate, while ignoring the energy consumption of UAVs. Therefore, there is a lack of research on the confidentiality energy efficiency of RIS-assisted UAV and user communication systems.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method and system for optimizing the confidentiality energy efficiency of drone communications based on deep reinforcement learning, which can combine deep reinforcement learning and consider the energy loss of drone flight trajectories to improve the confidentiality energy efficiency in drone communication systems.

[0005] To achieve the above objectives, an embodiment of the present application proposes a method for optimizing the energy efficiency of drone communication security based on deep reinforcement learning, the method comprising:

[0006] Construct a security, confidentiality and energy efficiency optimization model for NOMA UAV communication network assisted by intelligent reflective surfaces;

[0007] Iteratively optimize and solve the energy efficiency optimization model for the NOMA UAV communication network security and confidentiality assisted by the intelligent reflective surface to obtain the UAV communication network security and confidentiality parameters;

[0008] Set the state vector, action space and reward function of the deep reinforcement learning algorithm, and build a deep reinforcement learning optimization algorithm;

[0009] Based on the deep reinforcement learning optimization algorithm, the UAV communication network security and confidentiality parameters are iteratively optimized to obtain the optimal UAV NOMA confidential communication energy efficiency.

[0010] In some embodiments, the construction of a smart reflective surface-assisted NOMA UAV communication network security and confidentiality energy efficiency optimization model includes:

[0011] The smart reflective surface assisted NOMA UAV communication network security confidentiality energy efficiency optimization model is defined to include several legitimate users, eavesdroppers, smart reflective surfaces and UAVs;

[0012] Based on a three-dimensional Cartesian coordinate system, determining the positions of the legal users, the eavesdropper, the smart reflective surface, and the drone;

[0013] Define a channel matrix of the smart reflecting surface and a channel matrix of the drone, and determine a channel gain of the drone communication network, wherein the channel gain of the drone communication network includes a channel gain between the drone and the legitimate user, a channel gain between the drone and the eavesdropper, a channel gain between the smart reflecting surface and the legitimate user, and a channel gain between the smart reflecting surface and the eavesdropper;

[0014] Determine the decoding order of the legal users and the signal-to-noise ratio when the lth user decodes the kth user signal according to the continuous interference cancellation method;

[0015] According to the decoding order and the signal interference-noise ratio, the achievable rate of the legitimate user and the achievable eavesdropping rate of the eavesdropper are determined to obtain the confidentiality rate from the drone to the kth user;

[0016] Determine the achievable confidentiality energy efficiency between the drone and the legitimate user according to the propulsion energy of the drone and the flight speed of the drone in the horizontal direction;

[0017] The location of the legitimate user, the location of the eavesdropper, the location of the smart reflective surface, the location of the drone, the channel gain of the drone communication network, the confidentiality rate and the achievable confidentiality energy efficiency are integrated to construct a secure confidentiality energy efficiency optimization model for the smart reflective surface-assisted NOMA drone communication network.

[0018] In some embodiments, the expression of the smart reflective surface assisted NOMA UAV communication network security and confidentiality energy efficiency optimization model is specifically as follows:

[0019]

[0020] In the above formulas, formula (a) and formula (c) represent the movement constraints of the drone, formula (b) represents the maximum flight distance of the drone, formula (d) represents that the continuous interference cancellation method is successfully executed, formula (e) represents the phase shift constraint of the smart reflective surface, formula (f) represents the total power constraint, N represents the flight time slot of the drone, n represents the nth time slot, and w k represents the coordinates of the legitimate user, SEE[n] represents the achievable confidentiality energy efficiency, x[n], y[n], h[n] represent the coordinate system values ​​of the three-dimensional space coordinates of the drone, q[n] represents the three-dimensional space coordinates of the drone, q[n-1] represents the coordinates of the drone in the previous time slot, represents the decoding order of the legitimate users, Q represents the motion trajectory of the drone, B represents the motion boundary of the drone, and R j→k represents the rate of decoding the signal of user k at the jth user, R k→k represents the rate at which the kth user decodes its own signal, k and j represent the kth user and the jth user respectively, θ m represents the phase shift of the mth element of RIS, m represents the mth element of RIS, M represents the number of RIS elements, P max Indicates the maximum transmit power.

[0021] In some embodiments, the drone communication network security and confidentiality parameters include drone transmission power, minimum rate threshold for legal users, drone flight cycle size, and drone flight time slot length.

[0022] In some embodiments, the setting of the state vector, action space, and reward function of the deep reinforcement learning algorithm to construct the deep reinforcement learning optimization algorithm includes:

[0023] Determining a state vector of the deep reinforcement learning algorithm according to the position of the drone, the channel gain between the drone and the legitimate user, and the channel gain between the drone and the eavesdropper;

[0024] Determining an action space of the deep reinforcement learning algorithm according to the passive beamforming matrix of the smart reflective surface, the active beamforming matrix of the smart reflective surface, and the movement action of the drone;

[0025] Determining a reward function of the deep reinforcement learning algorithm according to a confidentiality rate between the drone and the kth user and an achievable confidentiality energy efficiency between the drone and the legitimate user;

[0026] The deep reinforcement learning optimization algorithm is constructed by combining the state vector, the action space and the reward function.

[0027] In some embodiments, the iterative optimization of the UAV communication network security and confidentiality parameters based on the deep reinforcement learning optimization algorithm to obtain the optimal UAV NOMA confidential communication energy efficiency includes:

[0028] Preprocessing the deep reinforcement learning optimization algorithm to obtain a preprocessed deep reinforcement learning optimization algorithm, wherein the preprocessed deep reinforcement learning optimization algorithm includes an actor target network, an actor training network, a critic target network, and a critic training network;

[0029] Constructing an experience replay pool, wherein the experience replay pool includes the state of the actor training network, the action of the actor training network, the reward obtained after the actor training network takes the action, and the state of the actor training network at the next moment;

[0030] According to the pre-processed deep reinforcement learning optimization algorithm and the experience replay pool, the UAV communication network security and confidentiality parameters are iteratively optimized to obtain the optimal UAV NOMA confidential communication energy efficiency.

[0031] In some embodiments, preprocessing the deep reinforcement learning optimization algorithm to obtain a preprocessed deep reinforcement learning optimization algorithm includes:

[0032] Introducing complex number processing into the state vector of the deep reinforcement learning optimization algorithm to obtain a state vector containing complex numbers;

[0033] According to the total power constraint, adjusting the action space of the deep reinforcement learning optimization algorithm to obtain an adjusted action space;

[0034] Adding a penalty term to the reward function of the deep reinforcement learning optimization algorithm to obtain an added reward function;

[0035] The state vector containing the complex number, the adjusted action space and the added reward function are integrated to obtain the preprocessed deep reinforcement learning optimization algorithm.

[0036] In some embodiments, the iterative optimization of the UAV communication network security parameters according to the pre-processed deep reinforcement learning optimization algorithm and the experience replay pool to obtain the optimal UAV NOMA secure communication energy efficiency includes:

[0037] Based on the preprocessed deep reinforcement learning optimization algorithm, obtaining the channel gain of the UAV communication network and the current position of the UAV;

[0038] Based on the channel gain of the UAV communication network and the current position of the UAV, the actor training network takes corresponding actions according to the state, obtains the current reward of the actor training network and the next state of the actor training network, and puts them into the experience replay pool;

[0039] Substitute the current reward of the actor training network and the next state of the actor training network into the critic training network to obtain the target Q value of the critic training network;

[0040] Based on the target Q value, the critic training network and the actor training network are updated by stochastic gradient descent method to obtain the gradient of the critic training network and the gradient of the actor training network;

[0041] According to the gradient of the critic training network and the gradient of the actor training network, the actor target network and the critic target network are updated to obtain an updated deep reinforcement learning optimization algorithm;

[0042] Based on the updated deep reinforcement learning optimization algorithm, the UAV communication network security and confidentiality parameters are iteratively optimized until the preset iteration conditions are met, and the optimal UAV NOMA confidential communication energy efficiency is output.

[0043] In some embodiments, the expression of the stochastic gradient descent method is specifically as follows:

[0044]

[0045] In the above formula, N B represents the size of a small batch of experiences, γ μ represents the actor network parameters, θ q represents the evaluator network parameters, represents the stochastic gradient descent method, s j represents the jth state, represents the gradient of the comment network for action a, represents the policy network parameter gradient.

[0046] To achieve the above purpose, another aspect of the embodiment of the present application proposes a UAV communication confidentiality energy efficiency optimization system based on deep reinforcement learning, the system comprising:

[0047] The first module is used to build a network security and energy efficiency optimization model for NOMA drone communications assisted by intelligent reflective surfaces;

[0048] The second module is used to iteratively optimize and solve the energy efficiency optimization model of NOMA UAV communication network security and confidentiality assisted by the intelligent reflective surface to obtain the UAV communication network security and confidentiality parameters;

[0049] The third module is used to set the state vector, action space and reward function of the deep reinforcement learning algorithm and build a deep reinforcement learning optimization algorithm;

[0050] The fourth module is used to iteratively optimize the UAV communication network security and confidentiality parameters based on the deep reinforcement learning optimization algorithm to obtain the optimal UAV NOMA confidential communication energy efficiency.

[0051] The embodiments of the present application include at least the following beneficial effects: The present application provides a method and system for optimizing the confidentiality energy efficiency of drone communications based on deep reinforcement learning. The scheme constructs a smart reflective surface-assisted NOMA drone communication network security and confidentiality energy efficiency optimization model to obtain drone communication network security and confidentiality parameters, and then iteratively optimizes the drone communication network security and confidentiality parameters through a deep reinforcement learning optimization algorithm, converting the optimization problem into a problem based on a Markov decision process (MDP), and improving the confidentiality energy efficiency in the drone communication system by jointly optimizing the active and RIS passive beamforming matrices of the drone aerial base station and the flight trajectory of the drone. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of a method for optimizing the energy efficiency of drone communication confidentiality based on deep reinforcement learning provided in an embodiment of the present application;

[0053] Figure 2 It is a structural schematic diagram of a UAV communication confidentiality energy efficiency optimization system based on deep reinforcement learning provided in an embodiment of the present application;

[0054] Figure 3 It is a structural schematic diagram of the IRS-assisted UAV communication system model provided in an embodiment of the present application;

[0055] Figure 4 is a schematic diagram of the flight trajectory of a UAV under the DDPG algorithm provided in an embodiment of the present application;

[0056] Figure 5 This is a schematic diagram of the periodic rewards obtained through training under the DDPG algorithm provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0058] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0059] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0062] 1) Non-orthogonal multiple access (NOMA) technology effectively improves spectrum utilization and network capacity by allowing multiple users to share signals on the same frequency band. The flexibility and height advantages of drones enable them to be flexibly deployed in different geographical locations, providing ground users with wider and more stable coverage. The combination of drones and NOMA technology not only improves the coverage and service quality of the network, but also provides strong support for the realization of efficient and flexible modern communication networks. In the field of secure communications, NOMA technology enhances the signal's anti-interference ability and reduces the risk of eavesdropping by optimizing power allocation and user pairing.

[0063] 2) Reconfigurable Intelligent Surfaces (RIS) is a breakthrough in wireless communication technology in recent years. This technology optimizes signal transmission by embedding adjustable surfaces in the signal propagation path to intelligently reflect, refract or scatter the signal. The notable features of RIS are its economy, high energy efficiency and ease of deployment. Since RIS is only responsible for adjusting the signal and does not generate the signal itself, it eliminates the need for complex RF links and high-energy amplifiers. In addition, by finely controlling the reflection phase, RIS can enhance the signal strength of the target receiver while weakening the signal of unauthorized receivers, effectively preventing information leakage and providing strong support for the security of drone network communications.

[0064] There are some deficiencies in the relevant technologies. For example, related studies use traditional convex optimization methods to deal with the problem of drone confidentiality energy efficiency, which requires complex formula derivation and iteration to solve the problem. In addition, in the RIS-assisted drone communication system, most of the research only focuses on solving the problem of maximizing the confidentiality rate, while ignoring the energy consumption of the drone. Therefore, there is a lack of research on the confidentiality energy efficiency of the RIS-assisted drone and user communication system.

[0065] In view of this, the embodiment of the present application provides a method for optimizing the energy efficiency of drone communication security based on deep reinforcement learning. In the presence of eavesdroppers, the drone serves multiple ground users at the same time. The energy efficiency of security in the communication system is improved by jointly optimizing the active and RIS passive beamforming of the drone air base station and the flight trajectory of the drone.

[0066] Reference Figure 1 , Figure 1 A flowchart of a method for optimizing the energy efficiency of drone communication security based on deep reinforcement learning provided by an embodiment of the present invention, referring to Figure 1 , the method comprises the following steps:

[0067] S100, build a smart reflective surface-assisted NOMA UAV communication network security and confidentiality energy efficiency optimization model;

[0068] It should be noted that, in some embodiments, step S100 may include: S110, defining a smart reflective surface assisted NOMA drone communication network security and confidentiality energy efficiency optimization model including a number of legitimate users, eavesdroppers, smart reflective surfaces and drones; S120, determining the positions of a number of legitimate users, the positions of eavesdroppers, the positions of smart reflective surfaces and the positions of drones based on a three-dimensional Cartesian coordinate system;

[0069] In steps S110 to S120 of some embodiments, Figure 3As shown in Figure 1, a multi-user wireless communication system for UAVs assisted by RIS is considered. There are K legitimate users and one eavesdropper e in the system. Each legitimate user and eavesdropper is equipped with a single antenna. The UAV is equipped with an N-element uniform linear array (ULA), and RIS is configured with M=m 2 The set of legal users is represented by a uniform planar array (UPA) of passive reflective elements. Assume that in the three-dimensional Cartesian coordinate system, the position coordinates of the legitimate user k and the eavesdropper are RIS fixed in position w R =(x R ,y R ,z R ) T The drone flies at a fixed altitude within a limited period of time, which is divided into N time slots, where T = Nδ n , and δ n represents the duration of each time slot. Therefore, in the nth time slot, the coordinates of the drone are expressed as q[n] = (x U [n],y U [n],h U )express.

[0070] S130, defining a channel matrix of the smart reflecting surface and a channel matrix of the drone, and determining a channel gain of the drone communication network, wherein the channel gain of the drone communication network includes a channel gain between the drone and a legitimate user, a channel gain between the drone and an eavesdropper, a channel gain between the smart reflecting surface and a legitimate user, and a channel gain between the smart reflecting surface and an eavesdropper;

[0071] In step S130 of some embodiments, the channel gains from the UAV (unmanned aerial vehicle) and the RIS to the kth user are respectively and Similarly, and They represent the channel gains from UAV and RIS to the eavesdropper respectively. The channel matrix between UAV and RIS is given by Therefore, the received signals of the kth user and the eavesdropper can be expressed as:

[0072]

[0073]

[0074] where Ψ=vec(Θ), is the reflection matrix of RIS, θ m ∈[0,2π), UAV transmits signal in is the beamforming vector at the UAV, sk represents the transmitted signal of the kth user. Assume that all channels consist of large-scale path loss and small-scale Rician fading, expressed as follows:

[0075]

[0076]

[0077] Where γ 1 , γ 2i , γ 3i is the path loss when the node reference distance is 1m, a 1 、a 2i and a 3i is the corresponding path loss index. UR d Ui d Ri are the distances between corresponding nodes, ε 2i , ε 3i represents the Ricean factor of the channel.

[0078] S140, according to the continuous interference cancellation method, determine the decoding order of the legitimate users and the signal interference-noise ratio when the lth user decodes the kth user signal; S150, according to the decoding order and the signal interference-noise ratio, determine the achievable rate of the legitimate users and the achievable eavesdropping rate of the eavesdropper, and obtain the confidentiality rate from the drone to the kth user;

[0079] In steps S140 to S150 of some embodiments, based on the NOMA principle, each legitimate user uses the successive interference cancellation (SIC) technique to reduce the inter-user interference to detect the superimposed signal. In the embodiment of the present invention, we arrange the decoding order according to the channel strength. Assume that the channel strength of user n in time slot satisfies Then the decoding order of all K users The signal-to-interference-to-noise ratio (SINR) when the lth user decodes the kth user signal can be defined as:

[0080]

[0081] Therefore, the achievable rate for user k is expressed as:

[0082]

[0083] The SINR at the eavesdropper e is:

[0084]

[0085] The corresponding achievable eavesdropping rate is R e→k [n] = log 2(1+SINR e→k [n]).

[0086] The confidentiality rate from the drone to the kth user can be defined as:

[0087]

[0088] Where [j] + =max(0,j).

[0089] S160, determining the achievable confidentiality efficiency between the drone and the legal user based on the propulsion energy of the drone and the flight speed of the drone in the horizontal direction;

[0090] In step S160 of some embodiments, the energy consumption of the drone mainly comes from two aspects: first, the energy consumed by the propulsion system, and second, the energy consumption of the communication module. In view of the fact that in most practical applications, the communication energy consumption is lower than the propulsion energy consumption, the embodiment of the present invention does not consider the communication energy consumption for the sake of simplifying the model. The propulsion energy of the drone is a function related to the flight speed ‖v[n]‖, and its expression is:

[0091]

[0092] The constant P i and P 0 They represent the induced power and blade profile power of the UAV in the hovering state, is the tip speed of the rotor blade, v 0 represents the average rotor induced speed during hovering. In addition, d 0 is the fuselage drag ratio, s is the rotor solidity. Finally, ρ is the air density, A r is the rotor disk area.

[0093] The horizontal flight speed of the UAV ‖v[n]‖ is expressed as follows:

[0094]

[0095] Then, according to the definition expression of the confidentiality rate from the UAV to the kth user and the expression of the propulsion energy of the UAV, the achievable confidentiality energy efficiency between the UAV and the legitimate user can be calculated as:

[0096]

[0097] S170. Integrate the location of legitimate users, the location of eavesdroppers, the location of smart reflective surfaces, the location of drones, the channel gain of drone communication networks, confidentiality rate, and achievable confidentiality energy efficiency, and build a secure confidentiality energy efficiency optimization model for NOMA drone communication networks assisted by smart reflective surfaces.

[0098] In step S170 of some embodiments, our goal is to jointly optimize the BS active beamforming and RIS reflection coefficients, as well as the UAV trajectory. Maximize the achievable confidentiality energy efficiency. Therefore, the optimization problem, that is, the intelligent reflective surface assisted NOMA UAV communication network confidentiality energy efficiency optimization model, is expressed as follows:

[0099]

[0100] In the above formulas, formula (a) and formula (c) represent the movement constraints of the drone, formula (b) represents the maximum flight distance of the drone, formula (d) represents that the continuous interference cancellation method is successfully executed, formula (e) represents the phase shift constraint of the smart reflective surface, formula (f) represents the total power constraint, N represents the flight time slot of the drone, n represents the nth time slot, and w k represents the coordinates of the legitimate user, SEE[n] represents the achievable confidentiality energy efficiency, x[n], y[n], h[n] represent the coordinate system values ​​of the three-dimensional space coordinates of the drone, q[n] represents the three-dimensional space coordinates of the drone, q[n-1] represents the coordinates of the drone in the previous time slot, K represents the decoding order of the legitimate user, Q represents the drone motion trajectory, B represents the drone motion boundary, R j→k represents the rate of decoding the signal of user k at the jth user, R k→k represents the rate at which the kth user decodes its own signal, k and j represent the kth user and the jth user respectively, θ m represents the phase shift of the mth element of RIS, m represents the mth element of RIS, M represents the number of RIS elements, P max Indicates the maximum transmit power.

[0101] S200, iteratively optimize and solve the energy efficiency optimization model of NOMA UAV communication network security and confidentiality assisted by intelligent reflective surface, and obtain the UAV communication network security and confidentiality parameters;

[0102] It should be noted that, in some embodiments, the UAV communication network security and confidentiality parameters in step S200 include the UAV transmission power, the minimum rate threshold of legitimate users, the UAV flight cycle size and the UAV flight time slot length.

[0103] S300, setting the state vector, action space and reward function of the deep reinforcement learning algorithm, and constructing a deep reinforcement learning optimization algorithm;

[0104] It should be noted that, in some embodiments, step S300 may include: S310, determining the state vector of the deep reinforcement learning algorithm according to the position of the drone, the channel gain between the drone and the legitimate user, and the channel gain between the drone and the eavesdropper; S320, determining the action space of the deep reinforcement learning algorithm according to the passive beamforming matrix of the intelligent reflection surface, the active beamforming matrix of the intelligent reflection surface and the movement of the drone; S330, determining the reward function of the deep reinforcement learning algorithm according to the confidentiality rate of the drone to the kth user and the achievable confidentiality energy efficiency between the drone and the legitimate user; S340, constructing a deep reinforcement learning optimization algorithm by combining the state vector, action space and reward function.

[0105] In some specific embodiments, since the intelligent reflective surface assisted NOMA UAV communication network security and confidentiality energy efficiency optimization model has a fractional form and the variables are tightly coupled, this makes the problem highly non-convex and difficult to solve using traditional methods. Therefore, we propose a DRL-based solution. Considering the continuity of the optimization variables, in the embodiment of the present invention, we apply the DDPG algorithm in DRL that is suitable for processing continuous action spaces.

[0106] Based on the DDPG drone trajectory, beamforming and phase shift optimization algorithm, the embodiment of the present invention transforms the optimization problem into an MDP-based problem so that it can be fully applied to the reinforcement learning environment. The process of applying the reinforcement learning algorithm to solve the optimization problem is roughly as follows: The result is as follows Figure 4 As shown, first, the state space and action space of the optimization problem are clearly defined, and a reward function is designed to evaluate the effect of the actions taken. Next, the parameters of the reinforcement learning model are initialized. At each decision step, the algorithm selects an action from the policy network based on the current state of the environment and applies it to the environment. The environment then provides feedback, including the immediate reward obtained by executing the action and the new state of the environment. The algorithm uses this feedback information to evaluate the performance of the current strategy and updates the policy network accordingly, with the aim of learning how to select actions in different states to maximize the cumulative reward. In order to effectively solve the smart reflective surface-assisted NOMA drone communication network security confidentiality energy efficiency optimization model, we define the state, action vector and reward function as follows:

[0107] 1) Observation state: The designed state vector includes the specific location information of the drone in time slot n, as well as the comprehensive channel state information between the drone and all legitimate users and potential eavesdroppers. This design can provide a comprehensive situational awareness for the TD3 algorithm, so that it can make more accurate decisions in the optimization problem of the continuous action space, which is expressed as:

[0108]

[0109] 2) Action space: The passive beamforming matrix Θ and the active beamforming matrix W as well as the movement of the drone are taken as actions. The movement of the drone consists of two parts: one is the distance it moves Δd[n], and the other is the horizontal azimuth angle it moves in the plane. Its expression is:

[0110]

[0111] 3) Reward function: Our optimization goal is to maximize the sum of the confidentiality energy efficiency of the system within time T. In order to quickly and effectively guide the agent to choose the appropriate action, we use the ratio of the user confidentiality rate to the flight energy of the drone in each time slot SEE as a reward, which is expressed as:

[0112] R[m]=SEE[n]

[0113] S400, based on the deep reinforcement learning optimization algorithm, iteratively optimizes the network security parameters of drone communication to obtain the optimal drone NOMA secure communication energy efficiency;

[0114] It should be noted that, in some embodiments, step S400 may include: S410, preprocessing the deep reinforcement learning optimization algorithm to obtain a preprocessed deep reinforcement learning optimization algorithm, wherein the preprocessed deep reinforcement learning optimization algorithm includes an actor target network, an actor training network, a critic target network, and a critic training network;

[0115] In some embodiments, step S410 may include: S411, introducing complex number processing to the state vector of the deep reinforcement learning optimization algorithm to obtain a state vector containing complex numbers; S412, adjusting the action space of the deep reinforcement learning optimization algorithm according to the total power constraint to obtain an adjusted action space; S413, adding a penalty term to the reward function of the deep reinforcement learning optimization algorithm to obtain an added reward function; S414, integrating the state vector containing complex numbers, the adjusted action space and the added reward function to obtain a preprocessed deep reinforcement learning optimization algorithm.

[0116] In a specific embodiment of the present invention, neural networks are generally unable to directly process complex numbers as input, mainly because their infrastructure, activation functions, optimization algorithms, and loss functions are all designed for real numbers. The introduction of complex numbers not only increases the dimension of the parameters, but also brings numerical stability issues and computational complexity. Therefore, in order to handle the real input problem, we split H into Re{H}+Im{H}, and similarly, let W=Re{W}+Im{W} to better adapt to the neural network. In addition, due to the limitation of the total power constraint, we readjust the beamforming matrix obtained by the strategy to obtain a new beamforming matrix in In the achievable rate of user k, due to the existence of min(·), the optimization variable can satisfy the constraint that the continuous interference cancellation method is successfully executed. In order to make the variable satisfy other constraints as much as possible, we add a penalty term to the reward function, so the new reward function is expressed as follows:

[0117]

[0118] ξ=SEE[n]-c 1 p 1 -c 2 p 2

[0119] where p 1 、p 2 They are the penalties for not meeting the maximum flight distance constraint of the drone and the movement constraint of the drone. i , i∈{1,2} is the coefficient that balances the penalty and SEE. ξ is similar to the tanh function graph, but it is smoother and the output range is also between [-1,1]. It can effectively limit the size of the reward and prevent excessive or small reward values ​​from having an adverse effect on training. Therefore, using it as a reward function can better guide the learning of the agent and improve the stability and efficiency of training.

[0120] S420, constructing an experience replay pool, the experience replay pool including the state of the actor training network, the action of the actor training network, the reward obtained after the actor training network takes the action, and the state of the actor training network at the next moment;

[0121] S430. Based on the pre-processed deep reinforcement learning optimization algorithm and the experience replay pool, the UAV communication network security and confidentiality parameters are iteratively optimized to obtain the optimal UAV NOMA confidential communication energy efficiency.

[0122] In some embodiments, step S430 may include: S431, based on the pre-processed deep reinforcement learning optimization algorithm, obtaining the channel gain of the drone communication network and the current position of the drone; S432, based on the channel gain of the drone communication network and the current position of the drone, the actor training network takes corresponding actions according to the state, obtains the current reward of the actor training network and the next state of the actor training network, and puts them into the experience replay pool; S433, substitutes the current reward of the actor training network and the next state of the actor training network into the critic training network to obtain the target state of the critic training network. Mark Q value; S434. Based on the target Q value, the critic training network and the actor training network are updated by the stochastic gradient descent method to obtain the gradient of the critic training network and the gradient of the actor training network; S435. According to the gradient of the critic training network and the gradient of the actor training network, the actor target network and the critic target network are updated to obtain the updated deep reinforcement learning optimization algorithm; S436. Based on the updated deep reinforcement learning optimization algorithm, the UAV communication network security and confidentiality parameters are iteratively optimized until the preset iteration conditions are met, and the optimal UAV NOMA confidential communication energy efficiency is output.

[0123] In some specific embodiments, Figure 5 As shown in the figure, in DDPG, the deterministic policy (DPG) network is used as the actor to select actions from the continuous action space, that is, a = μ(s,θ μ ), where θ μ are the parameters of DPG. The performance of the selected action is calculated using the Q network Q(s,a;θ q ), where θ q is the parameter of the Q network. The goal of DDPG is to increase the output Q value as much as possible. To achieve this goal, we use an experience replay D, just like in DQN, to reduce the correlation between different training samples. Furthermore, to address the problem of divergence in Q value updates of the single Q network, μ′(s,θ μ′ ) and Q′(s,a;θ q′ ) is called the target network, which is used to calculate the corresponding target value. They are copies of the actor and comment networks. Although the target network and its associated training network have the same structure, the parameters are different, namely θ q ≠θ q′ and θ μ ≠θ μ′ .

[0124] The agent creates a state s at each time step n given the channel information and the drone’s position and action in the previous state n The parameters of the evaluator network and the actor network are θ q and θ μ, actions W, Θ, Δx, Δy, and the experience replay buffer must all be initialized at the beginning of the algorithm. In this paper, we only use the identity matrix to initialize W and Θ.

[0125] In the initialization phase, four networks are established: the actor-target network θ μ′ , actor training network θ μ , critic target network θ q′ and critics train the network θ q , the parameters are uniformly distributed. At the beginning of each cycle, the phase shifts of all elements are randomly selected from 0 to 2π without loss of generality. We first calculate the channels involved in each cycle and the current position of the drone. The actor training network is based on the state s n Produce corresponding action a n =μ(s,θ μ ). Calculate the current reward r n , and then get the next state s n+1 . Will (s n ,a n ,r n ,s n+1 ) into D as a transition. Then the critic trains the network by sampling an N B Calculate the target Q value y from a small batch of experience of size i , whose expression is:

[0126]

[0127] Where γ is the discount factor. The loss function of the critic training network is as follows:

[0128]

[0129] Then, using stochastic gradient descent (SGD), the critic evaluation network can be updated by decreasing the agent’s policy gradient to update the actor network, which is written as:

[0130]

[0131] The updates for training the critic network and training the actor network are as follows:

[0132]

[0133] where μ c and μ a is the learning rate of the optimizer used to train the critic network and the actor network. The gradient of the target critic network with respect to the action is in represents the gradients for training the actor network. Finally, here are the updates for the target critic and target actor networks:

[0134] θ q′ ←τ c θ q +(1-τ c )θ q′

[0135] θ μ′ ←τ a θ μ +(1-τ a )θ μ′

[0136] In the above formula, θ q′ represents the updated target critic network, θ μ′ represents the updated target actor network.

[0137] In summary, the implementation steps of the embodiments of the present invention are summarized as follows:

[0138] 1) Consider establishing a smart reflective surface-assisted NOMA UAV communication network security and confidentiality energy efficiency maximization model under the condition of the presence of eavesdroppers;

[0139] 2) Iterative optimization solves the security and confidentiality energy efficiency maximization model, initializes the starting position of the UAV, generates the positions of ground users, eavesdroppers, and intelligent reflective surfaces, and determines the UAV transmission power, user minimum rate threshold, flight cycle size, and time slot length;

[0140] 3) Set the state, action space and reward function in the deep reinforcement learning algorithm. The designed state vector includes the specific location information of the drone in time slot n, as well as the comprehensive channel state information between the drone and all legitimate users and potential eavesdroppers. The RIS passive beamforming matrix and active beamforming matrix and the movement of the drone are taken as actions. Finally, the ratio of the user confidentiality rate of each time slot to the flight energy of the drone, the confidentiality energy efficiency (SEE), is used as a reward;

[0141] 4) At each moment, by inputting the state vector into the neural network, the policy network outputs the action selected in the time slot, calculates the corresponding reward, and enters the next state. The state vector and action vector are input into the comment network together to obtain the action value function of the action. Finally, the drone's location information, cycle reward, and confidential energy efficiency are saved in the corresponding file;

[0142] 5) Store the obtained <state, action, reward, next state> tuple into the experience replay pool. When the training conditions are met, randomly extract a small batch of experience from the experience replay pool to train the network. Update the policy network and comment network parameters through training;

[0143] 6) Determine whether the algorithm has converged or reached the maximum training cycle. If the algorithm has converged or reached the maximum training cycle, the training is terminated. If the termination condition has been met, the position of the drone at each moment recorded in 4) is plotted as a trajectory. If the algorithm has not terminated, go to 4) and 5) until convergence.

[0144] See also Figure 2 The embodiment of the present application also provides a UAV communication confidentiality energy efficiency optimization system based on deep reinforcement learning, which can implement the above-mentioned UAV communication confidentiality energy efficiency optimization method based on deep reinforcement learning. The system includes:

[0145] The first module 201 is used to build a network security and confidentiality energy efficiency optimization model for NOMA UAV communication assisted by intelligent reflective surface;

[0146] The second module 202 is used to iteratively optimize and solve the smart reflective surface assisted NOMA UAV communication network security and confidentiality energy efficiency optimization model to obtain the UAV communication network security and confidentiality parameters;

[0147] The third module 203 is used to set the state vector, action space and reward function of the deep reinforcement learning algorithm and construct a deep reinforcement learning optimization algorithm;

[0148] The fourth module 204 is used to iteratively optimize the security and confidentiality parameters of the drone communication network based on the deep reinforcement learning optimization algorithm to obtain the optimal drone NOMA confidential communication energy efficiency.

[0149] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0150] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for optimizing the energy efficiency of drone communication security based on deep reinforcement learning, characterized in that: The method comprises the following steps: Construct a security, confidentiality and energy efficiency optimization model for NOMA UAV communication network assisted by intelligent reflective surfaces; Iteratively optimize and solve the energy efficiency optimization model for the NOMA UAV communication network security and confidentiality assisted by the intelligent reflective surface to obtain the UAV communication network security and confidentiality parameters; Set the state vector, action space and reward function of the deep reinforcement learning algorithm, and build a deep reinforcement learning optimization algorithm; Based on the deep reinforcement learning optimization algorithm, the UAV communication network security and confidentiality parameters are iteratively optimized to obtain the optimal UAV NOMA confidential communication energy efficiency.

2. The method according to claim 1, characterized in that The construction of the intelligent reflective surface-assisted NOMA UAV communication network security and confidentiality energy efficiency optimization model includes: The smart reflective surface assisted NOMA UAV communication network security confidentiality energy efficiency optimization model is defined to include several legitimate users, eavesdroppers, smart reflective surfaces and UAVs; Based on a three-dimensional Cartesian coordinate system, determining the positions of the legal users, the eavesdropper, the smart reflective surface, and the drone; Define a channel matrix of the smart reflecting surface and a channel matrix of the drone, and determine a channel gain of the drone communication network, wherein the channel gain of the drone communication network includes a channel gain between the drone and the legitimate user, a channel gain between the drone and the eavesdropper, a channel gain between the smart reflecting surface and the legitimate user, and a channel gain between the smart reflecting surface and the eavesdropper; Determine the decoding order of the legal users and the signal-to-noise ratio when the lth user decodes the kth user signal according to the continuous interference cancellation method; According to the decoding order and the signal interference-noise ratio, the achievable rate of the legitimate user and the achievable eavesdropping rate of the eavesdropper are determined to obtain the confidentiality rate from the drone to the kth user; Determine the achievable confidentiality energy efficiency between the drone and the legitimate user according to the propulsion energy of the drone and the flight speed of the drone in the horizontal direction; The location of the legitimate user, the location of the eavesdropper, the location of the smart reflective surface, the location of the drone, the channel gain of the drone communication network, the confidentiality rate and the achievable confidentiality energy efficiency are integrated to construct a secure confidentiality energy efficiency optimization model for the smart reflective surface-assisted NOMA drone communication network.

3. The method according to claim 1, characterized in that The expression of the intelligent reflective surface assisted NOMA UAV communication network security and confidentiality energy efficiency optimization model is as follows: st|x[n]|,|y[n]|,|h[n]|≤B,#(b) i m ∈[0,2π),m={1,2…,M}#(e) In the above formulas, formula (a) and formula (c) represent the movement constraints of the drone, formula (b) represents the maximum flight distance of the drone, formula (d) represents that the continuous interference cancellation method is successfully executed, formula (e) represents the phase shift constraint of the smart reflective surface, formula (f) represents the total power constraint, N represents the flight time slot of the drone, n represents the nth time slot, and w k represents the coordinates of the legitimate user, SEE[n] represents the achievable confidentiality energy efficiency, x[n], y[n], h[n] represent the coordinate system values ​​of the three-dimensional space coordinates of the drone, q[n] represents the three-dimensional space coordinates of the drone, q[n-1] represents the coordinates of the drone in the previous time slot, represents the decoding order of the legitimate users, Q represents the trajectory of the drone, B represents the boundary of the drone, and R j→k represents the rate of decoding the signal of user k at the jth user, R k→k represents the rate at which the kth user decodes its own signal, k and j represent the kth user and the jth user respectively, θ m represents the phase shift of the mth element of RIS, m represents the mth element of RIS, M represents the number of RIS elements, P max Indicates the maximum transmit power.

4. The method according to claim 1, characterized in that: The UAV communication network security and confidentiality parameters include UAV transmission power, minimum rate threshold of legal users, UAV flight cycle size and UAV flight time slot length.

5. The method according to claim 2, characterized in that: The setting of the state vector, action space and reward function of the deep reinforcement learning algorithm and the construction of the deep reinforcement learning optimization algorithm include: Determining a state vector of the deep reinforcement learning algorithm according to the position of the drone, the channel gain between the drone and the legitimate user, and the channel gain between the drone and the eavesdropper; Determining an action space of the deep reinforcement learning algorithm according to the passive beamforming matrix of the smart reflective surface, the active beamforming matrix of the smart reflective surface, and the movement action of the drone; Determining a reward function of the deep reinforcement learning algorithm according to a confidentiality rate between the drone and the kth user and an achievable confidentiality energy efficiency between the drone and the legitimate user; The deep reinforcement learning optimization algorithm is constructed by combining the state vector, the action space and the reward function.

6. The method according to claim 5, characterized in that The iterative optimization of the UAV communication network security and confidentiality parameters based on the deep reinforcement learning optimization algorithm to obtain the optimal UAV NOMA confidential communication energy efficiency includes: Preprocessing the deep reinforcement learning optimization algorithm to obtain a preprocessed deep reinforcement learning optimization algorithm, wherein the preprocessed deep reinforcement learning optimization algorithm includes an actor target network, an actor training network, a critic target network, and a critic training network; Constructing an experience replay pool, wherein the experience replay pool includes the state of the actor training network, the action of the actor training network, the reward obtained after the actor training network takes the action, and the state of the actor training network at the next moment; According to the pre-processed deep reinforcement learning optimization algorithm and the experience replay pool, the UAV communication network security and confidentiality parameters are iteratively optimized to obtain the optimal UAV NOMA confidential communication energy efficiency.

7. The method according to claim 6, characterized in that The preprocessing of the deep reinforcement learning optimization algorithm to obtain the preprocessed deep reinforcement learning optimization algorithm includes: Introducing complex number processing into the state vector of the deep reinforcement learning optimization algorithm to obtain a state vector containing complex numbers; According to the total power constraint, adjusting the action space of the deep reinforcement learning optimization algorithm to obtain an adjusted action space; Adding a penalty term to the reward function of the deep reinforcement learning optimization algorithm to obtain an added reward function; The state vector containing the complex number, the adjusted action space and the added reward function are integrated to obtain the preprocessed deep reinforcement learning optimization algorithm.

8. The method according to claim 6, characterized in that The method of iteratively optimizing the UAV communication network security and confidentiality parameters according to the pre-processed deep reinforcement learning optimization algorithm and the experience replay pool to obtain the optimal UAV NOMA confidential communication energy efficiency includes: Based on the preprocessed deep reinforcement learning optimization algorithm, obtaining the channel gain of the UAV communication network and the current position of the UAV; Based on the channel gain of the UAV communication network and the current position of the UAV, the actor training network takes corresponding actions according to the state, obtains the current reward of the actor training network and the next state of the actor training network, and puts them into the experience replay pool; Substitute the current reward of the actor training network and the next state of the actor training network into the critic training network to obtain the target Q value of the critic training network; Based on the target Q value, the critic training network and the actor training network are updated by stochastic gradient descent method to obtain the gradient of the critic training network and the gradient of the actor training network; According to the gradient of the critic training network and the gradient of the actor training network, the actor target network and the critic target network are updated to obtain an updated deep reinforcement learning optimization algorithm; Based on the updated deep reinforcement learning optimization algorithm, the UAV communication network security and confidentiality parameters are iteratively optimized until the preset iteration conditions are met, and the optimal UAV NOMA confidential communication energy efficiency is output.

9. The method according to claim 8, characterized in that The expression of the stochastic gradient descent method is specifically as follows: In the above formula, N B represents the size of the mini-batch experience, θ μ represents the actor network parameters, θ q represents the evaluator network parameters, represents the stochastic gradient descent method, s j represents the jth state, represents the gradient of the comment network for action a, represents the policy network parameter gradient.

10. UAV communication confidentiality energy efficiency optimization system based on deep reinforcement learning, characterized by: The system comprises: The first module is used to build a network security and energy efficiency optimization model for NOMA drone communications assisted by intelligent reflective surfaces; The second module is used to iteratively optimize and solve the energy efficiency optimization model of NOMA UAV communication network security and confidentiality assisted by the intelligent reflective surface to obtain the UAV communication network security and confidentiality parameters; The third module is used to set the state vector, action space and reward function of the deep reinforcement learning algorithm and build a deep reinforcement learning optimization algorithm; The fourth module is used to iteratively optimize the UAV communication network security and confidentiality parameters based on the deep reinforcement learning optimization algorithm to obtain the optimal UAV NOMA confidential communication energy efficiency.

Images