Hidden secure transmission optimization method, system and device for unmanned aerial vehicle network in ground node interference cooperation scene, and medium

Abstract

A hidden secure transmission optimization method, system and device for an unmanned aerial vehicle network in a ground node interference cooperation scene, and a medium, the method comprising: constructing a system model of an unmanned aerial vehicle line-of-sight link, and deducing a concealment requirement under a binary detection method; constructing a target function for maximizing the average security covert communication throughput from the starting point to the end point under the constraint conditions of covert, security and the like; the optimization problem is converted into a single-agent Markov decision process, and the state, the action, the reward, the transition probability and the discount coefficient are converted; constructing an algorithm based on a double-delay depth deterministic strategy gradient to realize an optimal decision, and further realizing optimization solution of a target function; the system, the device and the medium are applied to a ground node interference cooperation unmanned aerial vehicle scene. According to the method, higher average security hidden throughput is realized in a ground node interference cooperation unmanned aerial vehicle network scene, and meanwhile, the training speed is higher; in addition, the optimal solution is a strategy solution, the power track strategy of the unmanned aerial vehicle can be conveniently set, and the method is more suitable for the dynamic complex environment.

Description

Technical Field

[0008] The present invention belongs to the field of covert security technology in drone networks, and in particular relates to a covert security transmission optimization method, system, equipment and medium for drone networks in a ground node interference cooperation scenario. Background Art

[0009] With the rapid development of integrated air-ground mobile communication technology, drones have been widely used in many fields such as cellular networks, disaster response, and military confrontation. However, due to the line-of-sight communication characteristics between drones and ground users, they are easily detected and eavesdropped by the outside world, which increases the security risks of the communication link. Artificially adding artificial noise to the drone communication process makes it difficult for unauthorized third parties to detect and obtain the transmitted information, which is of great significance to physical layer security and covert communication. Therefore, it is particularly important to design a strategic covert security transmission optimization method in a dynamic environment of drones.

[0010] Currently, many transmission optimization methods have been proposed. Classic optimization methods include: bat algorithm (Yang XS. A new metaheuristic bat-inspired algorithm [M] / / Nature inspired cooperative strategies for optimization (NICSO 2010). Berlin, Heidelberg: Springer Berlin Heidelberg, 2010: 65-74.), fruit fly optimization algorithm (Pan W TA new fruit fly optimization algorithm: taking the financial distress model as an example [J]. Knowledge-Based Systems, 2012, 26: 69-74.), feasibility rule optimization framework algorithm (Wang Y, Cai Z. Combining multiobjective optimization with differential evolution to solve constrained optimization problems [J] IEEE Transactions on Evolutionary Computation, 2012, 16 (1): 117-134.), and continuous approximation algorithm (Hong LJ, Yang Y, Zhang L. Sequential convex approximations to joint chance constrained programs: A Monte Carlo approach [J]. Operations Research,2011,59(3):617-630.).New optimization methods include: whale optimization algorithm (Mirjalili S, Lewis A. The whale optimization algorithm [J]. Advances in engineering software, 2016, 95: 51-67.), constrained optimization algorithm with differential evolution (Trivedi A, Sanyal K, Verma P, et al. A unified differential evolution algorithm for constrained optimization problems [C] / / 2017 IEEE congress on evolutionary computation (CEC). IEEE, 2017: 1231-1238.). Most of the optimization methods mentioned above are static algorithms. However, the actual UAV communication environment is dynamic and complex, and static algorithms are prone to fall into local optimal solutions and have poor adaptability to dynamic environments. As a result, the transmission performance of the transmission optimization scheme is reduced.

[0011] Scholars have proposed some transmission optimization methods in UAV network scenarios. C Zhong et al. proposed a UAV trajectory optimization method based on alternating optimization and continuous convex approximation technology, which improved the optimization performance of UAV secure transmission (Zhong C, Yao J, Xu J. Secure UAV communication with cooperative jamming and trajectory control [J]. IEEE Communications Letters, 2018, 23 (2): 286-289.), M Cui applied block coordinate descent method to UAV trajectory optimization, which effectively improved the average worst-case confidentiality rate of the UAV system (Cui M, Zhang G, Wu Q, et al. Robust trajectory and transmit power design for secure UAV communications [J]. IEEE Transactions on Vehicular Technology, 2018, 67 (9): 9042-9046.), L Shen et al. proposed a UAV trajectory optimization method based on dynamic iterative model to improve the performance of wireless communication system (Shen L, Wang N, Mu X. Iterative UAV trajectory optimization for physical layer secure mobile relaying [C] / / 2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC). IEEE, 2018: 19-194.), X Zhou et al. used alternating iterative algorithm and successive convex approximation technology to solve the UAV power trajectory optimization problem, and achieved better transmission performance than the fixed power scheme (Zhou X, Wu Q, Yan S, et al. UAV-enabled secure communications: Joint trajectory and transmit power optimization [J]. IEEE Transactions on Vehicular Technology, 2019, 68 (4): 4069-4073.).

[0012] The above transmission optimization method only gives a solution, which does not meet the timeliness of decision-making. Especially in the case of dynamic and complex actual drone environment, where there are two threats, monitoring and eavesdropping, the heuristic search optimization algorithm is not suitable for dynamic decision-making. To this end, a deep reinforcement learning algorithm for solving sequence problems can be used to improve the performance of drone security transmission optimization by adapting a covert security transmission optimization strategy based on control theory to dynamic decision-making.

[0013] Through the above analysis, the problems and defects of the prior art are as follows:

[0014] (1) Existing transmission optimization technologies only consider using an exact optimization solution to solve the optimization problem, but the exact optimization solution does not meet the real-time decision-making requirements of the UAV dynamic environment.

[0015] (2) Most existing transmission optimization methods use static algorithms for solution. However, static optimization algorithms have disadvantages such as local optimality problems and poor adaptability to dynamic environments. Summary of the invention

[0016] In order to overcome the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a covert and secure transmission optimization method, system, equipment and medium for UAV networks in a ground node interference cooperation scenario. By constructing an objective function that maximizes the average secure and covert communication throughput from the start point to the end point, and converting the optimization problem into a single-agent Markov decision process, and finally constructing a network based on a double-delay deep deterministic policy gradient to jointly optimize the UAV trajectory power, the covert and secure transmission optimization problem can be effectively solved, and good transmission performance can be achieved under higher concealment constraints.

[0017] In order to achieve the above object, the technical solution adopted by the present invention is:

[0018] A concealed and secure transmission optimization method for a UAV network in a ground node interference cooperation scenario, comprising:

[0019] Step 1: construct a system model of the artificial noise-assisted UAV network of ground nodes and construct the concealment constraints obtained by the binary detection analysis method of the UAV monitor;

[0020] Step 2: Based on the concealment constraint constructed in step 1, an objective function is constructed to maximize the average secure covert communication throughput from the start point to the end point under the constraints of the drone boundary and drone position.

[0021] Step 3: Under the optimization problem of the objective function constructed in step 2, the optimization problem is transformed by using the five-tuple of state observation space, reward setting, action setting, state transition probability setting, and discount rate;

[0022] Step 4: Using the Markov decision process obtained in step 3, a network based on deep delayed deterministic policy gradient is designed to jointly optimize the UAV power trajectory to make the target decision optimal and achieve the target solution.

[0023] Further, the specific process of step one is:

[0024] Based on the artificial noise assisted UAV network of ground nodes, it is assumed that the UAV transmitter Alice in the air sends useful information to the legitimate user Bob on the ground, and there is a monitor Willie, an eavesdropper Eve and a friendly jammer jammer on the ground; the concealment constraint can be expressed as:

[0025] ξ * >1-ò

[0026] Among them, ξ * is the derived Willie's minimum detection error probability, where ò is a decimal close to zero, which can be used to quantitatively describe the concealment performance of the system;

[0027] First, the monitor power binary detection is designed according to the UAV system model:

[0028]

[0029] Among them, T w [n] is the average power of the signal received by the assumed monitor, P a [n] and P j [n] represents the power transmitted by the UAV and the jammer in the nth time slot, |h jw [n]| 2 and |h aw [n]| 2 is the channel gain from the friendly jammer and the drone transmitter Alice to Willie; the background noise at Willie is considered as Gaussian noise and is expressed as

[0030] The probability of a false alarm is P FA =Pr{D1|H0}, the probability of missed detection is P MD =Pr{D0|H1}; by the preset monitoring threshold P th By comparison, the false alarm probability, missed detection probability and detection error probability of Willie are:

[0031]

[0032] ξ=P FA +P MD

[0033] According to the exponential distribution of the channel gain from the jammer to the monitor, we can get:

[0034]

[0035] Among them, τ f ' and τ m ' are the derivatives of their respective functions, respectively

[0036] Taking the derivative of the original function, the optimal detection power threshold is: Substituting the minimum detection error probability, we get:

[0037]

[0038] Further, the specific process of step 2 is:

[0039] First, according to the system model in step 1, the UAV boundary constraints, UAV speed constraints and UAV starting position constraints are set. Then, the mixed interruption probability is used to design the safety interruption constraints and construct the objective function of average safety concealed throughput.

[0040] Considering the interruption constraint, concealment constraint, UAV transmission power constraint, interference power of artificial noise carrier node and UAV flight constraint, the optimization problem of the average safe and concealed throughput of the UAV from the starting point to the end point can be given by the following formula:

[0041]

[0042] C7:q a [0] = q s ,q a [N F ]∈Q des

[0043] Among them, C1 represents the concealment constraint, which must ensure that the transmission is carried out when the probability of detection error by the monitor is greater than a certain value, and C2 is the probability P of transmission interruption. HOP Cannot exceed a certain threshold value P OUT , ensuring that users can receive data safely while preventing eavesdroppers from stealing more confidential information; the power P of the drone transmitter and the ground jammer a [n], P j [n] should meet the requirement of less than less than the maximum power C5 expresses that the moving distance of the drone between any two consecutive time slots, such as n to n+1, is subject to the maximum speed V maxC6 is the flight area D setting of the drone, which is used to ensure that the mission is completed within a reasonable area; C7 is the end area Q generated by the pre-set takeoff mission requirements of the flight. des ;

[0044] The average security concealment throughput is the optimization target of this optimization problem, which is calculated by comparing the throughput R sent to legitimate users in each time slot. s (n) and the throughput R obtained by the eavesdropper e (n), ensuring that the difference is non-negative, and then taking the average value of the total time slots; jointly design the power and trajectory to maximize the optimization target average security covert throughput.

[0045] Further, the specific process of step three is:

[0046] Convert the optimization problem obtained in step 2 into a Markov decision process quintuple, namely, state observation space, reward setting, action setting, state transition probability setting, and discount rate;

[0047] The state observation space transformation is as follows:

[0048] S={s n =(x a [n],y a [n],l a [n])|n=1,2,3,…,N F}

[0049] Among them, the position coordinates of the drone are x a [n] and y a [n] will represent the horizontal position of the drone and the variable l that describes the constraint state of the drone a [n] is a binary value that determines whether the drone satisfies all constraints; if all constraints are satisfied, it is 1, otherwise it is 0;

[0050] The reward settings are converted as follows:

[0051]

[0052] Among them, R[n] is the instantaneous reward of the drone at each step, which combines the distance and instantaneous confidentiality rate rewards, as well as the constraint penalty term; R final is the reward for the drone to reach the target area, representing the flight termination condition of the drone; represents the average confidentiality capacity of the drone during the entire flight time, and α is a measure of the contribution of the average confidentiality capacity to the total reward; is the penalty for the drone flying out of the set boundary, and the detailed explanation of the instantaneous reward is given by the following formula:

[0053] R[n]=ρs R sec [n]+ρ d R des [n]-c1p ò -c2p o

[0054] Among them, R sec is the instantaneous confidentiality rate as the instantaneous reward for each move; R des is the distance reward of the drone from the end point; p ò , p o They are the penalty for the minimum detection error probability of the monitor being too large and the penalty for the mixed interruption probability of UAV transmission; ρ s and ρ d is the reward weight, which is adjusted to make the drone flight balance between instantaneous confidentiality capacity and terminal distance; while c1 to c2 are the weight coefficients of each penalty item; other constraints have been limited in the action space range;

[0055] The action settings are converted as follows:

[0056] A={V x [n],V y [n],P a [n],P j [n]}

[0057] Among them, V x [n] is the instantaneous moving speed of the drone along the x-axis at time slot n, V y [n] is the instantaneous moving speed of the drone along the y-axis at time slot n. Considering the angle problem, the moving speed can be negative; both of them satisfy the maximum speed constraint; P a [n],P j [n] are the transmit powers of the UAV and the friendly jammer, respectively, and these two powers satisfy the maximum power constraint;

[0058] The state transition probability is the probability that the drone will transfer to each possible next state after performing a certain action in the current state; this transition function depends on the current state s n , current action a n , giving the next state s n+1 , which is Pr(s n+1 |s n ,a n ); The discount rate defines the influence of future rewards on current decisions, indicating the extent to which the drone’s future rewards are taken into account in its current decision.

[0059] Further, the specific process of step 4 is:

[0060] After mapping the UAV trajectory optimization problem into step three of the Markov decision process, a deep reinforcement learning algorithm based on double-delayed deep deterministic policy gradient is used to solve it;

[0061] First, initialize the actor network and critic network parameters, the locations of each node in the drone communication system, and the experience replay pool B;

[0062] Initialize state s0 in each drone training iteration and observe the current drone state s n Output drone action a n ; Then observe s n+1 , r and done to determine whether the drone state is terminated; store the transfer sample (s n ,a n ,r,s n+1 ,done) to the experience replay pool B. If the drone flies out of the boundary, does not meet the concealment safety constraint, or reaches the target area, the termination signal done = 1, and the environment state is reset at this time;

[0063] If the set number of update training iterations is reached, a batch of transfer samples (s n ,a n ,r,s n+1 , done), which is used to update the actor network and the critic network until convergence. At this time, the strategy output by the network can maximize the reward obtained by the drone, and maximizing the reward can also maximize the average covert security throughput.

[0064] Among them, the framework of the deep reinforcement learning algorithm based on double-delayed deep deterministic policy gradient has six networks, an online actor network, a target actor network, two online critic networks, and two target critic networks;

[0065] The task of the online actor network is to estimate the flight and power adjustment strategies of the UAV and n Output continuous action value μ(s n |θ μ ); the target actor network is used for soft update, and its structure is consistent with the online actor network; the two online critic networks are used for Q φ Value estimation, two target critic networks are used to smooth the update process.

[0066] Furthermore, an optimization system for the concealed and secure transmission optimization method of the UAV network in the above-mentioned ground node interference cooperation scenario includes:

[0067] The optimization problem construction and Markov decision process conversion module constructs various constraints of the drone network, including safety constraints, transmission interruption probability constraints, etc., and sets up the optimization objective function. Then the optimization problem is converted into a Markov decision process, and the simulation environment of the drone network is constructed.

[0068] The deep reinforcement learning optimal training module is used to perform deep reinforcement learning training based on the double-delayed deep deterministic policy gradient algorithm. This module iteratively inputs the five-tuple in the simulation environment and updates the parameters of the actor network and the critic network until convergence. Finally, the trained optimal strategy is output to achieve the concealed and secure transmission optimization of the UAV network in the scenario of ground node interference cooperation.

[0069] Furthermore, a concealed and secure transmission optimization device for a UAV network in a ground node interference cooperation scenario includes:

[0070] Memory for storing computer programs;

[0071] A processor is used to implement the covert and secure transmission optimization method of the UAV network in the ground node interference cooperation scenario described in any one of steps 1 to 4 when executing the computer program.

[0072] Furthermore, a computer-readable storage medium stores a computer program, which, when executed by a processor, can optimize the concealed and secure transmission of an artificial noise-assisted drone network in a ground node interference cooperation scenario.

[0073] The beneficial effects of the present invention are as follows:

[0074] 1. The present invention constructs an objective function that maximizes the average secure and covert communication throughput from the start point to the end point, converts the optimization problem into a single-agent Markov decision process, and finally constructs a network based on double-delay deep deterministic policy gradient to jointly optimize the UAV trajectory power. It can effectively solve the covert secure transmission optimization problem and has good transmission performance under higher concealment constraints.

[0075] 2. The present invention realizes the driven exploration of the drone's own environmental data by constructing a deep reinforcement learning environment, and by integrating a unique reward shaping mechanism into the Markov decision process, it can effectively reduce the local convergence of the algorithm during training and enable rapid convergence during training.

[0076] 3. The present invention realizes the covert and secure transmission of UAV networks under node interference and cooperation scenarios by converting the UAV optimization problem into a Markov decision process and solving it using a deep reinforcement learning network. It can provide theoretical guidance for the design of covert and secure transmission methods for UAVs assisted by ground nodes, expand the application scope of deep reinforcement learning, and enhance the dynamic transmission capabilities of UAV networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0078] Figure 1 It is a flow chart of a covert and secure transmission optimization method for a UAV network in a ground node interference cooperation scenario according to an embodiment of the present invention.

[0079] Figure 2 It is a structural diagram of a covert and secure transmission optimization system for a UAV network in a ground node interference cooperation scenario according to an embodiment of the present invention.

[0080] Figure 3 is a schematic diagram of the concealed and secure transmission performance of the UAV network in the ground node interference cooperation scenario according to an embodiment of the present invention; wherein, Figure 3 (a) shows the average covert secure transmission throughput performance comparison chart based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) and Deep Deterministic Policy Gradient (DDPG) frameworks, using the reward shaping mechanism (Power and Trajectory Control, PTC), based on the average power (average power, AP) and without the reward shaping mechanism (no reward shaping, NRS) strategy. TD3-PTC is the method of the present invention; Figure 3 (b) is the average hidden security throughput curve of the method of the present invention under different constraints. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0082] In view of the problems existing in the prior art, the present invention provides a covert and secure transmission optimization method for a UAV network in a ground node interference cooperation scenario. The present invention is described in detail below with reference to the accompanying drawings.

[0083] like Figure 1 As shown, a concealed and secure transmission optimization method for a UAV network in a ground node interference cooperation scenario according to an embodiment of the present invention includes the following steps:

[0084] S101, construct the system model of the artificial noise-assisted UAV network of ground nodes, and construct the concealment constraints obtained by the binary detection analysis method of the UAV monitor;

[0085] S102, based on the concealment constraint constructed in S101, construct an objective function for maximizing the average secure covert communication throughput from the start point to the end point under constraints such as the drone boundary and the drone position;

[0086] S103, under the optimization problem of the objective function constructed in S102, the optimization problem is transformed by using the five-tuple of state observation space, reward setting, action setting, state transition probability setting, and discount rate;

[0087] S104, using the Markov decision process obtained by conversion in S103, a UAV power trajectory joint optimization scheme based on deep delayed deterministic policy gradient is designed to optimize the target decision, thereby achieving the target solution.

[0088] like Figure 2 As shown, the concealed and secure transmission optimization system of the UAV network in the ground node interference cooperation scenario of the embodiment of the present invention includes:

[0089] Optimization problem construction module 1 is used to construct various constraints of the UAV network, including safety constraints, transmission interruption probability constraints, etc., and establish the optimization objective function;

[0090] Markov decision process conversion module 2, used to convert the optimization problem into a Markov decision process and build a simulation environment for the UAV network;

[0091] The deep reinforcement learning optimal training module 3 is used to perform deep reinforcement learning training based on the double-delayed deep deterministic policy gradient algorithm. This module iteratively inputs the five-tuple in the simulation environment and updates the parameters of the actor network and the critic network until convergence. Finally, the trained optimal strategy is output to achieve the optimization of concealed and secure transmission of artificial noise-assisted drone networks in the scenario of ground node interference cooperation.

[0092] The present invention will be further described below in conjunction with the embodiments.

[0093] A method for optimizing concealed and secure transmission of a UAV network in a ground node interference cooperation scenario according to an embodiment of the present invention includes the following steps:

[0094] S101, construct a system model of the artificial noise-assisted UAV network of ground nodes, and construct the concealment constraints obtained by the binary detection analysis method of UAV monitors.

[0095] Based on the artificial noise assisted drone network of ground nodes, it is assumed that the drone transmitter Alice in the air sends useful information to the legitimate user Bob on the ground, and there is a monitor Willie, an eavesdropper Eve and a friendly jammer jammer on the ground. The concealment constraint can be expressed as:

[0096] ξ * >1-ò

[0097] Among them, ξ * is the derived Willie's minimum detection error probability, where ò is a decimal close to zero and can be used to quantitatively describe the concealment performance of the system.

[0098] First, the monitor power binary detection is designed according to the UAV system model:

[0099]

[0100] Among them, T w [n] is the average power of the signal received by the assumed monitor, P a [n] and P j [n] represents the power transmitted by the UAV and the jammer in the nth time slot, |h jw [n]| 2 and |h aw [n]| 2 is the channel gain from the friendly jammer and the drone transmitter Alice to Willie. The background noise at Willie is considered as Gaussian noise and is expressed as

[0101] The probability of a false alarm is P FA =Pr{D1|H0}, the probability of missed detection is P MD =Pr{D0|H1}. The monitoring threshold P is set in advance. th By comparison, the false alarm probability, missed detection probability and detection error probability of Willie are:

[0102]

[0103] ξ=P FA +P MD

[0104] According to the exponential distribution of the channel gain from the jammer to the monitor, we can get:

[0105]

[0106] Among them, τ f ' and τ m ' are the derivatives of their respective functions, respectively

[0107] Taking the derivative of the original function, the optimal detection power threshold is: Substituting the minimum detection error probability, we get:

[0108]

[0109] S102, construct an objective function that maximizes the average secure covert communication throughput from the start point to the end point under constraints such as the drone boundary and the drone position.

[0110] Firstly, the UAV boundary constraints, UAV speed constraints and UAV starting position constraints are set. Then, the mixed interruption probability is used to design the safety interruption constraints and construct the objective function of average safety covert throughput.

[0111] Considering the interruption constraint, concealment constraint, UAV transmission power constraint, interference power of artificial noise carrier node and UAV flight constraint, the optimization problem of the average safe and concealed throughput of the UAV from the starting point to the end point can be given by the following formula:

[0112]

[0113] C7:q a [0] = q s ,q a [N F ]∈Q des

[0114] Among them, C1 represents the concealment constraint, which must ensure that the transmission is carried out when the probability of detection error by the monitor is greater than a certain value, and C2 is the probability P of transmission interruption. HOP Cannot exceed a certain threshold value P OUT , ensuring that users can receive data safely while preventing eavesdroppers from stealing more confidential information. The power P of the drone transmitter and the ground jammer a [n], P j [n] should meet the requirement of less than less than the maximum power C5 expresses that the moving distance of the drone between any two consecutive time slots, such as n to n+1, is subject to the maximum speed V maxC6 is the flight area D setting of the drone, which is used to ensure that the mission is completed within a reasonable area, and C7 is the end area Q generated by the pre-set takeoff mission requirements. des .

[0115] The average security concealment throughput is the optimization target of this optimization problem, which is calculated by comparing the throughput R sent to legitimate users in each time slot. s (n) and the throughput R obtained by the eavesdropper e The difference between (n) is ensured to be non-negative, and then the average value of the total time slots is taken. The power and trajectory are jointly designed to maximize the average security covert throughput of the optimization target.

[0116] S103, using the five-tuple of state observation space, reward setting, action setting, state transition probability setting, and discount rate to transform the optimization problem.

[0117] The optimization problem is converted into a Markov decision process quintuple, namely, state observation space, reward setting, action setting, state transition probability setting, and discount rate.

[0118] The state observation space transformation is as follows:

[0119] S={s n =(x a [n],y a [n],l a [n])|n=1,2,3,…,N F}

[0120] Among them, the position coordinates of the drone are x a [n] and y a [n] will represent the horizontal position of the drone and the variable l that describes the constraint state of the drone a [n] is a binary value that determines whether the drone satisfies all constraints. If all constraints are satisfied, it is 1, otherwise it is 0.

[0121] The reward settings are converted as follows:

[0122]

[0123] Among them, R[n] is the instantaneous reward of the drone at each step, which combines the distance and instantaneous confidentiality rate rewards, as well as the constraint penalty term. final It is the reward for the drone to reach the target area, representing the termination condition of the drone’s flight. represents the average confidentiality capacity of the drone during the entire flight time, and α measures the contribution of the average confidentiality capacity to the total reward. is the penalty for the drone flying out of the set boundary, and the detailed explanation of the instantaneous reward is given by the following formula:

[0124] R[n]=ρ s R sec [n]+ρ d R des [n]-c1p ò -c2p o

[0125] Among them, R sec is the instantaneous confidentiality rate as the instantaneous reward for each move. des is the distance reward of the drone from the end point. ò , p o They are the penalty for the minimum detection error probability of the monitor being too large and the penalty for the mixed interruption probability of UAV transmission. s and ρ d is the reward weight, which is adjusted to make the drone fly in a balance between instantaneous confidentiality capacity and terminal distance. And c1 to c2 are the weight coefficients of each penalty term. Other constraints have been limited in the action space.

[0126] The action settings are converted as follows:

[0127] A={V x [n],V y [n],P a [n],P j [n]}

[0128] Among them, V x [n] is the instantaneous moving speed of the drone along the x-axis at time slot n, V y [n] is the instantaneous moving speed of the drone along the y-axis at time slot n. Considering the angle problem, the moving speed can be negative. Both of them satisfy the maximum speed constraint. a [n],P j [n] are the transmit powers of the UAV and the friendly jammer, respectively, and these two powers satisfy the maximum power constraint.

[0129] The state transition probability is the probability that the drone will transfer to each possible next state after performing a certain action in the current state. This transition function depends on the current state s n , current action a n , giving the next state s n+1 , which is Pr(s n+1 |s n ,a n ). The discount rate defines the degree of influence of future rewards on current decisions, indicating the extent to which the drone’s future rewards are taken into account in its current decisions.

[0130] S104, using the converted Markov decision process, a UAV power trajectory joint optimization scheme based on deep delayed deterministic policy gradient is designed to optimize the target decision, thereby achieving the target solution.

[0131] After mapping the UAV trajectory optimization problem into step three of the Markov decision process, a deep reinforcement learning algorithm based on double-delayed deep deterministic policy gradient is used to solve it.

[0132] First, initialize the actor network and critic network parameters, the locations of each node in the drone communication system, and the experience replay pool B.

[0133] Initialize state s0 in each drone training iteration and observe the current drone state s n Output drone action a n Then observe n+1 , r and done to determine whether the drone state is terminated. Storage transfer samples (s n ,a n ,r,s n+1 ,done) to the experience replay pool B. If the drone flies out of the boundary, does not meet the concealment safety constraints, or reaches the target area, the termination signal done = 1, and the environment state is reset.

[0134] If the set number of update training iterations is reached, a batch of transfer samples (s n ,a n ,r,s n+1 , done), which is used to update the actor network and the critic network until convergence. At this time, the strategy output by the network can maximize the reward obtained by the drone, and maximizing the reward can also maximize the average covert security throughput.

[0135] Among them, the framework of the deep reinforcement learning algorithm based on double-delay deep deterministic policy gradient has six networks, an online actor network, a target actor network, two online critic networks, and two target critic networks.

[0136] The task of the online actor network is to estimate the flight and power adjustment strategies of the UAV and n Output continuous action value μ(s n |θ μ ). The target actor network is used for soft updates and has the same structure as the online actor network. The two online critic networks are used for Q φ Value estimation, two target critic networks are used to smooth the update process.

[0137] The technical effects of the present invention are described in detail below in conjunction with simulation experiments.

[0138] In order to evaluate the performance of the present invention, simulation verification was carried out. The simulation environment was built on Python 3.9 and Gym0.21.0 platforms. The present invention proposes a power trajectory joint optimization algorithm based on TD3 to achieve covert secure transmission, namely the TD3-PTC (Power and Trajectory Control) scheme, which uses an actor network and a critic network with two hidden layers of 256 nodes, and adopts the ReLU activation function, and uses the Adam optimizer to train the iterative episode of reinforcement learning. In order to verify the effect of the TD3-PTC algorithm, two baseline schemes are introduced for comparison. Among them, Scheme 1, namely TD3-AP (averagepower), fixes the transmission power of the drone and the interference power of the jammer. The drone uses the average power transmission, and the jammer uses the minimum interference power that meets the concealment constraint. This scheme only optimizes the trajectory of the drone. Scheme 2, namely TD3-NRS (no reward shaping), completely excludes the endpoint reward mechanism, and the reward only depends on the performance of each step, rather than the average covert transmission rate reward of the endpoint. Figure 3 (a) shows the average covert secure transmission throughput performance comparison chart based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) and Deep Deterministic Policy Gradient (DDPG) framework, using the reward shaping mechanism (Power and Trajectory Control, PTC), based on the average power (average power, AP) and without the reward shaping mechanism (no reward shaping, NRS) strategy. TD3-PTC is the method of the present invention. It can be seen that the performance of TD3-PTC is better than other algorithms under most time slot number settings, which proves the effectiveness of the TD3-PTC method. The average secure covert throughput of the method proposed in the present invention under different constraints is shown in the figure. Figure 3 (b) is shown. Figure 3 (b) It can be seen that the performance of the proposed method is better than that of the DDPG algorithm in all three cases, and the proposed method is robust under various constraints.

[0139] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A concealed and secure transmission optimization method for UAV networks in a ground node interference cooperation scenario, characterized in that: include: Step 1: construct a system model of the artificial noise-assisted UAV network of ground nodes and construct the concealment constraints obtained by the binary detection analysis method of the UAV monitor; Step 2: Based on the concealment constraint constructed in step 1, an objective function is constructed to maximize the average secure covert communication throughput from the start point to the end point under the constraints of the drone boundary and drone position. Step 3: Under the optimization problem of the objective function constructed in step 2, the optimization problem is transformed by using the five-tuple of state observation space, reward setting, action setting, state transition probability setting, and discount rate; Step 4: Using the Markov decision process obtained in step 3, a network based on deep delayed deterministic policy gradient is designed to jointly optimize the UAV power trajectory to make the target decision optimal and achieve the target solution.

2. The method for optimizing concealed and secure transmission of a UAV network in a ground node interference cooperation scenario as claimed in claim 1, characterized in that: The specific process of step one is: Based on the artificial noise assisted UAV network of ground nodes, it is assumed that the UAV transmitter Alice in the air sends useful information to the legitimate user Bob on the ground, and there is a monitor Willie, an eavesdropper Eve and a friendly jammer jammer on the ground; the concealment constraint can be expressed as: x * >1-ò Among them, ξ * is the derived Willie's minimum detection error probability, where ò is a decimal close to zero, which can be used to quantitatively describe the concealment performance of the system; First, the monitor power binary detection is designed according to the UAV system model: Among them, T w [n] is the average power of the signal received by the assumed monitor, P a [n] and P j [n] represents the power transmitted by the UAV and the jammer in the nth time slot, |h jw [n]| 2 and |h aw [n]| 2 is the channel gain from the friendly jammer and the drone transmitter Alice to Willie; the background noise at Willie is considered as Gaussian noise and is expressed as The probability of a false alarm is P FA =Pr{D1|H0}, the probability of missed detection is P MD =Pr{D0|H1}; by the preset monitoring threshold P th By comparison, the false alarm probability, missed detection probability and detection error probability of Willie are: ξ=P FA +P MD According to the exponential distribution of the channel gain from the jammer to the monitor, we can get: Among them, τ f ' and τ m ' are the derivatives of their respective functions, respectively Taking the derivative of the original function, the optimal detection power threshold is: Substituting the minimum detection error probability, we get:

3. The method for optimizing concealed and secure transmission of a UAV network in a ground node interference cooperation scenario as claimed in claim 1, characterized in that: The specific process of step 2 is as follows: First, according to the system model in step 1, the UAV boundary constraints, UAV speed constraints and UAV starting position constraints are set. Then, the mixed interruption probability is used to design the safety interruption constraints and construct the objective function of average safety concealed throughput. Considering the interruption constraint, concealment constraint, UAV transmission power constraint, interference power of artificial noise carrier node and UAV flight constraint, the optimization problem of the average safe and concealed throughput of the UAV from the starting point to the end point can be given by the following formula: C7:q a [0]=q s ,q a [N F ]∈Q des Among them, C1 represents the concealment constraint, which must ensure that the transmission is carried out when the probability of detection error by the monitor is greater than a certain value, and C2 is the probability P of transmission interruption. HOP Cannot exceed a certain threshold value P OUT , ensuring that users can receive data safely while preventing eavesdroppers from stealing more confidential information; the power P of the drone transmitter and the ground jammer a [n], P j [n] should meet the requirement of less than less than the maximum power C5 expresses that the moving distance of the drone between any two consecutive time slots, such as n to n+1, is subject to the maximum speed V max C6 is the flight area D setting of the drone, which is used to ensure that the mission is completed within a reasonable area; C7 is the end area Q generated by the pre-set takeoff mission requirements of the flight. des ; The average security concealment throughput is the optimization target of this optimization problem, which is calculated by comparing the throughput R sent to legitimate users in each time slot. s (n) and the throughput R obtained by the eavesdropper e (n), ensuring that the difference is non-negative, and then taking the average value of the total time slots; jointly design the power and trajectory to maximize the optimization target average security covert throughput.

4. The method for optimizing concealed and secure transmission of a UAV network in a ground node interference cooperation scenario as claimed in claim 1, characterized in that: The specific process of step three is: Convert the optimization problem obtained in step 2 into a Markov decision process quintuple, namely, state observation space, reward setting, action setting, state transition probability setting, and discount rate; The state observation space transformation is as follows: S={s n =(x a [n],y a [n],l a [n])|n=1,2,3,…,N F } Among them, the position coordinates of the drone are x a [n] and y a [n] will represent the horizontal position of the drone and the variable l that describes the constraint state of the drone a [n] is a binary value that determines whether the drone satisfies all constraints; if all constraints are satisfied, it is 1, otherwise it is 0; The reward settings are converted as follows: Among them, R[n] is the instantaneous reward of the drone at each step, which combines the distance and instantaneous confidentiality rate rewards, as well as the constraint penalty term; R final is the reward for the drone to reach the target area, representing the flight termination condition of the drone; represents the average confidentiality capacity of the drone during the entire flight time, and α is a measure of the contribution of the average confidentiality capacity to the total reward; is the penalty for the drone flying out of the set boundary, and the detailed explanation of the instantaneous reward is given by the following formula: R[n]=ρ s R sec [n]+p d R des [n]-c1p ò -c2p o Among them, R sec is the instantaneous confidentiality rate as the instantaneous reward for each move; R des is the distance reward of the drone from the end point; p ò , p o They are the penalty for the minimum detection error probability of the monitor being too large and the penalty for the mixed interruption probability of UAV transmission; ρ s and ρ d is the reward weight, which is adjusted to make the drone flight balance between instantaneous confidentiality capacity and terminal distance; while c1 to c2 are the weight coefficients of each penalty item; other constraints have been limited in the action space range; The action settings are converted as follows: A={V x [n],V y [n],P a [n],P j [n]} Among them, V x [n] is the instantaneous moving speed of the drone along the x-axis at time slot n, V y [n] is the instantaneous moving speed of the drone along the y-axis at time slot n. Considering the angle problem, the moving speed can be negative; both of them satisfy the maximum speed constraint; P a [n],P j [n] are the transmit powers of the UAV and the friendly jammer, respectively, and these two powers satisfy the maximum power constraint; The state transition probability is the probability that the drone will transfer to each possible next state after performing a certain action in the current state; this transition function depends on the current state s n , current action a n , giving the next state s n+1 , which is Pr(s n+1 |s n ,a n ); The discount rate defines the influence of future rewards on current decisions, indicating the extent to which the drone’s future rewards are taken into account in its current decision.

5. The method for optimizing concealed and secure transmission of a UAV network in a ground node interference cooperation scenario as claimed in claim 1, characterized in that: The specific process of step 4 is as follows: After mapping the UAV trajectory optimization problem into step three of the Markov decision process, a deep reinforcement learning algorithm based on double-delayed deep deterministic policy gradient is used to solve it; First, initialize the actor network and critic network parameters, the locations of each node in the drone communication system, and the experience replay pool B; Initialize state s0 in each drone training iteration and observe the current drone state s n Output drone action a n ; Then observe s n+1 , r and done to determine whether the drone state is terminated; store the transfer sample (s n ,a n ,r,s n+1 ,done) to the experience replay pool B. If the drone flies out of the boundary, does not meet the concealment safety constraint, or reaches the target area, the termination signal done = 1, and the environment state is reset at this time; If the set number of update training iterations is reached, a batch of transfer samples (s n ,a n ,r,s n+1 , done), which is used to update the actor network and the critic network until convergence. At this time, the strategy output by the network can maximize the reward obtained by the drone, and maximizing the reward can also maximize the average covert security throughput.

6. The method for optimizing concealed and secure transmission of a UAV network in a ground node interference cooperation scenario as claimed in claim 5, characterized in that: In the step 4, the framework of the deep reinforcement learning algorithm based on double-delayed deep deterministic policy gradient has six networks, an online actor network, a target actor network, two online critic networks, and two target critic networks; The task of the online actor network is to estimate the flight and power adjustment strategies of the UAV and n Output continuous action value μ(s n |θ μ ); the target actor network is used for soft update, and its structure is consistent with the online actor network; the two online critic networks are used for Q φ Value estimation, two target critic networks are used to smooth the update process.

7. An optimization system for implementing the concealed and secure transmission optimization method of a UAV network in a ground node interference cooperation scenario as described in any one of claims 1 to 6, characterized in that: The artificial noise assisted UAV network optimization system comprises: The optimization problem building module is used to build various constraints of the drone network, including safety constraints, transmission interruption probability constraints, etc., and set up the optimization objective function; The Markov decision process conversion module is used to convert the optimization problem into a Markov decision process and build a simulation environment for the UAV network; The deep reinforcement learning optimal training module performs deep reinforcement learning training on the double-delayed deep deterministic policy gradient network; this module iteratively inputs the quintuple in a simulation environment and updates the parameters of the actor network and the critic network until convergence; it finally outputs the trained optimal strategy to achieve covert and secure transmission optimization of the UAV network in the scenario of ground node interference cooperation.

8. A concealed and secure transmission optimization device for UAV networks in a ground node interference cooperation scenario, characterized in that: include: Memory for storing computer programs; A processor is used to implement the covert and secure transmission optimization method of the UAV network in the ground node interference cooperation scenario described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, can optimize the concealed and secure transmission of a drone network in a ground node interference cooperation scenario.

Images