Resource allocation method for multi-hop relay secure communication system of unmanned aerial vehicle

Abstract

The unmanned aerial vehicle multi-hop relay secret communication system resource allocation method belongs to the technical field of unmanned aerial vehicle communication.The unmanned aerial vehicle group assisted multi-hop relay secret communication model is constructed, and based on the communication model, a joint optimization model is established, the minimum secrecy rate in a cycle is maximized as a joint optimization objective function, and the unmanned aerial vehicle flight speed, flight angle, transmission power and user scheduling coefficient are solved;based on the joint optimization model, a deep deterministic policy gradient algorithm architecture is built, deep reinforcement learning training is carried out, the optimal action of the unmanned aerial vehicle group and the maximum reward are obtained, the optimal action is the optimal resource allocation of the unmanned aerial vehicle multi-hop relay secret communication system, and the maximum reward is the maximum minimum secrecy rate reached by the unmanned aerial vehicle group assisted multi-hop relay secret communication model.The information security problem of long-distance communication obstruction is solved, the safety communication is ensured, and the optimal resource configuration of the communication system is realized.

Description

Technical Field

[0001] This invention belongs to the field of UAV communication technology, and particularly relates to a resource allocation method for a multi-hop relay secure communication system for UAVs. Background Technology

[0002] With the continuous improvement of network technology and the arrival of the 5G era, people have increasingly higher requirements for network coverage and speed. Unmanned Aerial Vehicles (UAVs) offer a way to meet these requirements. A UAV is an unmanned aerial vehicle that is remotely controlled via wireless communication and controlled by its own software. UAVs have low daily maintenance costs, are easy to maintain, and their small size makes them easy to carry. With advancements in wireless communication and UAV technology, UAVs can also serve as mobile relays to assist in communication. When there are mountains or large buildings between multiple communication nodes, using UAVs as mobile relays, and setting appropriate forwarding protocols based on the actual channel environment, allows for signal processing and forwarding, facilitating normal communication between the source and destination nodes.

[0003] In general, most anti-eavesdropping technologies used in wireless communication systems research primarily include key encryption and Wireless Physical Layer Security (WPLS). Compared to key encryption, WPLS can guarantee system security without complex algorithms and keys, making it a widely used anti-eavesdropping technology in wireless communication. Relay is a common WPLS method for improving system communication security. When UAVs are used as mobile relays, they can leverage their flexibility to improve the security performance of the communication system. Specifically, by adjusting their flight direction and speed in real time, they can establish a wireless channel with better quality with legitimate receivers and a lower quality wireless channel with unauthorized eavesdroppers, thereby achieving secure information transmission in the UAV relay system and ensuring reliable communication between the source and destination nodes. Therefore, UAV relay technology has significant practical implications for improving the security performance of UAV relay communication systems.

[0004] Existing research on the physical layer security of UAV relay communication mainly focuses on optimizing the transmission power or flight trajectory of a single UAV relay to maximize the confidentiality of the communication system. It does not consider the confidentiality of communication between long-distance mobile nodes, i.e., the confidentiality of UAV multi-hop relay communication, and lacks research on improving the security performance of long-distance communication systems by cooperating to interfere with UAVs.

[0005] In researching UAV relay communication systems, the minimum security ratio is often used as an indicator to evaluate the system's security performance. Studying how to obtain the optimal system resource allocation to maximize the minimum security ratio involves problems in continuous action spaces and complex state spaces. The system often has multiple constraints and nonlinear characteristics, which transforms it into a non-convex problem with various constraints, making it difficult to solve using traditional convex optimization methods. With the development of reinforcement learning technology, researchers have also explored using reinforcement learning to solve this problem, but the results have not been ideal.

[0006] Deep Reinforcement Learning (DRL) encompasses many algorithms, such as Deep Q Networks (DQN). These combine the powerful fitting capabilities of neural networks with learning algorithms to help agents learn optimal policies. Q-learning is a value function-based algorithm that stores states, actions, and their corresponding state values ​​in a table. It updates the Q-value by selecting the maximum Q-value in the next state. This learning method is well-suited for problems with discrete and relatively small action and state spaces. However, when the number of states is large, the table becomes very dimensional, significantly slowing down the process and placing high demands on device memory. The DQN algorithm incorporates artificial neural networks to avoid the problem of excessively large Q-table dimensions in Q-learning when the state-action space is large. However, DQN is also a value function-based algorithm, meaning that the action space it obtains is also discrete. When the variable being optimized is a continuous action, discretizing the values ​​may make it difficult to find the optimal policy.

[0007] In reinforcement learning (RL), another type is policy-based methods. In this approach, the policy is represented as a continuous function, and the optimal policy is obtained directly by optimizing it. Policy gradient (PG) is one such policy-based RL algorithm. PG uses the current state of the system as input to the neural network and outputs the probability or probability distribution of taking each action in the current state. As analyzed earlier, neural network training largely relies on backpropagation, i.e., minimizing the loss by gradient descent of the error function to obtain the corresponding policy. In the PG algorithm, to directly obtain the policy, a reward function is used as the objective function, increasing the probability of taking actions that yield higher rewards until the objective function converges and the optimal policy is output. While the PG algorithm can solve the problems of continuous actions and state space, its reward is based on a single round, resulting in low learning efficiency. To address this issue, the AC (actor-critic) network is introduced. It consists of two parts: an actor network (AN) and a critic network (CN). The actor network is often translated as a policy network or actor network. AN, derived from PG, can select suitable actions from the continuous action space. Critics networks, often translated as value networks or critic networks, employ the DQN algorithm from the value function-based approach to evaluate the actions output by the AN. Based on the feedback from the CN, the network updates its own parameters, i.e., updates the probability of selecting an action. While the AC algorithm addresses the inefficiency issue of the PG algorithm, it still faces challenges. When using neural networks to estimate the value function, the correlation between data is strong, and the AN's action selection probability is based on the value function estimated by the CN. Since the CN itself has difficulty converging, achieving convergence for both becomes even more difficult. Summary of the Invention

[0008] To address the aforementioned technical problems and more effectively improve the security performance of communication systems, meeting the needs of secure communication between paired mobile nodes in situations with potential eavesdroppers, this invention provides a resource allocation method for a UAV multi-hop relay secure communication system, comprising:

[0009] Constructing a drone swarm-assisted multi-hop relay secure communication model;

[0010] Based on the UAV swarm-assisted multi-hop relay secure communication model, a joint optimization model is established. In the joint optimization model, the objective function is to maximize the minimum security rate within a cycle, and the UAV flight speed, flight angle, transmission power and user scheduling coefficient are solved.

[0011] Based on the joint optimization model, a deep reinforcement learning architecture for a multi-hop relay secure communication network for unmanned aerial vehicles is constructed. The deep reinforcement learning architecture adopts a deep deterministic policy gradient algorithm architecture.

[0012] Using the aforementioned deep deterministic policy gradient algorithm architecture, reinforcement learning training is performed to obtain the optimal action and maximum reward of the UAV swarm. The optimal action is the optimal resource allocation of the UAV multi-hop relay secure communication system, namely the optimal trajectory, optimal user scheduling, and optimal UAV swarm transmission power. The maximum reward is the maximum minimum security rate achieved by the UAV swarm-assisted multi-hop relay secure communication model.

[0013] The beneficial effects of this invention are as follows: First, this invention constructs a model of a UAV swarm-assisted multi-hop relay secure communication system. This model includes potential eavesdroppers, mobile users (including target nodes and source nodes), UAV relays, and jamming UAVs, essentially constituting a UAV swarm-assisted multi-hop relay secure communication system model based on cooperative interference. This model constructs reliable communication links for long-distance mobile nodes or long-distance communication with obstacles, ensuring secure communication in the UAV swarm-assisted multi-hop relay secure communication system and solving the information security problem when long-distance communication is blocked. Second, with the goal of maximizing the minimum security rate of the system, a joint optimization model of the UAV swarm's flight trajectory, transmission power, and user scheduling is established. Then, based on this joint optimization model, a Deep Deterministic Policy Gradient (DDPG) architecture is constructed using Markov decision processes. Finally, through the DDPG architecture, the optimal resource allocation of the UAV swarm-assisted multi-hop relay secure communication system model is achieved, namely, the optimal trajectory of the UAV swarm, the optimal user scheduling, and the optimal transmission power of the UAV swarm, enabling the UAV swarm-assisted multi-hop relay secure communication system to reach the maximum minimum security rate. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the steps of an embodiment of the present invention;

[0015] Figure 2 This is a schematic diagram of the UAV swarm-assisted multi-hop relay secure communication system of the present invention;

[0016] Figure 3 This is a schematic diagram of the network structure based on the deep deterministic policy gradient algorithm in this invention;

[0017] Figure 4 This is a comparison chart of the minimum security rate of the system under different resource allocation schemes in the simulation environment of this invention embodiment. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] This invention provides a resource allocation method for a multi-hop relay secure communication system for unmanned aerial vehicles (UAVs), such as... Figure 1 As shown, it includes:

[0020] Step A: Construct a drone swarm-assisted multi-hop relay secure communication model;

[0021] Step B: Based on the UAV swarm-assisted multi-hop relay secure communication model, establish a joint optimization model. In the joint optimization model, the objective function is to maximize the minimum security rate within a cycle, and solve for the UAV flight speed, flight angle, transmission power, and user scheduling coefficient.

[0022] Step C: Based on the joint optimization model, a deep reinforcement learning architecture for a multi-hop relay secure communication network for unmanned aerial vehicles is built. The deep reinforcement learning architecture adopts a deep deterministic policy gradient algorithm architecture.

[0023] Step D: Using the deep deterministic policy gradient algorithm architecture, perform deep reinforcement learning training to obtain the optimal action and maximum reward of the UAV swarm. The optimal action is the optimal resource allocation of the UAV multi-hop relay secure communication system, namely the optimal trajectory, optimal user scheduling, and optimal UAV swarm transmission power. The maximum reward is the maximum minimum security rate achieved by the UAV swarm-assisted multi-hop relay secure communication model.

[0024] Furthermore, the UAV swarm-assisted multi-hop relay secure communication model is a complete eavesdropping model, which includes: source node S m Target node D m Several UAV relay R k Ground eavesdropper E and mobile jammer J, the source node S m Through several drone relays R k Send information to destination node D m The location of the ground eavesdropper E is unknown, and the drone relay R... k The mobile jammer J transmits jamming signals in response to the forwarding information signal.

[0025] like Figure 2 As shown, the UAV swarm-assisted multi-hop relay secure communication system includes source node S mTarget node D m Several UAV relay R k Ground-based eavesdropper E and mobile jammer J. The system model is a complete eavesdropping model, where source node S... m , and the destination node D m , M represents the number of source and destination nodes. Source and destination nodes are nodes that move according to their own task requirements and usually appear in pairs. Source and destination nodes are referred to as users. Source node S m and the destination node D m Direct communication is impossible due to long distances or signal obstruction. Unmanned aerial vehicles (UAVs) are used as mobile UAV relays to rebuild the communication link, achieving the goal of source node S... m and the destination node D m Communication between them. UAV relay R k , Where K is a positive integer, and R is the drone relay. k The set is represented as The location of the ground eavesdropper E is unknown to all source and destination nodes, but can be detected from a local oscillator unintentionally leaked from the RF front end. A drone is introduced as a mobile jammer J, which emits jamming signals to counter the ground eavesdropper E. The drone used as a relay is simply referred to as the relay drone R. k The drone used as a mobile jammer is simply called a jamming drone J. Although the jamming drone J also interferes with the drone relay R... k and the destination node D m However, by reasonably optimizing the position and transmission power of the jamming drone J, the interference caused to the eavesdropper can be greater than that caused to the legitimate receiver.

[0026] Furthermore, in the UAV swarm-assisted multi-hop relay secure communication model, the UAV trajectory is divided into connecting lines representing the positions of the UAVs in each time slot throughout their flight cycle. The positions of each UAV in adjacent time slots satisfy constraints. In time slot n, the relay UAV forwards information signals, while the interfering UAV transmits jamming signals. The relay UAV's transmission power P... R and the transmission power P of the jamming drone J Simultaneously satisfying the constraints; the relay UAV swarm uses user scheduling to serve multiple pairs of source and destination nodes, and the relay UAV swarm provides service to at most one pair of source and destination nodes in the nth time slot; calculate the source node S in the nth time slot. m Information transmission rate to R1 link Calculate the R of the relay UAV in n time slots M To the destination node D mInformation transmission rate Calculate the R of the relay UAV in n time slots k To relay drone R k+1 Information transmission rate of the link Calculate the rate at which ground eavesdropper E intercepts information in time slot n. For each time slot n, the information rate of the relay drone swarm satisfies the constraints.

[0027] In this embodiment of the invention, the flight altitude of all drones (including relay drone swarms and jamming drones) is fixed. Source node S m Target node D m Together with ground eavesdropper E, they form a ground node, with the horizontal coordinates of ground node i being w. i =[x i ,y i ] T The unit is meters, where i∈{S} m D m ,E},x i The x-coordinate of ground node i is represented by y. i Let represent the ordinate of ground node i. In this embodiment of the invention, the flight period of the UAV is set to T, and it is divided into N equal-length time slots, i.e., T = Nτ, where τ is the length of the unit time slot and N is a positive integer. This invention assumes that UAV j starts from a predetermined initial position. and final position Flight, among which The trajectory of a UAV can be divided into lines connecting the positions of each time slot within the entire cycle. The coordinates of each UAV, including the relay UAV swarm and the jamming UAV, in the nth time slot are denoted as q. j [n] = [x] j (n),y j (n)] T ,in Due to source node S m and target node D m It is mobile; its position in the nth time slot can also be represented as w. i (n)=[x i (n),y i (n)] T Since each UAV has limited flight capabilities, the position of each UAV in adjacent time slots satisfies the constraints.

[0028] In the UAV swarm-assisted multi-hop relay secure communication model, the UAV trajectory is divided into connecting lines representing the positions of the UAVs in each time slot throughout the entire flight cycle. The position of each UAV in adjacent time slots satisfies constraints, specifically:

[0029]

[0030]

[0031]

[0032] Where L=τv max Indicates drone relay R k Or, the maximum length of the mobile jammer J flying at maximum speed within a time slot, where τ is the length of the unit time slot, v max d represents the maximum flight speed of each UAV. min q represents the minimum safe distance between unmanned aerial vehicles (UAVs). j (n+1) represents the coordinates of UAV j in the (n+1)th time slot, q j (n) represents the coordinates of UAV j in the nth time slot. q represents the initial position of drone j. j (1) represents the coordinates of UAV j in the first time slot, q i (n) The coordinates of UAV i in the nth time slot.

[0033] In this embodiment of the invention, the system communication links in the UAV swarm-assisted multi-hop relay communication model are classified as follows: (1) Legal links: S m -R link is the source node S m - UAV relay R-link, RR-link, i.e., UAV relay R-link and RD m The link is the drone relay R-target node D. m (2) Eavesdropping link: RE link, i.e., UAV relay R-ground eavesdropper E link, where ground eavesdropper E eavesdrops on UAV relay UAV; (3) Jamming link: JE link, i.e., jamming UAV J-ground eavesdropper E link, JR link, i.e., jamming UAV J-UAV relay R link and JD m The link is the interference drone J-target node D m The signal that interferes with the transmission of UAV J is considered to be the UAV relay R. k and the destination node D mThe noise is reduced. Based on the node location of each link, the relay UAV and the jamming UAV have air-to-air (A2A) links, while the remaining links are air-to-ground (A2G) links. Both the air-to-air (A2A) and air-to-ground (A2G) links are controlled by line-of-sight (LOS) links, and the Doppler effect caused by UAV mobility is perfectly compensated. The communication channel between UAVs adopts a free-space path loss model. The power gain of the n-timeslot air-to-air (A2A) link channel is calculated using the following formula:

[0034]

[0035] in, Indicates the relay drone R k To another relay drone Or interfere with the J-channel gain of the drone. And j≠j * j represents a relay drone or jamming drone. * This indicates another relay drone or jamming drone, and ρ0 represents the channel power at a reference distance d0 = 1m. Indicates the relay drone R k To another relay drone Or interfere with the distance of drone J, q j (n) represents the coordinates of a relay UAV j or jamming UAV j in the nth time slot. This indicates another relay drone j * Or interfere with drones * The coordinates in the nth time slot N is a positive integer, and d0 is the near-field reference distance, which should be located in the far field of the antenna and is determined by measurements close to the transmitter.

[0036] Similarly, uplink and downlink communication between the UAV and all ground nodes (i.e., source node, destination node, and ground eavesdropper) adopts a typical free-space propagation model. Therefore, the power gain of the n-slot air-to-ground (A2G) channel is specifically calculated using the following formula:

[0037]

[0038] Among them, drone j includes relay drone j or jamming drone j, where j is denoted as Ground mobile node i includes source node, destination node, and ground eavesdropper, i∈{S} m D m,E},h j,i d represents the distance from UAV j to ground mobile node i and the channel gain. j,i ρ represents the distance from UAV j to ground mobile node i. ρ0 represents the channel power when the reference distance d0 = 1m, and q... j (n) represents the coordinates of a relay UAV j or jamming UAV j in the nth time slot, w i (n) represents the coordinates of the ground node in the nth time slot, and H represents the altitude at which UAV j flies. N is a positive integer.

[0039] P R P represents the transmit power of the relay UAV_R. J The transmission power of UAV_J is interfered with. In time slot n, relay UAV_R forwards information signals, interfering with UAV_J's transmission of jamming signals. Due to the limited real-time transmission power of the transmitter, P... R and P J Simultaneously constrained by both average power and maximum power, the following constraints are satisfied:

[0040] In the aforementioned UAV swarm-assisted multi-hop relay secure communication model, in time slot n, the relay UAV forwards the information signal, and the interfering UAV transmits the jamming signal. The transmission power P of the relay UAV... R and the transmission power P of the jamming drone J Simultaneously satisfying the constraint conditions, the specific constraint conditions are as follows:

[0041]

[0042]

[0043] in, Indicates relay drone R k The average transmit power, This represents the average transmit power of the jamming drone J. This indicates that at time n, the relay drone R... k Real-time transmit power, P J (n) represents the real-time transmission power of the interfering UAV J at time n. This indicates the maximum transmission power of the relay drone. This indicates the maximum transmit power of the jamming drone. N is a positive integer. Used to represent source node S m The transmit power. Set the source node S. m Its transmission power is much greater than the maximum transmission power of the relay drone. and the maximum transmission power of jamming drones

[0044] In the aforementioned UAV swarm-assisted multi-hop relay secure communication model, UAVs serve multiple pairs of source and destination nodes using user scheduling. The relay UAV swarm provides service to at most one pair of source and destination nodes in the nth time slot. User scheduling refers to the process of allocating resources for communication between multiple pairs of nodes within a time slot. The calculation of the source node S in time slot n... m Information transmission rate to R1 link Calculate the relay UAV R in time slot n. M To the destination node D m Information transmission rate Calculate the relay UAV R in time slot n. k To relay drone R k+1 Information transmission rate of the link Calculate the rate at which ground eavesdropper E intercepts information in time slot n. Specifically as follows:

[0045] Due to node mobility, mutual interference can occur when multiple pairs of nodes simultaneously use the same frequency band within a limited area. If each pair of nodes operates on different frequency bands, the UAV needs to carry more antennas, undoubtedly leading to greater energy consumption. Therefore, when serving multiple pairs of nodes, a user scheduling approach, i.e., time-division multiplexing, can be adopted. This reduces the number of antennas the UAV carries, thus lowering energy consumption. Specifically, the relay UAV can serve at most one pair of source and destination nodes in the nth time slot. User scheduling refers to the process of allocating resources for communication between multiple pairs of nodes within a time slot. Therefore, the main objective of user scheduling is to effectively select which pair of nodes communicates in the current time slot to maximize the minimum security rate.

[0046] In this embodiment of the invention, the relay drone R has sufficient buffer space to serve the uplink and downlink of a pair of nodes using frequency division multiplexing. Therefore, in time slot n, the source node S m , Information transmission rate to R1 link Its calculation formula can be expressed as:

[0047]

[0048]

[0049] Where, ζ m (n) represents the user scheduling coefficient in time slot n, meaning that the relay drone R can only serve at most one pair of nodes in one time slot. Used to represent source node S m The transmission power, It is the source node S m The channel gain between the relay UAV R1 and P J (n) represents the real-time transmission power of the interfering UAV J at time n. It is the channel gain between the interfering UAV J and the relay UAV R1, σ 2 This represents noise power, which is a constant. N is a positive integer.

[0050] Relay UAV R in time slot n M To the destination node D m , Information transmission rate The specific calculation formula is as follows:

[0051]

[0052] Where, ζ m (n) and ζ in equation (9) m (n) are the same, This indicates that at time n, the relay drone R... k Real-time transmit power, Indicates the relay drone R in the nth time slot K and the destination node D m The channel gain between P J (n) represents the real-time transmission power of the interfering UAV J at time n. This indicates the interference between the drone J and the target node D. m Channel gain between N is a positive integer, σ 2 This represents noise power and is a constant.

[0053] Similarly, in time slot n, the relay drone R k To relay drone R k+1 Information transmission rate of the link The specific calculation formula is as follows:

[0054]

[0055] in, This indicates that at time n, the relay drone R... k Real-time transmit power, Indicates the relay drone R in the nth time slot k and relay drone R k+1 The channel gain between P J (n) represents the real-time transmission power of the interfering UAV J at time n. This indicates the jamming drone J and the relay drone R.k+1 Channel gain between N is a positive integer, σ 2 This represents noise power and is a constant.

[0056] At time slot n, the information rate intercepted by ground eavesdropper E The specific calculation formula is as follows:

[0057]

[0058] in, This indicates that at time n, the relay drone R... k Real-time transmit power, It is a relay drone R k The channel gain between P and the ground eavesdropper E J (n) represents the real-time transmission power of the interfering UAV J at time n, h JE It interferes with the channel gain between the drone J and the ground eavesdropper E. N is a positive integer, σ 2 This represents noise power and is a constant.

[0059] In this embodiment of the invention, the processing duration of the relay UAV_R is one time slot, and it follows the information causality constraint of the relay UAV_R, only forwarding received data. Therefore, the information rate of the relay UAV_R in each time slot n should satisfy the constraint condition.

[0060] In the aforementioned UAV swarm-assisted multi-hop relay secure communication model, the information rate of the relay UAV in each time slot n should satisfy a constraint condition, which is specifically:

[0061]

[0062]

[0063]

[0064] in, This represents the sum of the information transmission rates of the link between relay UAV R1 and relay UAV R2 within the first n time slots. S represents the source node S within the first n-1 time slots. m , The sum of the information transmission rates of the relay drone R1 link, R represents the relay drone within the first n time slots. k To relay drone R k+1 The sum of the information transmission rates of the links, R represents the relay drone within the first n-1 time slots.k-1 To relay drone R k The sum of the information transmission rates of the links, R represents the relay drone during the first n time slots. k To the destination node D m , The sum of information transmission rates, where n is the time slot and N represents the total number of time slots, which is a positive integer.

[0065] Because this UAV multi-hop relay secure communication system selects decode-forward as the relay forwarding protocol, its end-to-end throughput is limited by the weakest link. Therefore, this embodiment of the invention uses the minimum security rate as an indicator to measure the security performance of this multi-UAV relay system with paired mobile nodes.

[0066] Furthermore, in the UAV swarm-assisted multi-hop relay secure communication model, the minimum security rate is used as an indicator to measure the security performance of the UAV swarm-assisted multi-hop relay secure communication model, and the relay UAV R is calculated. k To relay drone R k+1 The security rate of the link in time slot n Calculate the minimum security rate of the UAV swarm-assisted multi-hop relay model within one cycle.

[0067] The minimum confidentiality rate The calculation formula can be expressed as:

[0068]

[0069] in, n represents a time slot, and N represents the total number of time slots, which is a positive integer, indicating that time slot n belongs to the set of positive integers. R represents the relay drone within one period. k To relay drone R k+1 The security rate of the link, Represents the source node S within a period m Information transmission rate to the relay drone R1 link, Indicates the relay drone R in time slot n. k To the destination node D m , Information transmission rate Relay UAV R in time slot n k The information transmission rate to ground-based eavesdropper E R represents the relay drone within one period. k To the destination node D m The security rate of the link, [x] + =max(x,0).

[0070] Indicates relay drone R k To relay drone R k+1 The security rate of the link in time slot n can be calculated using the following formula:

[0071]

[0072] Based on the aforementioned UAV swarm-assisted multi-hop relay secure communication model and the constraints that the communication model should satisfy, a joint optimization model is constructed to achieve optimal resource allocation of the model, namely, optimal trajectory of the UAV swarm, optimal user scheduling, and optimal UAV swarm transmission power, under the condition that the minimum security rate of the communication model reaches its maximum.

[0073] Furthermore, the joint optimization model includes: the speed of each UAV. Collection of UAV flight angles UAV (Unmanned Aerial Vehicle) transmit power set Set of drone user scheduling coefficients The joint optimization model satisfies the following constraints:

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087] Wherein, function P is the joint optimization objective function of the joint optimization model, which represents the solution of the maximum UAV flight speed, flight angle, transmission power and optimal user scheduling coefficient under the condition of minimum security rate in one cycle of the UAV swarm-assisted multi-hop relay secure communication model;

[0088] C1 represents the speed limit for the relay drone swarm and the jamming drone J, where v j (n) represents the flight speed of UAV j in time slot n, v max This indicates the maximum flight speed of each drone. This indicates that drone j belongs to either the set of relay drones or the set of jamming drones;

[0089] C2 represents the flight angle limitations for the relay drone swarm and the jamming drone J, respectively, where deg j (n) represents the flight angle of UAV j in time slot n;

[0090] C3 is the maximum distance constraint that all UAVs can fly within a time slot, where q j (n+1) represents the coordinates of UAV j in the (n+1)th time slot, q j (n) represents the coordinates of UAV j in the nth time slot, L = τv max Indicates drone relay R k Or, the maximum length of the mobile jammer J flying at maximum speed within a time slot, where τ is the length of the unit time slot, v max This represents the maximum flight speed of each drone, n = 1, ..., N-1 represents the range of values ​​for time slot n, and N represents the total number of time slots, which is a positive integer. This indicates that drone j belongs to either the set of relay drones or the set of jamming drones;

[0091] C4 represents the initial and final position constraints for all unmanned aerial vehicles (UAVs), where, q represents the initial position of drone j. j (1) represents the coordinates of UAV j in the first time slot. This indicates the final location that drone j is scheduled to occupy. This indicates that drone j belongs to either the set of relay drones or the set of jamming drones;

[0092] C5 is the minimum distance maintained between all drones, where d min Indicates the minimum safe distance between drones. This indicates that drone i or j belongs to the set of relay drones or the set of jamming drones, and drone i or drone j is not the same drone.

[0093] C6 represents the transmit power constraint for the relay drone swarm, where, This indicates that at time n, the relay drone R... k Real-time transmit power, Indicates relay drone R k The average transmit power, This indicates the maximum transmit power of the relay drone, and the drone relay R... k , Where K is a positive integer; C7 represents the transmit power constraint of the jamming UAV, where, P represents the average transmit power of the jamming drone J. J (n) represents the real-time transmission power of the interfering UAV J at time n. This indicates the maximum transmit power of the jamming drone;

[0094] C8 is the user scheduling constraint, where ζ m (n) represents the user scheduling coefficient in time slot n, ζ m (n)∈{0,1} indicates that the relay drone can only serve at most one pair of nodes in a time slot. M represents the number of source nodes and destination nodes, which are called users;

[0095] C9-C11 represent relay causal information constraints, meaning only received data can be forwarded. This represents the sum of the information transmission rates of the link between relay UAV R1 and relay UAV R2 within the first n time slots. S represents the source node S within the first n-1 time slots. m The sum of the information transmission rates of the relay drone R1 link, R represents the relay drone within the first n time slots. k To relay drone R k+1 The sum of the information transmission rates of the links, R represents the relay drone within the first n-1 time slots. k-1 To relay drone R k The sum of the information transmission rates of the links, R represents the relay drone during the first n time slots. k To the destination node D m The sum of information transmission rates, where n is the time slot, N represents the total number of time slots (a positive integer); C12 represents the flight boundary constraints of the relay UAV swarm and the jamming UAVs, where x j (n) represents the x-coordinate of UAV j in time slot n, y j (n) represents the ordinate of UAV j in time slot n, l represents the length of the region, and w represents the width of the region.

[0096] Analysis shows that the joint optimization objective function P is a non-convex problem with multiple constraints, making it difficult to solve using traditional convex optimization methods. Therefore, this invention employs a deep reinforcement learning algorithm that can interact with time-varying environments to obtain the optimal solution. The joint optimization objective function P is transformed into a Markov decision process. The triplet representation of the Markov decision process is as follows:

[0097] The state space includes: the relay UAV R1 from the source node S during the first n-1 time slots. m Total of received information Relay UAV R within the first n time slots k Forwarded to relay drone R k+1 Information sum Within the first n time slots, from relay drone R K Forward to destination node D m Information sum The average transmit power of jamming drones and relay drone swarms in the first n time slots s n Let s∈S represent any state in the state space of the joint optimization objective function, where s is the current state. n Specifically, it is expressed as follows:

[0098]

[0099] in, This indicates that the relay drone R1 started from the source node S within the first n-1 time slots. m The sum of the received information, R represents the relay drone within the first n time slots. k Forwarded to relay drone R k+1 The sum of information, This indicates that within the first n time slots, the relay drone R... K Forward to destination node D m The sum of information, This represents the average transmit power of the jamming drone and the relay drone swarm over the first n time slots, [x j (n),y j (n)] T , This indicates that drone j belongs to either a jamming drone or a relay drone swarm.

[0100] In the joint optimization objective function P, the drone swarm (including relay drone swarms and jamming drones) is a super intelligent agent that can select a series of actions based on the state.

[0101] The action space consists of the flight angles (deg) of the relay drone swarm and the jamming drones. j(n), Transmission power P j (n), flight speed v j (n) and user scheduling coefficient ζ m Composed of (n), where The action space is defined as

[0102]

[0103] Instant Rewards: Instant Rewards r n It is the sum of the minimum security rate and penalty for the UAV swarm-assisted multi-hop relay secure communication model in time slot n, specifically expressed as:

[0104]

[0105] in,

[0106] R secmin (n) is the minimum security rate of the system in time slot n, which is expressed as:

[0107]

[0108] The penalty term, represented by formulas C6-C7, is specifically expressed as follows to satisfy the transmit power constraints of relay UAVs and jamming UAVs:

[0109]

[0110] In formula (23), c1 is a constant. This represents the average transmit power of the jamming drone and the relay drone swarm over the first n time slots. This indicates the maximum average transmit power of jamming drones and relay drone swarms. The penalty term in C12, representing the relay drone swarm and the interference drones, guarantees that the flight range constraints are met, and is specifically expressed as follows:

[0111]

[0112] In formula (24), c2 is a constant, x j (n)∈[0,l] represents the range of the x-coordinates of the interfering drones and the relay drone swarm from 0 to l, where l represents the length of the region, y j (n)∈[0,w] represents the range of the ordinates of the interfering drones and the relay drone swarm from 0 to w, where w represents the width of the region. The penalty terms, represented by C9-C11, ensure that the relay drone swarm satisfies the information causality constraint, specifically expressed as follows:

[0113]

[0114] In formula (25), c3 is a constant. This indicates that the relay drone R1 started from the source node S within the first n-1 time slots. m The sum of the received information, R represents the relay drone within the first n time slots. k Forwarded to relay drone R k+1 The sum of information, representing Within the first n time slots, from relay drone R K Forward to destination node D m The sum of information. The penalty term for C5, in order to prevent the drone from colliding, is specifically expressed as follows:

[0115]

[0116] In formula (25), c4 is a constant, and q j (n) represents the location of the interfering drone j or the relay drone j. q represents the minimum safe distance (constant) between drone j and drone i. i (n) represents the location of another interfering drone i or relay drone i besides j.

[0117] Furthermore, the actor dual network, the critic dual network, and the experience replay buffer are described. The actor dual network includes a policy network and a policy objective network with the same structure. The critic dual network includes a value network and a value objective network with the same structure. The actions and states in the actor dual network correspond to the action space representation and state space representation of the Markov decision process, respectively. The immediate reward in the critic dual network corresponds to the immediate reward representation of the Markov decision process.

[0118] The Deep Deterministic Policy Gradient (DDPG) algorithm is a model-free off-track policy algorithm based on Deep Neural Networks (DNNs) that can learn policies in a continuous action space. This algorithm combines a memory for random sample generation and employs an actor-critic dual network—a new network for estimation and an older network with a delay—to address network correlations and improve neural network convergence. In this algorithm, the policy network acts as a policy function, generating actions; the policy target network generates target actions to better guide policy network updates; the value network, a Q-value function, acts as a critic, evaluating the actor's performance and guiding subsequent actions; and the value target network, a copy of the value network, stably guides policy network updates.

[0119] In this embodiment of the invention, the drone swarm is used as the super agent based on the deep deterministic policy gradient algorithm architecture. The initial state s1 is obtained based on the position information of the source node, the destination node and the drone swarm. The initial state s1 is input into the policy network to generate an action. The value network evaluates the performance of the action and guides its subsequent actions.

[0120] The policy network comprises an input layer, two hidden layers, and an output layer. Specifically: the input layer receives state information as input; the two hidden layers consist of 400 neurons in the first hidden layer and 30 neurons in the second hidden layer; the output layer generates actions, which are constrained to ensure they occur within a suitable action space. The policy network is represented as μ(s|θ). μ (i.e., policy function), where s represents the state, θ μ These are the parameters of the policy network.

[0121] The structure of the policy target network μ' is the same as that of the policy network, but the parameters are different. The parameters of the policy target network μ' are θ. μ' It is used to calculate the target action.

[0122] The value network comprises one input layer, two hidden layers, and one output layer. Specifically: The input layer receives state and action information as input; the two hidden layers consist of 400 neurons in the first hidden layer and 30 neurons in the second hidden layer; the output layer outputs the state-action value (i.e., the Q-value). The value network Q(s,a|θ) Q (i.e., the Q-value function), where s represents the state, a represents the action, and θ Q These are the parameters of the value network.

[0123] The structure of the value target network Q' is the same as that of the value network, but the parameters are different. The parameters of the value target network are θ. Q' Used to calculate the target state-action value.

[0124] Experience replay buffer: This is equivalent to the memory of the algorithm. The experience gained from training is stored in this buffer and then sampled in small batches.

[0125] The value network Q(s,a|θ) Q The loss function L(θ) can be updated via backpropagation. Q Specifically, it is expressed as:

[0126]

[0127] Where, L(θ) Q θ represents the loss function of the value network. Q The parameter s represents the value network.j Indicates the state of the j-th time slot, a j Let Q(s) represent the action in the j-th time slot. j ,a j |θ Q ) indicates that in state s j Next, execute action a. j The obtained Q value, where H represents the sampling batch size;

[0128] y j The expected value target, or Q value, is typically determined by the reward corresponding to the current state and action, as well as the expected value target of subsequent states, i.e., the Q value. j Calculated through the value target network, it is specifically expressed as follows:

[0129] y j =r j +γQ'(s j+1 ,μ'(s j+1 |θ μ' )|θ Q' (28)

[0130] Where, r j Let γ represent the reward in the j-th time slot, and γ represent the discount rate, a hyperparameter used to balance the importance of current reward and expected future reward. γ is used to measure the impact of future reward on the current decision. j+1 Represents the state of the j-th time slot, μ'(s) j+1 |θ μ' ) indicates that in state s j+1 Next, based on the policy determined by the policy network, i.e., the selected action, θ Q 'Represents the parameters of the value target network.

[0131] In the Deep Deterministic Policy Gradient Algorithm (DDPG), the loss function L(θ) Q It is mainly used to measure the prediction and value target value (i.e., y) of the current Q value (action value function). i The loss function of DDPG is the prediction error of the Q-value as a function of the mean squared error. By minimizing the loss function, the network adjusts its parameters to make the predicted Q-value as close as possible to the target Q-value. Specifically, the loss function can be expressed as the difference between the predicted Q-value and the target Q-value. The target Q-value is typically calculated using the Bellman equation and is used to guide the network's training and updates. By optimizing the loss function, the network can progressively improve its estimation of the Q-value, thereby improving the decision-making ability of the agent (i.e., the drone swarm).

[0132] In the Deep Deterministic Policy Gradient (DDPG) algorithm, the policy network learns and outputs a deterministic policy, i.e., a mapping from state to action. The DDPG algorithm uses the chain rule (a rule for calculating the derivative of a composite function) to compute the gradient of the policy function with respect to the network parameters. These gradients are then used to update the network parameters to improve the performance of the policy function and maximize the expected reward. The goal of the policy network is to output the optimal action, and the policy gradient is the tool used to update the network parameters. The gradient is a vector of all partial derivatives of a function at a given point; it tells the network in which direction to adjust the parameters to maximize the expected reward.

[0133] The policy gradient refers to the gradient used when optimizing a policy (or policy network). It is a gradient with respect to the policy parameters, informing how to adjust the policy network's parameters to increase the expected reward under the current policy. Policy gradient methods maximize expected reward by directly parameterizing the policy and using gradient ascent. This increases the probability of choosing a particular action in a given state, thereby improving long-term cumulative rewards. The policy gradient guides the policy network to generate better action policies, while the value gradient updates the value network to better estimate the Q-value function. Both work together to improve the estimation of the policy and value function, resulting in better decision-making and behavioral policies.

[0134] Value gradients involve the gradient of the value function (Q-value function). The value function measures the long-term reward of taking an action in a given state. Value gradients can guide an agent to adjust its policy so that it chooses actions that produce higher long-term rewards in a given state. In DDPG, value gradients are used to update the parameters of the value network to better estimate the Q-values ​​of state-action pairs.

[0135] The gradient of the policy function with respect to the policy parameters is calculated using the chain rule, and its specific expression is as follows:

[0136]

[0137] J represents the objective function of the policy function. The objective function J represents the policy parameter θ. μ The gradient, where H represents the mini-batch sample size. The parameter θ represents the policy network. μ Find the gradient. Indicates that in state s j Below, the target value obtained by performing action a is the Q value, where a = μ(s) j |θ μ ), μ(s j |θ μ ) indicates that in state s j Next, based on the policy determined by the policy network, i.e., the selected action, ▽a This represents the gradient of action a, μ(s|θ) μ ) represents the action selected in state s according to the policy determined by the policy network.

[0138] like Figure 3 As shown, the DDPG algorithm model uses two different deep neural networks (DNNs) to approximate the policy function and the Q-value function. The actor dual network includes an optimizer and a current network (i.e., the policy network μ(s|θ)). μ The critical dual network consists of the optimizer and the current network (i.e., the value network Q(s,a|θ)). Q )) and the value target network Q'.

[0139] Furthermore, the entire training process based on the Deep Deterministic Policy Gradient Algorithm (DDPG) includes:

[0140] S1: Initialize network parameters, initialize the experience replay buffer β, batch size, training epoch length, and training sample length. Based on the location information of the source node, destination node, and UAV swarm, obtain the initial state s1; input the initial state into the policy network μ(s|θ). μ After one training step, the output is μ(s). n |θ μ By adding behavioral noise ξ n To construct the action space to obtain action a n =μ(s) n |θ μ )+ξ n The action a n The corresponding action space representation of a Markov decision process, where ξ n Follows a Gaussian distribution μ e This is the average value. That is the standard deviation.

[0141] S2: Execute action a in the DDPG algorithm environment n Then, the drone swarm, acting as a super-intelligent agent, can observe the next state s. n+1 and instant rewards n The state s n+1 and instant rewards n Corresponding to the state-space representation and immediate reward representation of a Markov decision process, the tuple (s) n ,a n ,r n ,s n+1 ) are stored in the experience replay buffer. When the experience replay buffer is full, H tuples (s) are randomly selected.j ,a j ,r j ,s j+1 A small batch is formed in the experience replay buffer, that is, a small batch sampling is performed to obtain a small batch sample.

[0142] S3: Input the mini-batch samples into the policy network μ(s|θ) μ ) and value network Q(s,a|θ Q After training, the policy objective network μ' outputs policy behavior μ'(s). j+1 θ μ' The mini-batch samples and the policy action μ'(s) are then used to construct the value target network Q'; within the value target network Q', the mini-batch samples and the policy action μ'(s) are used to construct the value target network Q'. j+1 |θ μ' ), calculate the target value y j ; Regarding the loss function with respect to parameter θ Q Gradient descent is used to update the parameters of the value network, which are updated by the optimizer of the given value network.

[0143] S4: Iteratively update the policy network: Use the chain rule to calculate the gradient of the policy function with respect to the network parameters, and then use these gradients to update the parameters of the policy network to improve the performance of the policy function. Specifically, it is expressed as follows:

[0144] actor network μ(s|θ) μ ) will perform small batch actions a = μ(s) j |θ μ The value gradient (Q-value gradient) of action a in state s is sent to the value network to obtain the value gradient of action a in state s. The policy gradient of the policy network and the aforementioned value gradient Network parameters in the policy network can be updated, specifically as follows:

[0145]

[0146] in, The objective function J of the policy function is expressed with respect to the policy parameters θ. μ gradient, Indicates that in state s j The Q value obtained by performing action a, where a = μ(s) j |θ μ ), μ(s j |θ μ ) indicates that in state s j Next, based on the policy determined by the policy network, i.e., the selected action, This indicates that the gradient of action a is calculated. The parameter θ represents the policy network.μ Find the gradient.

[0147] S5: The DDPG agent uses a soft update method to update the policy objective network μ'(s|θ). μ' ) and value target network Q'(s,a|θ Q' ):

[0148]

[0149]

[0150] in, It is a soft update coefficient;

[0151] In this embodiment of the invention, the DDPG agent refers to a swarm of unmanned aerial vehicles (UAVs).

[0152] S6: Iterative training to obtain the optimal action and maximum reward of the UAV swarm, specifically including: the optimal action is the optimal resource allocation of the UAV multi-hop relay secure communication system, namely the optimal trajectory of the UAV swarm, the optimal user scheduling, and the optimal UAV swarm transmission power, wherein the optimal trajectory includes the optimal flight speed and flight angle; the maximum reward is the minimum security rate that the UAV swarm assists the multi-hop relay secure communication system to achieve.

[0153] The software implementation process of the DDPG algorithm is specifically represented as follows:

[0154] Step 1: Initialize the experience replay buffer and sampling batch size, specifically including:

[0155] Initialize the experience playback buffer β, with a designed capacity of B;

[0156] Initialize the sampling batch size to H;

[0157] Step 2: Initialize network parameters, specifically including:

[0158] Randomly initialize the parameters θ of the policy network μ ;

[0159] Randomly initialize the parameters θ of the value network Q ;

[0160] Copy the parameters of the value network to the parameters of its corresponding value target network Q': θ Q' ←θ Q ;

[0161] Copy the parameters of the policy network to the parameters of its corresponding policy target network μ': θ μ' ←θ μ ;

[0162] Step 3: Initialize the training round length and training sample length, specifically including:

[0163] The initial training round length is set to I;

[0164] The number of initial training samples is equal to the total number of time slots in one cycle, defined as N;

[0165] Step 4: Perform I training rounds. Each round includes:

[0166] Initialize the behavioral noise ξ;

[0167] Initialize the location information of the user and the drone swarm to obtain the initial state s1;

[0168] The execution process for each time step n is as follows, where n = 1, 2, ..., N:

[0169] Based on the current state s n According to behavioral noise ξ n and policy network θ μ Get the current action a n =μ(s) n |θ μ )+ξ n ;

[0170] The drone is performing the current action a n Receive reward r n and the next state s n+1 ;

[0171] Determine whether the drone violated position and collision constraints after performing the action. If it did, recalibrate the drone's position, that is, return the drone to a position that satisfies the constraints.

[0172] Determine if the experience playback buffer is full. If not, then process the obtained transformation sequence (s). n ,a n ,r n ,s n+1 ) is stored in β;

[0173] If it is full, the resulting transformation sequence (s) n ,a n ,r n ,s n+1 ) Randomly replace one set of transformation sequences in β, and randomly draw H sets of transformation sequences (s) from β. j ,a j ,r j ,s j+1 A small batch is formed and used as training data for the network.

[0174] This small batch is then processed into smaller batches, and the network parameters are updated, as follows:

[0175] The policy target network μ' will take action μ'(s) j+1 |θ μ' The input is fed into the value target network. Mini-batch processing and μ'(s) are then used. j+1 |θ μ' The value target network can calculate the target value y. j =r j +γQ'(s j+1 ,μ'(s j+1 |θ μ' )|θ Q' ).

[0176] For loss function Regarding the parameter θ Q Gradient descent is used to update the parameters θ of the value network. Q This update will be performed by the given optimizer.

[0177] The policy network μ will generate mini-batch actions a = μ(s) j |θ μ Send it to the value network to realize the gradient of action a. parameter Derived from the given optimizer. Using these two gradients, update the policy network as follows:

[0178]

[0179] The policy target network and value target network are updated using a soft update method, as follows:

[0180]

[0181] Step 5: After the cycle ends, the optimal action and maximum reward will be obtained, including:

[0182] Optimal action: also known as optimal resource allocation, which includes the optimal trajectory of the drone swarm, optimal user scheduling, and optimal drone swarm transmission power;

[0183] Maximum reward: The drone swarm-assisted multi-hop relay secure communication system achieves the maximum minimum security rate.

[0184] The software environment used was PyCharm as the primary code editing, running, and debugging environment, and TensorFlow as the main deep learning framework. The DDPG model built in PyCharm using TensorFlow was trained and tested in a UAV multi-hop relay secure communication system environment. Simulation results are as follows: Figure 4 As shown, Figure 4 In the diagram: r&j represents the minimum security rate of the resource allocation scheme based on DDPG proposed in this invention with a period of 100 seconds; r&j-np represents the minimum security rate of the scheme based on DDPG that simultaneously optimizes the trajectories of relay UAV swarms and jamming UAVs, as well as the transmission power of the relay UAV swarm, with a fixed transmission power of the jamming UAV; r&j-nt represents the minimum security rate of the scheme based on DDPG that simultaneously optimizes the transmission power of relay UAV swarms and jamming UAVs, as well as the trajectory of the jamming UAV, with a fixed trajectory of the jamming UAV; r represents the minimum security rate of the scheme based on DDPG that simultaneously optimizes the trajectory and power of the relay UAV swarm, without considering the introduction of jamming UAVs, with a period of 100 seconds.

[0185] As can be seen from the simulation diagram, the minimum security rate of the system in all schemes gradually increases with time. However, the minimum security rate of the system in r&j, that is, the resource allocation scheme based on DDPG proposed in this invention, is the highest, which proves the feasibility and effectiveness of this invention.

[0186] The beneficial effects of this invention are as follows: First, this invention constructs a model of a UAV swarm-assisted multi-hop relay secure communication system. This model includes potential eavesdroppers, mobile users (including target nodes and source nodes), UAV relays, and jamming UAVs, essentially constituting a UAV swarm-assisted multi-hop relay secure communication system model based on cooperative interference. This model constructs reliable communication links for long-distance mobile nodes or long-distance communication with obstacles, ensuring secure communication in the UAV swarm-assisted multi-hop relay secure communication system and solving the information security problem when long-distance communication is blocked. Second, with the goal of maximizing the minimum security rate of the system, a joint optimization model of the UAV swarm's flight trajectory, transmission power, and user scheduling is established. Then, based on this joint optimization model, a Deep Deterministic Policy Gradient (DDPG) architecture is constructed using Markov decision processes. Finally, through the DDPG architecture, the optimal resource allocation of the UAV swarm-assisted multi-hop relay secure communication system model is achieved, namely, the optimal trajectory of the UAV swarm, the optimal user scheduling, and the optimal transmission power of the UAV swarm, enabling the UAV swarm-assisted multi-hop relay secure communication system to reach the maximum minimum security rate.

[0187] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0188] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A resource allocation method for a multi-hop relay secure communication system for unmanned aerial vehicles (UAVs), characterized in that, include: Construct a drone swarm-assisted multi-hop relay secure communication model, which includes: source node , destination node Several drone relays Ground eavesdropper E and mobile jammer J, the source node via several drone relays Send information to the destination node The location of the ground eavesdropper E is unknown, and the drone relay... The mobile jammer J transmits jamming signals in order to forward information signals. Based on a UAV swarm-assisted multi-hop relay secure communication model, a joint optimization model is established. In this model, maximizing the minimum security rate within a cycle is the joint optimization objective function. The model solves for UAV flight speed, flight angle, transmission power, and user scheduling coefficients. The joint optimization model includes the speed of each UAV. A collection of drone flight angles The collection of drone's transmission power The set of drone user scheduling coefficients ; The joint optimization model satisfies the constraints, specifically expressed as follows: Wherein, function P is the joint optimization objective function of the joint optimization model, which represents the solution of the optimal UAV flight speed, flight angle, transmission power and optimal user scheduling coefficient under the condition of maximizing the minimum security rate within a period of the UAV swarm-assisted multi-hop relay secure communication model; C1 represents the speed limit for the relay drone swarm and the jamming drone J, where, This represents the flight speed of UAV j in time slot n. This indicates the maximum flight speed of each drone. This indicates that drone j belongs to either the set of relay drones or the set of jamming drones; C2 represents the flight angle restrictions for the relay drone swarm and the jamming drone J, respectively. This represents the flight angle of UAV j in time slot n; C3 is the maximum distance constraint that all UAVs can fly within a time slot, where... Indicates that drone j is in the... Coordinates on each time slot Indicates that drone j is in the... Coordinates on each time slot Indicates drone relay Or the maximum length that mobile jammer J can fly at maximum speed within a time slot, It is the length of the unit time slot. This indicates the maximum flight speed of each drone. The value range of time slot n is represented by N, which is the total number of time slots and is a positive integer. This indicates that drone j belongs to either the set of relay drones or the set of jamming drones; C4 represents the initial and final position constraints for all unmanned aerial vehicles (UAVs), where, Indicates drone The predetermined initial position, Indicates drone Coordinates in the first time slot Indicates drone The final location was originally planned. This indicates that drone j belongs to either the set of relay drones or the set of jamming drones; C5 is the minimum distance maintained between all drones, where, Indicates the minimum safe distance between drones. This indicates that drone i or drone j belongs to a set of relay drones or a set of jamming drones, and that drone i or drone j is not the same drone. C6 represents the transmit power constraint for the relay drone swarm, where, This indicates that at time n, the relay drone Real-time transmit power, Indicates relay drone The average transmit power, This indicates the maximum transmission power of the relay drone; drone relay. , , where K is a positive integer; C7 represents the transmit power constraint for the jamming drone, where, This represents the average transmit power of the jamming drone J. This represents the real-time transmission power of the interfering UAV J at time n. This indicates the maximum transmit power of the jamming drone; C8 is the user scheduling constraint, where... Indicates in User scheduling coefficients during time slots This means that a relay drone can only serve at most one pair of nodes within a time slot. M represents the number of source nodes and destination nodes, which are called users; C9-C11 represent relay causal information constraints, meaning only received data can be forwarded. This represents the sum of the information transmission rates of the link between relay UAV R1 and relay UAV R2 within the first n time slots. Indicates the source node within the first n-1 time slots To relay drone The sum of the information transmission rates of the links This indicates the relay drone within the first n time slots. To relay drone The sum of the information transmission rates of the links This indicates the relay drone within the first n-1 time slots. To relay drone The sum of the information transmission rates of the links Relay drones in the first n time slots to the destination node The sum of information transmission rates, where n is the time slot and N represents the total number of time slots, which is a positive integer; C12 represents the flight boundary constraints for relay drone swarms and jamming drones, where, This represents the x-coordinate of UAV j in time slot n. This represents the ordinate of UAV j in time slot n. Represents the length of the region. Represents the width of the area; Based on the joint optimization model, a deep reinforcement learning architecture for a multi-hop relay secure communication network for unmanned aerial vehicles is constructed. The deep reinforcement learning architecture adopts a deep deterministic policy gradient algorithm architecture. Using the aforementioned deep deterministic policy gradient algorithm architecture, deep reinforcement learning training is performed to obtain the optimal action and maximum reward of the UAV swarm. The optimal action is the optimal resource allocation of the UAV multi-hop relay secure communication system, namely the optimal trajectory, optimal user scheduling, and optimal UAV swarm transmission power. The maximum reward is the maximum minimum security rate achieved by the UAV swarm-assisted multi-hop relay secure communication model.

2. The resource allocation method for a UAV multi-hop relay secure communication system according to claim 1, characterized in that, In the UAV swarm-assisted multi-hop relay secure communication model, the UAV trajectory is divided into lines connecting the positions of the UAVs in each time slot throughout the entire flight cycle. The position of each UAV in adjacent time slots satisfies constraints. In time slot n, the relay UAV forwards information signals, and the interfering UAV transmits jamming signals. The transmission power of the relay UAV... and jamming the transmission power of drones Simultaneously satisfying the constraints; the UAV uses user scheduling to serve multiple pairs of source and destination nodes, and the relay UAV in the [missing information]... Each time slot can provide services for a maximum of one pair of source and destination nodes; Calculation in Time slot source node arrive Information transmission rate of the link Calculated in Time-slot relay drones to the destination node Information transmission rate Calculated in Time-slot relay drones To relay drone Information transmission rate of the link Calculated in The rate at which ground eavesdropper E steals information during time slots Each time slot The information rate of the relay drone meets the constraints.

3. The resource allocation method for a UAV multi-hop relay secure communication system according to claim 2, characterized in that, In the UAV swarm-assisted multi-hop relay communication model, the minimum security rate is used as an indicator to measure the security performance of the UAV swarm-assisted multi-hop relay model. The minimum security rate of the UAV swarm-assisted multi-hop relay model system within one cycle is calculated. The minimum confidentiality rate The specific formula for calculation is as follows: in, Here, n represents a time slot, and N represents the total number of time slots, which is a positive integer, indicating that time slot n belongs to the set of positive integers. Relay drones within a period To relay drone The security rate of the link, Represents the source node within a period To relay drone Information transmission rate of the link This indicates that the relay drone is in time slot n. to the destination node , Information transmission rate Relay UAV in time slot n The information transmission rate to ground-based eavesdropper E Relay drones within a period to the destination node Link security rate .

4. The resource allocation method for a UAV multi-hop relay secure communication system according to claim 1, characterized in that, The actor dual network, critic dual network, and experience replay buffer are defined as follows: the actor dual network includes a policy network and a policy objective network with the same structure; the critic dual network includes a value network and a value objective network with the same structure; the actions and states in the actor dual network correspond to the action space representation and state space representation of the Markov decision process, respectively; and the immediate reward in the critic dual network corresponds to the immediate reward representation of the Markov decision process. The value network The loss function can be updated via backpropagation. Specifically, it is expressed as follows: in, The loss function represents the value network. Parameters representing the value network, Indicates the first The state of each time slot Indicates the first Actions in a time slot Indicates the state Next, execute the action. The results obtained value, Indicates the sample batch size; The expected value target, or Q value, is typically determined by the reward corresponding to the current state and action, as well as the expected value target of subsequent states, i.e., the Q value. Calculated through the value target network, specifically expressed as: in, Indicates the first Rewards for each time slot, The discount rate is a hyperparameter used to balance the importance of current rewards and expected future rewards. Indicates the first The state of each time slot Indicates the state Next, based on the policy determined by the policy network, i.e., the selected action, Parameters representing the value target network; The gradient of the policy function with respect to the policy parameters is calculated using the chain rule, and its specific expression is as follows: The objective function of the policy function is represented by the objective function. The objective function J represents the policy parameters. gradient, Indicates the sample batch size. Represents the parameters of the policy network Find the gradient. Indicates the state Next, execute the action. The obtained value target value is Value, of which , Indicates the state Next, based on the policy determined by the policy network, i.e., the selected action, Indicates the action Find the gradient. Indicates the state The next step is to select an action based on the policy determined by the policy network.

5. The resource allocation method for a UAV multi-hop relay secure communication system according to claim 4, characterized in that, The training process based on the deep deterministic policy gradient algorithm architecture includes: S1: Initialize network parameters and initialize the experience playback buffer. The initial state is determined by setting the sampling batch size, initial training round length, and training sample length, based on the location information of the source node, destination node, and UAV swarm. Input the initial state into the policy network. After one training step, the output is... By adding behavioral noise To construct the action space in order to obtain the action This action The corresponding action space representation of a Markov decision process, where Follows Gaussian distribution , This is the average value. It is the standard deviation; S2: Execute actions in an environment based on a deep deterministic policy gradient architecture. Then, the drone swarm, acting as a super-intelligent agent, observes the next state. and instant rewards The state and instant rewards Corresponding to the state-space representation and immediate reward representation of a Markov decision process, the tuples Stored in the experience playback buffer When the experience replay buffer is full, H tuples are randomly selected. A small batch is formed in the experience replay buffer, and small batch sampling is performed to obtain a small batch sample. S3: Input the mini-batch samples into the policy network and value network Training the policy-target network Output strategy behavior To Value Target Network In the value target network In this process, the small batch of samples and the policy behavior are utilized. Calculate the target value Loss function for value network Regarding parameters Perform gradient descent to update the parameters of the value network; S4: Iteratively update the policy network: Use the chain rule to calculate the gradient of the policy function with respect to the network parameters, and use the gradient to update the parameters of the policy network. The specific process is as follows: Policy Network small batch actions Send it to the value network to obtain the state. Next action value gradient Policy gradients derived using the optimizer of the policy network. and the value gradient The network parameters in the updated policy network are specifically represented as follows: in, The value objective function J represents the value with respect to the policy parameters. gradient, Indicates the state Next, execute the action. The results obtained Value, of which , Indicates the state Next, based on the policy determined by the policy network, that is, the selected action; S5: The target network of the drone swarm soft update strategy and value target network : in, It is a soft update coefficient; S6: Iterative training to obtain the optimal action and maximum reward of the UAV swarm, specifically including: the optimal action is the optimal resource allocation of the UAV multi-hop relay secure communication system, namely the optimal trajectory of the UAV swarm, the optimal user scheduling, and the optimal UAV swarm transmission power, wherein the optimal trajectory includes the optimal flight speed and flight angle; the maximum reward is the minimum security rate that the UAV swarm assists the multi-hop relay communication system to achieve.

Images