A high-energy-efficiency resource management method for an unmanned aerial vehicle-assisted space-air-ground-sea communication network under jamming attack

Abstract

The application discloses a high-energy-efficiency resource management method for an unmanned aerial vehicle (UAV) assisted space-air-ground-sea communication network under interference attack, belongs to the fields of UAV communication, space-air-ground-sea integrated network, UAV trajectory optimization, anti-interference secure communication and the like, combines important information data such as a current flight position, residual energy and user channel state of the UAV, considers multiple constraints such as secure communication requirements in the space-air-ground-sea integrated network, a UAV flight angle and UAV energy, and proposes a UAV energy-saving and resource scheduling scheme based on a deep reinforcement learning and a double-delay deep deterministic policy gradient algorithm, so as to balance the energy efficiency of the UAV and a user fairness index to the greatest extent. Under the premise of ensuring the anti-interference effect of the UAV, the energy efficiency of the UAV is improved, and the fairness of the user is increased.

Description

Technical Field

[0001] This invention belongs to the fields of UAV communication, integrated air-space-ground-sea network, UAV trajectory optimization, and anti-interference secure communication. Specifically, it relates to a high-energy-efficiency resource management method for UAV-assisted air-space-ground-sea communication network under interference attacks in an integrated air-space-ground-sea network based on deep reinforcement learning. Background Technology

[0002] Drones, with their low-latency, high-bandwidth data transmission capabilities, play a crucial role in the integrated air-space-ground-sea network emerging alongside sixth-generation communication. Their provision of network connectivity and assistance in disaster response and emergency communications further solidifies their indispensable role in this integrated network. However, the line-of-sight (Line-of-Sight) nature of drones makes their communications more susceptible to interference. In severely disrupted environments, user fairness is a pressing issue. Furthermore, drone endurance remains a significant constraint on drone services. Therefore, intelligent energy conservation and resource management for drones in an integrated air-space-ground-sea network under interference attacks is a particularly critical challenge.

[0003] Unmanned aerial vehicle (UAV) trajectory optimization is a highly effective anti-jamming strategy. Existing technology [Y.Wu, W.Yang, X.Guanand Q.Wu, Energy-Efficient Trajectory Design for UAV-Enabled Communication Under Malicious Jamming, IEEE Wireless Communications Letters, vol.10, no.2, pp.206-210, Feb.2021.] proposes an iterative algorithm based on continuous convex approximation and Tinkelbach's algorithm to optimize UAV trajectories to maximize UAV energy efficiency, achieve a better balance between throughput and energy consumption, and improve system safety performance.

[0004] Deep reinforcement learning demonstrates the ability to adapt to and learn from the environment through interaction, effectively solving complex problems and challenges even when environmental information is unknown. Chinese patent application CN116820126A employs a deep reinforcement learning scheme based on multi-UAV path planning to improve the robustness of UAV path planning. Chinese patent application CN116828539A proposes a joint computational transfer and UAV trajectory optimization method based on deep reinforcement learning. Using a multi-agent, dual-delay deep deterministic policy gradient algorithm, it effectively solves the joint optimization problem of computational transfer and UAV trajectory, exhibiting stable performance in the optimization of a series of continuous action spaces.

[0005] Furthermore, improving UAV energy efficiency and user fairness in interference-prone UAV communication environments is a topic of close research interest. H. Yang et al. [H. Yang and X. Xie, Energy-Efficient Joint Scheduling and Resource Management for UAV-Enabled Multicell Networks, IEEE Systems Journal, vol. 14, no. 1, pp. 363-374, March 2020.] discussed energy-efficient joint scheduling and resource management in multi-cell networks supporting UAVs to maximize network energy efficiency. Chinese patent application CN115616906A proposes a reinforcement learning-based method for UAV trajectory optimization and user association optimization to achieve user fairness and improve the throughput and long-term energy efficiency of the communication system. Summary of the Invention

[0006] The purpose of this invention is to utilize deep reinforcement learning algorithms, based on the idea of ​​UAV trajectory optimization, to jointly improve the energy efficiency and user fairness coefficient of UAVs in the integrated air-space-ground-sea network, thereby improving the anti-interference performance of the integrated air-space-ground-sea communication and providing a high-energy-efficiency resource management method for UAV-assisted air-space-ground-sea communication networks under interference attacks.

[0007] This invention includes the following steps:

[0008] Step 1: Set up a system with K users, including ground users and maritime users, all randomly distributed on a plane at zero altitude. The jammer's position is fixed. Divide the total UAV flight time into N time slots at equal intervals. In each time slot, the UAV flies to the next location, collects user data, and transmits it to the satellite. Both of these actions consume the UAV's energy. The UAV adjusts its flight trajectory to counter interference attacks in the wireless communication network. The UAV's flight angle and flight distance constitute its current action. For the action space; Step 2: Construct three main neural networks: neural network A, neural network S, and neural network V, with network parameters θ, φ1, and φ2, respectively. Neural network A is responsible for learning and outputting the action policy, while neural networks S and V are used to estimate the action value function. When calculating the target value, the smaller value between S and V is selected to avoid overestimation. To improve the stability of training, three corresponding target neural networks A are constructed. t S t and V t The network parameter is θ t φ 1，t and φ 2，t ;

[0009] Step 3: Initialize network parameters θ, φ1, and φ2, and clear the experience replay buffer.

[0010] Step 4: Set the target network parameters to be the same as those of the main network θ t ←θ, φ 1，t ←φ1, φ 2，t ←φ2;

[0011] Step 5: In the nth time slot, set the coordinates q of the UAV's current position. (n) The drone's current remaining energy E (n) User-to-drone transmission rate Channel state information from user to drone Construct the current state vector s (n) ={q (n) E (n) R (n) Z (n)}, and input it into neural network A, which outputs the current action. Where μ θ (s (n) ) is the current policy μ θ Based on the current state s (n) The output value, It is Gaussian noise;

[0012] Step 6: The drone executes the current action x (n) It can choose to fly to the next location or hover, collect user data, and send it to low-Earth orbit satellites;

[0013] Step 7: Calculate the current communication rate of the system. Where B represents the current communication channel bandwidth. This indicates the signal-to-interference ratio (SIR) of the communication link between the user and the drone. Indicates the signal-to-interference ratio (SIR) of the communication link between the UAV and the low-Earth orbit satellite; calculates the user fairness index. Where c k This represents the proportion of data transmitted by user k to the total data transmitted by all users; it also calculates the energy consumption ΔE of the UAV in the current time slot. (n) =E (n) -E (n-1) ;

[0014] Step 8: Through Calculate the drone's current reward value and apply the experience e (n) =(s (n) ,x (n) ,r (n) ,s (n+1) Stored in the experience replay buffer pool middle;

[0015] Step 9: If the number of data in the experience replay buffer pool exceeds a certain number, then randomly draw data from the buffer pool. data

[0016] Step 10: Calculate the target value

[0017] Step 11: Update neural networks S and V using the calculated target values ​​via stochastic gradient descent.

[0018] Step 12: After neural networks S and V are updated d times, neural network A is updated using the policy gradient algorithm:

[0019]

[0020] Step 13: Adjust the target network parameters θ t and φ j,t Perform soft updates, where ρ is the update rate of the neural network:

[0021] φ j,t ←ρφ j,t +(1-ρ)φ j (j=1,2),

[0022] θ t ←ρθ t +(1-ρ)θ

[0023] Step 14: Repeat steps 5-13 until the algorithm converges.

[0024] This invention has the following outstanding advantages:

[0025] Unlike existing methods for improving UAV energy efficiency and user fairness, this invention proposes a highly reliable method for intelligent energy saving and fair resource management of UAVs in an integrated air-space-ground-sea network based on deep reinforcement learning algorithms, addressing interference attacks. This method optimizes UAV flight trajectories, enhances the anti-interference capability of the communication system, achieves highly reliable and interference-resistant secure communication, and improves UAV energy efficiency and user fairness. Simulation results show that, compared with the deep deterministic policy gradient algorithm, the deep reinforcement learning-based algorithm provided in this invention, namely the dual-delay deep deterministic policy gradient algorithm, offers significant improvements in fairness index and UAV energy efficiency. Attached Figure Description

[0026] Figure 1 A flowchart illustrating a high-efficiency resource management method for an unmanned aerial vehicle-assisted integrated air-ground-sea communication network under interference attacks, provided as an embodiment of the present invention;

[0027] Figure 2 This is a model diagram of the intelligent energy-saving and resource management method described in an embodiment of the present invention;

[0028] Figure 3 This is a comparison chart of the cumulative rewards of the intelligent energy-saving and resource management algorithm described in this embodiment of the invention under different iteration numbers. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the following embodiments will be used in conjunction with the accompanying drawings to further illustrate the invention. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0030] like Figure 1 As shown in the figure, an embodiment of the present invention provides a high-efficiency resource management method for an unmanned aerial vehicle-assisted air-ground-sea integrated communication network under interference attacks, comprising the following steps:

[0031] 1) Construct three main neural networks and their corresponding target neural networks, wherein the three main neural networks are neural network A, neural network S and neural network V;

[0032] 2) Initialize network parameters and clear the experience replay buffer to obtain the system's initial observations of the environment;

[0033] 3) Set the network parameters of the target neural network to be the same as those of the main neural network;

[0034] 4) Observe the current state of the UAV, construct a state vector and input it into the neural network A, and output the current action; the current state of the UAV includes the current position of the UAV, the current remaining energy of the UAV, the transmission speed, and the channel state information;

[0035] 5) The drone executes the current action output in step 4), selects to fly to the next location or hover, collects user data and sends it to low-Earth orbit satellites;

[0036] 6) Calculate the system's current communication rate, user fairness index, energy consumption of the current time slot, and the drone's current reward value;

[0037] 7) Store the experience in the experience replay buffer pool;

[0038] 8) Determine if the number of data in the experience playback buffer pool is greater than the number of samples. If not, return to step 4).

[0039] 9) Randomly sample data from the experience playback buffer and calculate the target value;

[0040] 10) Asynchronously update the parameters of each main neural network: using the target value calculated in step 9), update neural network S and neural network V through stochastic gradient descent; after neural network S and neural network V are updated d times, update neural network A through policy gradient algorithm;

[0041] 11) Softly update the parameters of the target neural network;

[0042] 12) Repeat steps 4)-11) until the algorithm converges.

[0043] In step 1), the construction of the main neural network and its corresponding target neural network specifically involves: constructing three main neural networks: neural network A, neural network S, and neural network V, with network parameters θ, φ1, and φ2, respectively; and constructing three target neural networks A corresponding to the three main neural networks. t S t and V t Network parameters θ t φ 1，t and φ 2，t .

[0044] In step 2), the specific steps for initializing network parameters and clearing the experience replay buffer are: initializing the network parameters θ, φ1, and φ2 of the three main neural networks, and clearing the experience replay buffer.

[0045] In step 3), the network parameters of the target neural network are set to be the same as those of the main neural network. t ←θ, φ 1，t ←φ1, φ 2，t ←φ2;

[0046] In step 4), the specific steps for constructing the state vector input neural network A include: observing the current state of the UAV, and in the nth time slot, setting the coordinates q of the UAV's current position. (n) The drone's current remaining energy E (n) User-to-drone transmission rate Channel state information from user to drone Constructed as a state vector s (n) ={q (n) E (n) R (n) Z (n)}, input it into neural network A, and it outputs the current action. Where, μ θ (s (n) () is based on the current strategy μ θ The output action, It is Gaussian noise.

[0047] In step 6), the current communication rate of the computing system is as follows:

[0048]

[0049] Where B represents the current communication channel bandwidth, This indicates the signal-to-interference ratio (SIR) of the communication link between the user and the drone. This indicates the signal-to-interference ratio (SIR) of the communication link between the drone and the low-Earth orbit satellite.

[0050] The user fairness index is calculated as follows:

[0051]

[0052] Among them, c k This represents the proportion of data transmitted by user k to the total data transmitted by all users, where K is the number of users in the system.

[0053] Calculate the energy consumption ΔE of the UAV in the current time slot. (n) As shown in the following formula:

[0054] ΔE (n) =E (n) -E (n-1)

[0055] Among them, E (n) E represents the remaining energy at the current moment. (n-1) This represents the remaining energy from the previous moment.

[0056] The current reward value of the drone is calculated as follows:

[0057]

[0058] In the formula, f (n) For the sake of fairness at the present moment, Indicates the energy efficiency at the current moment.

[0059] In step 7), storing the experience in the experience replay buffer specifically means: experience e (n) =(s (n) x (n) r (n) s (n+1) ); where s (n) Let x represent the current state vector. (n) Indicates the current action, r (n) This represents the current reward value of the drone, s (n+1) This represents the state vector for the next time slot;

[0060] In step 9), the random sampling of data from the experience playback buffer pool refers to sampling data from the experience playback buffer pool. Randomly selected There are 10 empirical data points, where the i-th empirical data point can be represented as:

[0061] The target value is calculated as follows:

[0062]

[0063] In the formula, y (i) Represents the target value, r (i) This indicates the current reward value;

[0064] In step 10), the asynchronous update of the parameters of each main neural network specifically involves:

[0065] Use the calculated target value y (i) The neural networks S and V are updated using stochastic gradient descent:

[0066]

[0067]

[0068] In the formula, y (i) It is the target value. and These represent the predicted values ​​given by neural networks S and V, respectively. This represents the expected value calculated.

[0069] After neural networks S and V are updated d times, neural network A is updated using the policy gradient algorithm:

[0070]

[0071] In the formula, express Regarding x (i) gradient value, μ θ (s (i) The gradient value with respect to θ This indicates the calculation of the expected value;

[0072] In step 11), the soft update of the target network parameters refers to updating the target network parameters θ. t and φ j，t Perform a soft update:

[0073] φ j，t ←ρφ 1，t +(1-ρ)φ1,

[0074] φ j，t ←ρφ 2，t +(1-ρ)φ2

[0075] θt ←ρθ t +(1-ρ)θ

[0076] In the formula, ρ is the update rate of the neural network.

[0077] The following combination Figure 2 A specific embodiment is provided, including the following steps:

[0078] Step 1: The specific system model of the present invention is as follows Figure 2 As shown, the integrated air-space-ground-sea communication system has 30 users, including ground users and maritime users, randomly distributed on a plane, while the jamming transmitter's position is fixed. Each time slot, the UAV flies to the next location and collects user data to transmit to the satellite; both actions consume the UAV's energy. The UAV adjusts its flight trajectory to counter jamming attacks in the wireless communication network; the action space is denoted as... The vector consisting of the drone's flight angle and flight distance is denoted as...

[0079] Step 2: Construct three main neural networks with three fully connected layers: neural network A, neural network S, and neural network V, with network parameters θ, φ1, and φ2; the first fully connected layer consists of 4 neurons, the second fully connected layer consists of 64 neurons, and the third fully connected layer consists of 4 neurons; the three target neural networks A... t S t and V t The number of layers and their size are consistent with the corresponding main network, and its network parameters are θ. t φ 1，t and φ 2，t Experience replay buffer pool The size is 200,000;

[0080] Step 3: Initialize network parameters θ, φ1, and φ2, and clear the experience replay buffer.

[0081] Step 4: Set the target network parameters to be the same as the main network parameters θ t ←θ, φ j，t ←φ j ;

[0082] Step 5: In the nth time slot, set the coordinates q of the UAV's current position. (n) The drone's current remaining energy E (n) User-to-drone transmission rate Channel state information from user to drone Construct the current state vector s (n) ={q (n) E (n) R(n) Z (n)}, input it into neural network A, and it outputs an action. Where, μ θ (s (n) ) is the current policy μ θ Based on the current state s (n) The output value, It is Gaussian noise;

[0083] Step 6: The drone performs the action x (n) It flies to the next location or hovers at the current location to collect user data and send it to low-Earth orbit satellites;

[0084] Step 7: Calculate the current communication rate of the system. Where B represents the current communication channel bandwidth, It is the signal-to-interference ratio (SIR) of the communication link between the user and the drone. It is the signal-to-interference ratio (SIR) of the communication link between the UAV and the low-Earth orbit satellite; calculating the user fairness index. Where c k This represents the proportion of data transmitted by user k compared to all users; calculate the current energy consumption ΔE of the drone. (n) =E (n) -E (n-1) ;

[0085] Step 8: Through Calculate the drone's current reward value and apply the experience e (i) =(s (n) x (n) r (n) s (n+1) Stored in the experience replay buffer pool middle;

[0086] Step 9: If the number of data in the experience replay buffer pool is greater than 3200, then randomly draw from the buffer pool. data

[0087] Step 10: Calculate the target value Where γ = 0.95;

[0088] Step 11: Update neural networks S and V using the calculated target values ​​via stochastic gradient descent:

[0089] Step 12: After neural networks S and V are updated d = 2 times, neural network A is updated using the policy gradient algorithm:

[0090]

[0091] Step 13: Parameterize the target network and φ j，t Soft updates are performed, where ρ = 0.99 is the update rate of the neural network: φ j，t ←ρφ j，t +(1-ρ)φ j (j=1,2,θ t ←ρθ t +(1-ρ)θ;

[0092] Step 14: Repeat steps 5-13 above until the algorithm converges.

[0093] Simulation results:

[0094] Figure 3 A comparison chart of the cumulative rewards of the intelligent energy-saving and resource management algorithm described in this embodiment of the invention at different iteration numbers is provided. Figure 3 It can be seen that the cumulative reward of the proposed dual-delay deep deterministic policy gradient algorithm increases with the number of iterations. A higher cumulative reward indicates higher system energy efficiency and fairness. Compared to the deep deterministic policy gradient algorithm, when the jamming transmitter's transmit power is 20 dBm and the algorithm has 200 iterations, the cumulative reward of the proposed algorithm is 209.06% higher. The cumulative reward curve is shaded by a 90% confidence interval; the interval for the proposed algorithm is much narrower than that for the compared algorithm, indicating that the proposed algorithm has significantly higher stability and demonstrates greater consistency and reliability.

[0095] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A high-efficiency resource management method for UAV-assisted air-space-ground-sea communication networks under jamming attacks, characterized in that... Includes the following steps: 1) Construct three main neural networks and their corresponding target neural networks. The three main neural networks are neural networks. Neural networks and neural networks ; 2) Initialize network parameters and clear the experience replay buffer to obtain the system's initial observations of the environment; 3) Set the network parameters of the target neural network to be the same as those of the main neural network; 4) Observe the current state of the drone and construct a state vector input neural network. The current action is output; the current state of the UAV includes the current position of the UAV, the current remaining energy of the UAV, the transmission speed, and the channel status information. 5) The drone executes the current action output in step 4), selects to fly to the next location or hover, collects user data and sends it to low-Earth orbit satellites; 6) Calculate the system's current communication rate, user fairness index, energy consumption of the current time slot, and the drone's current reward value; The current communication rate of the computing system is as follows: in, Indicates the current communication channel bandwidth. This indicates the signal-to-interference ratio (SIR) of the communication link between the user and the drone. This indicates the signal-to-interference ratio (SIR) of the communication link between the drone and the low-Earth orbit satellite. The user fairness index is calculated as follows: in, Representing the present The proportion of data transmitted by each user out of all users. The number of users in the system; Calculate the energy consumption of the drone in the current time slot. As shown in the following formula: in, The remaining energy at the current moment. This represents the remaining energy from the previous moment. The current reward value of the drone is calculated as follows: In the formula, For the sake of fairness at the present moment, Indicates the energy efficiency at the current moment; 7) Store the experience in the experience replay buffer pool; 8) Determine if the number of data in the experience playback buffer is greater than the number of samples. If not, return to step 4). 9) Randomly sample data from the experience playback buffer and calculate the target value; 10) Asynchronously update the parameters of each main neural network: Using the target value calculated in step 9), update the neural network through stochastic gradient descent. and neural networks Neural Network renew Subsequently, the neural network is updated using the policy gradient algorithm. ; 11) Softly update the parameters of the target neural network; 12) Repeat steps 4)-11) until the algorithm converges.

2. The high-efficiency resource management method for UAV-assisted air-space-ground-sea communication networks under interference attacks as described in claim 1, characterized in that... In step 1), the construction of three main neural networks and their corresponding target neural networks specifically involves: constructing three main neural networks: neural networks Neural networks Neural networks The network parameters are respectively , and Construct three target neural networks corresponding to the three main neural networks. , and The network parameters are , and .

3. The high-efficiency resource management method for UAV-assisted air-space-ground-sea communication networks under interference attacks as described in claim 1, characterized in that... In step 2), the specific steps for initializing network parameters and clearing the experience replay buffer are: initializing the network parameters of the three main neural networks. , and Clear the experience replay buffer pool .

4. The high-efficiency resource management method for UAV-assisted air-space-ground-sea communication networks under interference attacks as described in claim 2, characterized in that... In step 3), the network parameters of the target neural network are set to be the same as those of the main neural network. .

5. The high-efficiency resource management method for UAV-assisted air-space-ground-sea communication networks under interference attacks as described in claim 1, characterized in that... In step 4), the state vector input neural network is constructed. The specific steps include: observing the current status of the drone, the first... The time slot will display the coordinates of the drone's current location. The drone's current remaining energy User-to-drone transmission rate and channel state information from the user to the drone Construct as a state vector Input it into the neural network Output the current action ,in, Based on the current strategy The output action, It is Gaussian noise.

6. The high-efficiency resource management method for UAV-assisted air-space-ground-sea communication networks under interference attacks as described in claim 1, characterized in that... In step 7), storing the experience in the experience replay buffer specifically means: experience ;in, Represents the current state vector. Indicates the current action. This indicates the drone's current reward value. This represents the state vector for the next time slot.

7. The high-efficiency resource management method for UAV-assisted air-space-ground-sea communication networks under interference attacks as described in claim 1, characterized in that... In step 9), the random sampling of data from the experience playback buffer pool refers to sampling data from the experience playback buffer pool. Randomly selected from The first piece of empirical data, of which the A piece of empirical data can be represented as ; The target value is calculated as follows: In the formula, Indicates the target value. This represents the current reward value.

8. The high-efficiency resource management method for UAV-assisted air-space-ground-sea communication networks under interference attacks as described in claim 1, characterized in that... In step 10), the asynchronous update of the parameters of each main neural network is as follows: Use the calculated target value Update the neural network using stochastic gradient descent and neural networks : In the formula, It is the target value. and Representing neural networks and neural networks The given predicted value, This represents the expected value calculated. Neural Networks renew Subsequently, the neural network is updated using the policy gradient algorithm. : In the formula, express about gradient value, express about gradient value, This indicates the calculation of expectation.

9. A high-efficiency resource management method for UAV-assisted air-space-ground-sea communication networks under interference attacks as described in claim 2, characterized in that... In step 11), the soft update of the target neural network parameters refers to updating the target network parameters. and Perform a soft update: In the formula, It is the update rate of the neural network.

Images