An Anti-Jamming Communication Method for UAV Edge Computing Based on Reinforcement Learning

Abstract

A method for anti-jamming communication of UAV edge computing based on reinforcement learning, which involves fields such as UAV communication, UAV path planning, edge computing, anti-jamming communication, etc. It fully exploits key information such as the UAV's position, remaining battery power, and communication rate, combines parameters such as the communication channel state and communication performance between the UAV and ground devices and jammers, and proposes an intelligent resource management and control scheme based on deep reinforcement learning to dynamically optimize the UAV's trajectory and the user's transmission power, effectively resisting malicious jammer attacks while completing mobile edge computing tasks. In a complex malicious jamming attack environment, it still efficiently improves the UAV's communication rate and ensures the throughput performance of edge computing tasks.

Description

Technical Field

[0001] The present invention relates to the fields of unmanned aerial vehicle (UAV) communication, UAV path planning, edge computing, anti-jamming communication, etc., and particularly relates to a UAV edge computing anti-jamming communication method based on reinforcement learning. Background Art

[0002] Due to advantages such as flexibility, low equipment cost, and strong computing power, UAVs can serve as computing nodes to assist device operations well, and thus have good application prospects in the field of edge computing. However, when UAVs are attacked by malicious interference signals, problems such as slower communication rates and increased UAV transmission energy consumption may occur, reducing the working efficiency of UAVs, causing economic losses, and even threatening information transmission security. Therefore, UAV anti-jamming edge computing is an important research problem.

[0003] Mobile edge computing systems can help share the computing pressure of mobile devices and improve the computing efficiency of devices. When performing edge computing, the edge computing efficiency is improved by planning paths and transmission powers. Chinese Patent CN202110609006.9 proposes a UAV communication anti-jamming method based on three-dimensional trajectory power optimization, which uses the block coordinate descent algorithm and the successive convex approximation algorithm to plan the UAV transmission power and trajectory to achieve the purpose of anti-jamming.

[0004] Some methods based on reinforcement learning have also attracted increasing attention. [L.Xiao, X.Lu, T.Xu, X.Wan, W.Ji, and Y.Zhuang, Reinforcement Learning-Based Mobile Offloading for Edge Computing Against Jamming and Interference, IEEE Trans.Communications, vol.68, no.10, pp.6114-6126, Oct 2020] proposes a reinforcement learning-based edge computing anti-jamming scheme, which uses secure reinforcement learning to avoid selecting offloading strategies that do not meet the task computing delay requirements, and can autonomously select the transmission power and offloading rate to counteract interference, reduce energy consumption, and improve the signal-to-noise ratio.

[0005] Chinese Patent CN202011312164.X adopts a method for anti-hostile interference of UAV-assisted Internet of Things based on reinforcement learning, introducing the WoLF-PHC algorithm to dynamically optimize the transmission power of ground sensor nodes, the transmission power of UAVs and the movement trajectory of UAVs to counter interference. J. Li et al. [J. Li, Q. Liu, P. Wu, F. Shu and S. Jin, Task Offloading for UAV-based Mobile Edge Computing via Deep Reinforcement Learning, 2018 IEEE / CIC International Conference on Communications in China (ICCC), Beijing, China, ISSN: 2377, Oct 2018] proposed an edge computing method based on deep reinforcement learning. Under the conditions of known UAV position and remaining battery power, this method selects the optimal task offloading route and tries to increase the amount of offloaded tasks as much as possible under the condition of limited battery power. However, this scheme does not take the communication rate as the quality of service and does not discuss communication anti-interference. Summary of the Invention

[0006] The object of the present invention is to provide a method for anti-interference communication of UAV edge computing based on reinforcement learning, which uses a deep reinforcement learning algorithm to optimize the UAV flight trajectory and user transmission power to adapt to the UAV communication environment with dynamic interference, and solves the path planning problem of efficiently performing task offloading and resisting interference attacks by UAVs under the condition of limited battery power. To improve the system revenue, communication rate and throughput, the present invention uses the UAV position, the remaining battery power of the UAV and the user communication rate as state information, combines the UAV throughput and the user transmission rate as the quality of service, performs edge computing under the interference of the jammer, and uses the deep Q network to jointly optimize the UAV flight trajectory and user transmission power to improve the anti-interference performance of the system.

[0007] The present invention includes the following steps:

[0008] Step 1: Let the time required for the UAV battery to run out be K, and the time 1 ≤ k ≤ K; the number of hover positions that the UAV wireless communication device can hover at is denoted as N, and the hover position at time k is denoted as

[0009] Step 2: Initialize the positions of users, the positions and powers of jammers in the system model The number of users is denoted as M, and the position where the user is located is Position The task volume of the user at The number of available transmission power of the jammer is denoted as N2, and the transmission power The position of the jammer is

[0010] Step 3: Initialize the power b of the UAV (0) , and the communication transmission speed of the UAV is The initial position of the UAV The exploration rate ε = ε max ;

[0011] Step 4: Construct neural network A and neural network B, with network parameters ω1 and ω2 respectively; in the model, the jammer will continuously move towards the UAV. At time slot k, obtain the position information of the jammer The distance between the UAV and the jammer Calculate the channel gain between the UAV and the user through the distance between the UAV and the user Calculate the channel gain between the UAV and the jammer through the distance between the UAV and the jammer

[0012] Step 5: Construct the current state vector

[0013] Step 6: Input the state vector s (k) into neural network A to obtain the estimated value Q(s (k) , x (k) ); With a probability of 1 - ε, select the next hovering position and control signal of the UAV with the maximum Q i value; When performing the action, the UAV first flies to the next hovering position as required and sends the user transmission power strategy to the user, and the user transmits power according to this strategy Upload data packets; The number of user transmission powers is denoted as N1, and the user transmission power

[0014] Step 7: The UAV flies to the next hovering point at speed V according to the selected action, and the measured jammer power is And it is assumed that the position of the jammer can be observed; At this time, the task volume of each user follows a Poisson distribution, and the user 's task volume is Then, obtain the communication rate

[0015] Communication rate:

[0016] Step 8: Calculate the flight energy consumption of the UAV:

[0017] Among them, the moving distance P g is the flight power;

[0018] Calculate the hovering energy consumption of the UAV:

[0019] At this time, the jammer adjusts the transmission power of the jammer according to the greedy algorithm to maximize the hovering energy consumption;

[0020] Calculate the computing energy consumption of the UAV:

[0021] Calculate the remaining battery power of the UAV

[0022] Step 9: Specify the quality of service;

[0023] Condition 1:

[0024] Condition 2:

[0025] Step 10: Calculate the utility and reward;

[0026] Utility:

[0027] When the QoS condition 1 is satisfied, Otherwise

[0028] When the QoS condition 2 is satisfied, Otherwise

[0029] Reward:

[0030] Step 11: Transmit the detected transmission power of the jammer Offloading volume Transmission speed Form feedback information And feedback it to the UAV through channels such as the control channel;

[0031] Step 12: Store the historical edge computing anti-jamming experience e (k) ={s (k) ,x (k) ,U (k)} into the cache pool H. After Z time slots, randomly take out a historical experience e (i) , and input the state vector s (k) into the neural network B. Denote the output of the neural network B as

[0032] Step 13: Update the weight parameters of the neural network A using the stochastic gradient descent algorithm That is:

[0033]

[0034] And every c time slots apart, let

[0035] Step 14: If the UAV's power runs out, end this round;

[0036] Step 15: Repeat steps 4 to 14 until |Q(s (k+1) , x (k+1) ) - Q(s (k+1) , x (k+1) )| < 0.01, that is, the algorithm converges.

[0037] Compared with the prior art, the present invention has the following prominent advantages:

[0038] The UAV constructs state information using position, power, and communication rate, continuously optimizes the UAV task offloading trajectory and transmission power based on this state information, and sets the corresponding quality of service, selects an optimization strategy according to the state and quality of service, so as to adaptively resist interference while performing edge computing. The simulation results show that compared with the comparative algorithm, the benefits, communication rate, and task offloading volume of the method of the present invention are effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the edge computing anti-interference communication method model described in the embodiment of the present invention.

[0040] Figure 2 It is the average benefit convergence graph of the edge computing anti-interference communication method described in the embodiment of the present invention.

[0041] Figure 3 It is the communication rate of the edge computing anti-interference communication method described in the embodiment of the present invention under different maximum transmission power values.

[0042] Figure 4 It is the average throughput of the edge computing anti-interference communication method described in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the following embodiments will further illustrate the present invention in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0044] The specific embodiments of the present invention include the following steps:

[0045] Step 1: There are 30 positions where the UAV can hover,

[0046] Step 2: There are 20 users in the system, randomly distributed on a plane. Some users will move in various directions, and the remaining users will not move. The task volume of the users is The starting position of the jammer is random. The number of available transmission powers is 20, evenly distributed between [0, 0.05].

[0047] Step 3: Initialize the drone's power b (0) = 10, the communication transmission speed of the drone The initial position of the drone The exploration rate ε = 0.9;

[0048] Step 4: Construct a random network A and a neural network B, both composed of three-layer neural networks. The initial network parameters are ω1 = ω2 = 0. The input of the first input layer is 5, the dimension size of the second hidden layer is 20, and the output of the third output layer is 30×5 = 150. In the model, every time slot k, the jammer will move 10m towards the drone. At time slot k, obtain the position information of the jammer The distance between the drone and the jammer Calculate the channel gain between the drone and the user by the distance between the drone and the user Calculate the channel gain between the drone and the jammer by the distance between the drone and the jammer

[0049] Step 5: Construct the current state vector

[0050] Step 6: Input the state vector s (k) into the neural network A to obtain the estimated value Q(s (k) , x (k) ) of the state-action pair; with a probability of 1 - ε, select the next hovering position of the drone with the maximum Q i value, as well as the control signal. The number of available transmission powers of the users is denoted as N1, and the user transmission power The user transmission power levels are 5, 0.0325, 0.055, 0.0775, 0.1;

[0051] Step 7: The drone flies to the next hovering point at a speed of V = 15m / s according to the selected action, and the height H is fixed at 100m. The measured power of the jammer is And it is assumed that the position of the jammer can be observed. At this time, the task volumes of each user follow a Poisson distribution with an average value of 8. The task volume of the user is Then, obtain the communication rate

[0052] Communication rate: Among them, the sub-channel bandwidth K = 0.3 MHz;

[0053] Step 8: Calculate the flight energy consumption of the UAV:

[0054] The moving distance among them

[0055] Calculate the hovering energy consumption of the UAV:

[0056] At this time, the jammer adjusts the transmission power of the jammer according to the greedy algorithm;

[0057] Calculate the computing energy consumption of the UAV:

[0058] Calculate the remaining power of the UAV

[0059] Step 9: Specify the quality of service;

[0060] Condition 1: λ1 = 500000

[0061] Condition 2: λ2 = 8.5

[0062] Step 10: Calculate the utility and revenue;

[0063] Utility:

[0064] When the quality of service condition 1 is satisfied, Otherwise λ3 = 1

[0065] When the quality of service condition 2 is satisfied, Otherwise

[0066] Revenue:

[0067] Step 11: Use the detected transmission power of the jammer Offloading volume Transmission speed Form feedback information And feedback it to the UAV through channels such as the control channel;

[0068] Step 12: The historical edge computing anti-jamming experience e (k) = {s (k) , x (k) , U (k)It is stored in the cache pool H. After Z time slots, a historical experience e is randomly retrieved from the cache pool H (i) , and the state vector s (k) is input into the neural network B. Denote the output of the neural network B as

[0069] Step 13: Update the weight parameters of the neural network A using the stochastic gradient descent algorithm That is

[0070]

[0071] And every c time slots, let where δ = 0.01, c = 50;

[0072] Step 14: If the UAV's power runs out, end the round;

[0073] Step 15: Repeat steps 4 to 14 until |Q(s (k+1) , x (k+1) ) - Q(s (k+1) , x (k+1) )| < 0.01, that is, the algorithm converges.

[0074] Figure 1 is the model diagram of the edge computing anti-interference communication method described in the embodiment of the present invention. Figure 2 is the benefit of the edge computing anti-interference communication method described in the embodiment of the present invention. Among them, the deep Q network is the method of the present invention. In the method of the present invention, the UAV trajectory and transmission power are selected by the deep Q network. In the comparison algorithm, the deep Q network - randomly selecting the transmission power means that the transmission power is randomly selected. Figure 3 is the communication rate of the edge computing anti-interference communication method described in the embodiment of the present invention. When the number of rounds is greater than 500, the benefit and communication transmission speed tend to be stable. Comparing the communication rates under different maximum transmission powers, from Figure 3 it can be obtained that the algorithm of the present invention can achieve the optimal communication rate. Figure 4 is the throughput of the edge computing anti-interference communication method described in the embodiment of the present invention. From Figures 1 to 4 it can be obtained that the anti-interference edge computing method based on reinforcement learning proposed by the present invention can effectively improve the benefit, communication rate and average task throughput of the system. The present invention fully excavates key information such as the UAV position, the remaining power of the UAV and the communication rate of the UAV, combines parameters such as the communication channel state and communication performance between the UAV and ground equipment and jammers, and proposes an intelligent resource management and control scheme based on deep reinforcement learning to dynamically optimize the UAV trajectory and the transmission power of users, effectively resisting malicious jammer attacks while completing mobile edge computing tasks. In a complex malicious interference attack environment, it still efficiently improves the communication rate of the UAV and guarantees the throughput performance of edge computing tasks.

[0075] The above embodiments are only preferred embodiments of the present invention and cannot be considered as limiting the scope of implementation of the present invention. Any equivalent changes and improvements made within the scope of the application of the present invention shall still fall within the scope covered by the patent of the present invention.

Claims

1. A method for anti-interference communication of UAV edge computing based on reinforcement learning, characterized in that Including the following steps: Step 1: Let the time required for the UAV to run out of power be K, where 1 ≤ k ≤ K; the number of hover positions that the UAV wireless communication device can reach is denoted as N, and the hover position at time k is denoted as Step 2: Initialize the positions of the users, the positions and powers of the jammers in the system model The number of users is denoted as M, and the positions of the users are Position The user task volume of The number of available transmission power levels of the jammer is denoted as N2, and the transmission power The position of the jammer is Step 3: Initialize the power b of the drone (0) , the communication transmission speed of the drone is Initial position of the drone Exploration rate ε = ε max ; Step 4: Construct neural network A and neural network B with network parameters ω1 and ω2 respectively; in the model, the jammer keeps moving towards the UAV direction, and at time slot k, obtain the jammer position information The distance between the UAV and the jammer Obtain the channel gain between the UAV and the user based on the distance between the UAV and the user Obtain the channel gain between the UAV and the jammer based on the distance between the UAV and the jammer Step 5: Construct the current state vector Step 6: Input the state vector s (k) into neural network A to obtain the estimated value Q(s (k) , x (k) ); Select the next hovering position and control signal of the UAV with the maximum Q i value with a probability of 1 - ε; When performing the action, the UAV first flies to the next hovering position as required and sends the user's transmit power policy to the user, and the user transmits power according to this policy to upload the data packet; The number of the user's transmit powers is denoted as N1, and the user's transmit power Step 7: The UAV flies to the next hovering point at speed V according to the selected action, and the measured jammer power is and it is assumed that the position of the jammer can be observed; the task volumes of each user follow a Poisson distribution, and the task volume of the user is After that, the communication rate is obtained Communication rate: Step 8: Calculate the flight energy consumption of the drone: The moving distance therein P g is the flight power; Calculate the hovering energy consumption of the drone: At this time, the jammer adjusts the transmission power of the jammer according to the greedy algorithm to maximize the hovering energy consumption; Calculate the computing energy consumption of the UAV: Calculate the remaining power of the drone Step 9: Specify the quality of service; Condition 1: Condition 2: Step 10: Calculate the utility and reward; When the QoS condition 1 is satisfied, Otherwise When the QoS condition 2 is satisfied, Otherwise Step 11: The detected transmission power of the jammer Unloading amount Transmission speed Form feedback information And feed it back to the UAV through the control channel Step 12: Store the historical edge computing anti-interference experience e (k) ={s (k) ,x (k) ,U (k)} in the cache pool H. After Z time slots, randomly take out a historical experience e (i) from the cache pool H, and input the state vector s (k) into the neural network B. Denote the output of the neural network B as Step 13: Update the weight parameters of neural network A using the stochastic gradient descent algorithm That is: And every c time slots apart, let Step 14: If the UAV runs out of power, end; Step 15: Repeat steps 4 to 14 until |Q(s (k+1) ,x (k+1) ) - Q(s k ,x k )| < 0.01, which means the algorithm converges.

Images