Method for optimizing flight path of robust communication between unmanned aerial vehicles under malicious interference

Abstract

According to the method for optimizing the flight path of robust communication between the unmanned aerial vehicles under malicious interference, the system maximizes the communication capacity between the unmanned aerial vehicles while ensuring that communication links between the unmanned aerial vehicles are not interrupted in the face of malicious interference through synchronization and cooperation of flight strategies of the unmanned aerial vehicles, and therefore the communication quality of the unmanned aerial vehicles is effectively improved. Based on this, an unmanned aerial vehicle trajectory optimization scheme with maximization of communication capacity between unmanned aerial vehicles as an optimization target is established. And then, solving the communication capacity optimization problem between the unmanned aerial vehicles by utilizing the constructed double-delay depth deterministic strategy gradient depth reinforcement learning framework. A numerical result shows that the method can realize robust and stable communication between the unmanned aerial vehicles in a malicious interference environment.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle communication and control technology, and in particular to a method for optimizing the flight trajectory of robust communication between unmanned aerial vehicles under malicious interference. Background Art

[0002] With the rapid development of drone communication technology, various means of interfering with drone communications have emerged one after another, and the demand for drone communication security is growing. The strong maneuverability and flexibility of traditional drones make it possible for them to avoid malicious interference signals by planning flight trajectories and ensure the security of drone communication links.

[0003] With the rapid development of unmanned aerial vehicles (UAV) technology and its increasingly wide application scenarios, communication between UAVs has become increasingly important. However, with the continuous evolution of malicious interference technology, UAV secure communication faces severe challenges. Traditional anti-interference technology relies on static channel models and cannot effectively cope with highly dynamic and complex interference environments, making it difficult to ensure stable communication of UAVs under malicious interference.

[0004] Because deep reinforcement learning has demonstrated powerful optimization decision-making capabilities in highly dynamic scenarios, it can adapt to complex highly dynamic scenarios in real time by building an interaction model between the intelligent agent and the environment, conducting self-learning and exploration, and has broad application prospects in highly dynamic UAV communications. In an inter-UAV communication environment accompanied by malicious interference, deep reinforcement learning can optimize the flight trajectory of the UAV and adjust the communication strategy, effectively responding to the challenges brought by malicious interference. Compared with traditional optimization methods, deep reinforcement learning has higher flexibility and adaptability. It can take advantage of the high maneuverability of the UAV in an interference environment to actively adjust the communication parameters and flight path, thereby improving the overall anti-interference ability and communication efficiency of the UAV network, and providing the possibility of ensuring the security of the UAV communication link. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a robust and stable communication path optimization strategy between drones in a malicious interference environment, and to formulate flight action decisions of multiple drones through a deep reinforcement learning algorithm to avoid malicious interference signals and prevent interruption of communication links between drones.

[0006] To achieve the above-mentioned object of the invention, the present invention provides a flight trajectory optimization method for robust communication between unmanned aerial vehicles under malicious interference, comprising:

[0007] S1. Establish a two-dimensional plane system model of UAV air communication consisting of UAV and multiple interference sources;

[0008] S2. discretize the trajectory of the UAV and discretize the execution time of the UAV;

[0009] S3. Establish a channel model, including a channel model between UAVs and a channel model between UAVs and interference sources;

[0010] S4. Establish a UAV trajectory optimization problem with the goal of maximizing the inter-UAV communication capacity. Through the synchronization and coordination of UAV flight strategies, the inter-UAV communication link is guaranteed to be uninterrupted in the face of malicious interference while maximizing the inter-UAV communication capacity, thereby effectively improving its communication quality.

[0011] S5. By constructing a Twin Delayed Deep Deterministic Policy Gradient (TD3) deep reinforcement learning framework, the problem of optimizing the inter-UAV communication capacity is solved.

[0012] Preferably, in step S1, a bounded square map is created, and the map size is L map ×L map , where a drone network and malicious interference sources are deployed. The drone network consists of two drones, which use full-duplex communication with each other; several malicious interference sources are randomly distributed on the map, and their locations are i∈{1,2,3…},is i represents the i-th interference source; the starting points of the two drones are and The end points are and The drones maintained communication while performing the mission of flying from the origin to the destination; all drones and the jammer were equipped with a single omnidirectional antenna.

[0013] Preferably, in step S2, the map is divided into N×N cells, each cell having a side length of l net =L map / N; The maximum time taken by the drone to perform a mission is T, and the time is divided into M time slots, and the size of each time slot is T / M.

[0014] Preferably, in step S3, the UAV and the interference source are in free space, and the channel between the UAV and the channel between the interference source and the UAV are both free space fading, and the channel can be modeled as:

[0015]

[0016] where a∈{{u i},{is j}}, i∈{1,2}, j∈{1,2,3…} is the information source, b∈{u i}, i∈{1,2} is the destination, u i represents the i-th drone.

[0017] Preferably, in step S4, since the flight area of ​​the UAV is limited by physical space, the UAV needs to meet the following constraints at any time slot position:

[0018]

[0019] in For drone u i The abscissa at time slot t, For drone u i The ordinate at time slot t.

[0020] The drone flies from the starting point to the end point, and the flight speed is expressed as The maximum flight speed of the drone is v max , so the flight speed needs to meet the following constraints:

[0021]

[0022] Since the signal coverage of drones is limited, the distance between drones should not be too far to ensure the stability of the drone communication link. When the distance between drones is too close, there is a possibility of collision and crash. Therefore, the distance between drones needs to meet the following constraints:

[0023]

[0024] Where D min is the safe distance between drones to avoid collision, D max The maximum distance for the drone to maintain communication. The distance between drones can be expressed as

[0025] The communication capacity of the sink drone is expressed as Where B is the bandwidth, The source UAV transmits the signal, The interference source transmits a signal. is the Gaussian white noise power. To ensure that the UAV communication link is not interrupted, the sink UAV communication capacity needs to meet the following constraints:

[0026]

[0027] The time slot of the drone to the destination is expressed as The drone needs to reach the destination within the maximum mission time T and needs to meet the following constraints:

[0028]

[0029] Under the constraints of the mission area, flight speed, distance between drones and communication capacity between drones, the drone trajectory optimization problem with the goal of maximizing the communication capacity between drones can be modeled as follows:

[0030]

[0031]

[0032] Preferably, in step S5, the TD3 deep reinforcement learning framework needs to formulate the action space, state space and reward function of the drone communication environment under aerial malicious interference. The environmental parameters are input into the TD3 deep reinforcement learning framework for iterative training to obtain the optimal flight strategy. The specific steps include:

[0033] S501. The TD3 deep reinforcement learning framework used is suitable for continuous action space. In a two-dimensional plane environment, the drone can adjust the horizontal flight speed and forward angle by itself, so the action space of the drone can be defined as:

[0034]

[0035] S502. Communication capacity of drones Distance from drone Directly affects the communication quality of the drone, and the drone needs to monitor the distance to the end point at all times It is guaranteed to arrive at the destination within the Mth time slot, so the state space of the drone communication environment under aerial malicious interference can be defined as:

[0036]

[0037] The distance vector that the drone moves in each time slot is The discretization is expressed as Therefore, the update of the drone position can be expressed as:

[0038]

[0039] State variables and The update of is related to the position of the UAV, among which the relationship between the UAV communication capacity and the UAV position is expressed as follows:

[0040] S503. Constructing the reward function requires ensuring that the action decision model after iteration can satisfy the goal (7a) and constraints (7e), (7f) and (7g). The reward function can be divided into three sub-rewards: For (7a) and (7f), the sub-reward For (7e), in principle, when Appropriate approach D minWhen the communication quality is good, the reward should be high. Too close to D min When the risk of drone collision is high, the reward should be very low, and the sub-reward This relationship can be well described, where μ1, μ2, μ3>0, r t (2) (D min )=0,r t (2) (D max )=0; For (7g), the constraint on the time slot of reaching the destination can be converted into a constraint on the distance between the drone and the destination, and the sub-reward So the reward function of this system can be defined as:

[0041]

[0042] Among them, ρ1, ρ2, ρ3 are reward factors.

[0043] S504. Initialize the Actor network π in the TD3 deep reinforcement learning framework φ Parameter φ, Critic1 network With Critic 2 Network Parameters and corresponding Target network π φ′ , and parameter.

[0044] Based on the network parameters, input the current state s to the Actor network and obtain the action a=π φ (s)+∈,∈~N(0,σ). Input action a into the environment, obtain reward r and next state s′. Store (s,a,r,s′) into the experience replay buffer; input s and a into the dual critic network to obtain Q value Input the expected state s′ to the Target Actor network to obtain the expected action ε~N(0,σ), the expected action Input the dual Target Critic network to obtain the target Q value

[0045] In each cycle, the model extracts N sets of samples from the experience replay buffer, calculates the Q value and the target Q value, and uses Q θ The minimum mean square error between (s, a) and y is used as the loss function, and the gradient descent method is used to update the Critic network parameters θ i ; Periodically, every d cycles, the model extracts N sets of samples from the experience replay buffer, calculates the expected Q value, and uses the gradient ascent method to update the Actor network parameters φ, θ i, the update of φ can be expressed as:

[0046]

[0047] Where α and β are learning rates.

[0048] Periodically, every d cycles, update the Target network parameter θ i ′,φ′, the parameter update can be expressed as:

[0049] θ i ′←τθ i +(1-τ)θ i ′(13a)

[0050] φ′←τφ+(1-τ)φ′(13b)

[0051] Here τ is a hyperparameter less than 1.

[0052] The above steps are iterated repeatedly to optimize the network parameters until the reward converges, thereby maximizing the amount of drone information transmission under malicious interference.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The present invention provides a method for optimizing the flight trajectory of robust communication between UAVs under malicious interference based on deep reinforcement learning. Compared with traditional methods for resisting malicious interference, the present invention uses the airspace characteristics of electromagnetic signals and the flexibility of UAVs to enable UAVs to dynamically avoid areas with severe malicious interference signals. At the same time, the present invention uses the TD3 deep reinforcement learning algorithm to solve the problem of optimizing the flight trajectory of UAVs under malicious interference with the goal of maximizing the communication capacity between UAVs. Compared with traditional optimization problem solving methods, this method can analyze highly dynamic and complex environments, give full play to the high maneuverability of UAVs, quickly adjust UAV flight strategies, effectively avoid interference sources, and maximize the communication capacity between UAVs. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only preferred embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 It is a model diagram of a drone flight trajectory optimization strategy system aimed at maximizing the communication capacity between drones under malicious interference provided by an embodiment of the present invention.

[0057] Figure 2The present invention provides a schematic diagram of a flight trajectory for robust communication between drones under malicious interference provided by an embodiment of the present invention.

[0058] Figure 3 It is a comparison chart of the changes of UAV inter-flight communication capacity and UAV straight-line flight communication capacity over time under a method for optimizing the flight trajectory of robust communication between UAVs under malicious interference provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The enumerated embodiments are only used to explain the present invention and are not used to limit the scope of the present invention.

[0060] This embodiment provides a method for optimizing the flight trajectory of robust communication between drones under malicious interference, and the specific steps are as follows:

[0061] S1. Figure 1 As shown in the figure, a two-dimensional plane system model of UAV aerial communication consisting of UAV and interference source is established. The UAV mission area is a bounded square map with a map size of L map ×L map , where a drone network and malicious interference sources are deployed; the drone network consists of two drones, which use full-duplex communication with each other; several malicious interference sources are randomly distributed on the map, and their locations are i∈{1,2,3…},is i represents the i-th interference source; the starting points of the two drones are and The end points are and The drones maintained communication while performing the mission of flying from the origin to the destination; all drones and the jammer were equipped with a single omnidirectional antenna.

[0062] S2. Discretize the UAV flight trajectory and divide the map into N×N cells, each with a side length of l net =L map / N; discretize the UAV mission execution time. The maximum time for the UAV to execute the mission is T. The time is divided into M time slots, and the size of each time slot is T / M.

[0063] S3. Establish a channel model. The UAV and the interference source are in free space. The channel between UAVs and the channel between the interference source and the UAV are both free space fading. The channel can be modeled as:

[0064]

[0065] where a∈{{u i},{is j}}, i∈{1,2}, j∈{1,2,3…} is the information source, b∈{u i}, i∈{1,2} is the destination, u i represents the i-th drone.

[0066] S4. Establish a UAV trajectory optimization problem with the goal of maximizing the inter-UAV communication capacity. Since the UAV flight area is limited by physical space, the UAV has the following constraints at any time slot position:

[0067]

[0068] in For drone u i At the time slot t, For drone u i The t ordinate in the time slot.

[0069] The drone flies from the starting point to the end point, and the flight speed is expressed as The maximum flight speed of the drone is v max , so the flight speed needs to meet the following constraints:

[0070]

[0071] Since the signal coverage of drones is limited, the distance between drones should not be too far to ensure the stability of the drone communication link. At the same time, if the distance between drones is too close, there is a possibility of collision and crash. Therefore, the distance between drones needs to meet the following constraints:

[0072]

[0073] Where D min is the safe distance between drones to avoid collision, D max is the maximum distance for drones to maintain communication. The distance between drones can be expressed as

[0074] The inter-UAV communication capacity is expressed as Where B is the bandwidth, The source UAV transmits the signal, The interference source transmits a signal. is the Gaussian white noise power. To ensure that the communication link between UAVs is not interrupted, the communication capacity of the sink UAV needs to meet the following constraints:

[0075]

[0076] The time slot of the drone to the destination is expressed as The drone needs to reach the destination within the maximum mission time T and needs to meet the following constraints:

[0077]

[0078] Under the constraints of the mission area, flight speed, distance between drones and communication capacity between drones, the drone trajectory optimization problem with the goal of maximizing the communication capacity between drones can be modeled as follows:

[0079]

[0080] S5. Solve the problem of maximizing the communication capacity between drones by building a TD3 deep reinforcement learning framework. The TD3 deep reinforcement learning framework needs to formulate the action space, state space and reward function of the drone communication environment under aerial malicious interference. Input the environmental parameters into the TD3 deep reinforcement learning framework for iterative training to obtain the optimal flight strategy. The specific steps include:

[0081] S501. The TD3 deep reinforcement learning framework used is suitable for continuous action space. In a two-dimensional plane environment, the drone can adjust the horizontal flight speed and forward angle by itself, so the action space of the drone can be defined as:

[0082]

[0083] S502. UAV inter-machine communication capacity Distance from the machine Directly affects the quality of inter-UAV communication. UAVs need to monitor the distance to the destination at all times. It is guaranteed to arrive at the destination within the Mth time slot. Therefore, the state space of the drone communication environment under malicious interference can be defined as:

[0084]

[0085] The distance vector that the drone moves in each time slot is The discretization is expressed as Therefore, the update of the drone position can be expressed as:

[0086]

[0087] Obviously the state variable and The update of is related to the position of the UAV, among which the relationship between the UAV communication capacity and the UAV position is expressed as follows:

[0088] S503. Constructing the reward function requires ensuring that the action decision model after iteration can satisfy the goal (7a) and constraints (7e), (7f) and (7g). The reward function can be divided into three sub-rewards: For (7a) and (7f), the sub-reward For (7e), in principle, when Appropriate approach D min When the communication quality is good, the reward should be high. Too close to D min When the risk of drone collision is high, the reward should be very low, and the sub-reward This relationship can be well described, where μ1, μ2, μ3>0, r t (2) (D min )=0,r t (2) (D max )=0; For (7g), the constraint on the time slot of reaching the destination can be converted into a constraint on the distance between the drone and the destination, and the sub-reward So the reward function of this system can be defined as:

[0089]

[0090] Among them, ρ1, ρ2, ρ3 are reward factors.

[0091] S504. Initialize the Actor network π in the TD3 deep reinforcement learning framework φ Parameter φ, Critic1 network With Critic 2 Network Parameters and corresponding Target network π φ′ , and parameter.

[0092] Based on the network parameters, input the current state s to the Actor network and obtain the action a=π φ (s)+∈,∈~N(0,σ), input action a into the environment, obtain reward r and next state s′. Store (s,a,r,s′) into the experience replay buffer; input s and a into the dual critic network to obtain Q value Input the expected state s′ to the Target Actor network to obtain the expected action ~N(0,σ), the expected action Input the dual Target Critic network to obtain the target Q value

[0093] In each cycle, the model extracts N sets of samples from the experience replay buffer, calculates the Q value and the target Q value, and uses Q θ The minimum mean square error between (s, a) and y is used as the loss function, and the gradient descent method is used to update the Critic network parameters θ i; Periodically, every d cycles, the model extracts N sets of samples from the experience replay buffer, calculates the expected Q value, and uses the gradient ascent method to update the Actor network parameters φ, θ i , the update of φ can be expressed as:

[0094]

[0095] Where α and β are learning rates.

[0096] Periodically, every d cycles, update the Target network parameter θ′ i ,φ′, the parameter update can be expressed as:

[0097] θ′ i ←τθ i +(1-τ)θ′ i (13a)

[0098] φ′←τφ+(1-τ)φ′(13b)where τ is a hyperparameter less than 1.

[0099] The above steps are iterated repeatedly to optimize the network parameters until the reward converges, thereby achieving drone trajectory optimization with the optimization goal of maximizing the inter-drone communication capacity under malicious interference.

[0100] Figure 2 The flight trajectory of the drones after the TD3 deep reinforcement learning model reward convergence is given. It can be observed that the two drones can fly along a circular trajectory centered on the interference source during the mission, effectively avoiding the interference signal; at the same time, the flight trajectories of the two drones partially overlap, and they fly at a suitable distance from each other, ensuring that the drones do not collide due to being too close or interrupt communication due to being too far away. Figure 3 The comparison of inter-UAV communication capacity before and after flight strategy optimization is given. It can be seen that when the UAV flies straight to the destination, the UAV directly crosses the interference source, the communication capacity drops sharply, and the UAV communication is continuously interrupted; when the proposed algorithm is used, the inter-UAV communication capacity is high and stable. The experimental results show that the introduction of the UAV flight trajectory optimization method based on the TD3 algorithm ensures the robust communication between UAVs under malicious interference and significantly improves the inter-UAV communication capacity.

[0101] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A flight trajectory optimization method for robust communication between drones under malicious interference, characterized in that: The following steps are involved: S1. Establish a two-dimensional planar system model of UAV aerial communication consisting of UAVs and multiple interference sources; S2. discretize the trajectory of the UAV and discretize the execution time of the UAV; S3. Establish a channel model, including a channel model between UAVs and a channel model between UAVs and interference sources; S4. Establish a UAV trajectory optimization problem with the goal of maximizing the inter-UAV communication capacity. Through the synchronization and coordination of UAV flight strategies, the inter-UAV communication link is guaranteed to be uninterrupted in the face of malicious interference while maximizing the inter-UAV communication capacity, thereby effectively improving its communication quality. S5. By constructing a Twin Delayed Deep Deterministic Policy Gradient (TD3) deep reinforcement learning framework, the problem of optimizing the inter-UAV communication capacity is solved.

2. The method for optimizing flight trajectory of robust communication between unmanned aerial vehicles under malicious interference according to claim 1 is characterized in that: In step S1, a bounded square map is created with a map size of L map ×L map , where a drone network and malicious interference sources are deployed. The drone network consists of two drones, which use full-duplex communication with each other; several malicious interference sources are randomly distributed on the map, and their locations are is i represents the i-th interference source; the starting points of the two drones are and The end points are and The drones maintained communication while performing the mission of flying from the origin to the destination; all drones and the jammer were equipped with a single omnidirectional antenna.

3. The method for optimizing flight trajectory of robust communication between unmanned aerial vehicles under malicious interference according to claim 1 is characterized in that: In step S2, the map is divided into N×N cells, each cell has a side length of l net =L map / N; The maximum time taken by the drone to perform a mission is T, and the time is divided into M time slots, and the size of each time slot is T / M.

4. The method for optimizing flight trajectory of robust communication between unmanned aerial vehicles under malicious interference according to claim 1 is characterized in that: In step S3, the UAV and the interference source are in free space, and the channel between UAVs and the channel between interference sources and UAVs are both free space fading. The channel can be modeled as: where a∈{{u i },{is j }}, i∈{1,2}, j∈{1,2,3…} is the information source, b∈{u i }, i∈{1,2} is the destination, u i represents the i-th drone.

5. The method for optimizing flight trajectory of robust communication between unmanned aerial vehicles under malicious interference according to claim 1 is characterized in that: In step S4, since the flight area of ​​the UAV is limited by physical space, the UAV needs to meet the following constraints at any time slot position: in For drone u i The abscissa at time slot t, For drone u i The ordinate at time slot t; The drone flies from the starting point to the end point, and the flight speed is expressed as The maximum flight speed of the drone is v max , so the flight speed needs to meet the following constraints: Since the signal coverage of drones is limited, the distance between drones should not be too far to ensure the stability of the drone communication link. When the distance between drones is too close, there is a possibility of collision and crash. Therefore, the distance between drones needs to meet the following constraints: Where D min is the safe distance between drones to avoid collision, D max The maximum distance for UAVs to maintain communication is , and the distance between UAVs can be expressed as The communication capacity of the sink drone is expressed as Where B is the bandwidth, The source UAV transmits the signal, The interference source transmits a signal. is the Gaussian white noise power. To ensure that the UAV communication link is not interrupted, the sink UAV communication capacity needs to meet the following constraints: The time slot of the drone to the destination is expressed as The drone needs to reach the destination within the maximum mission time T and needs to meet the following constraints: Under the constraints of the mission area, flight speed, distance between drones and communication capacity between drones, the drone trajectory optimization problem with the goal of maximizing the communication capacity between drones can be modeled as follows: 。 6. The method for optimizing flight trajectory of robust communication between unmanned aerial vehicles under malicious interference according to claim 1 is characterized in that: In step S5, the TD3 deep reinforcement learning framework needs to formulate the action space, state space and reward function of the drone communication environment under aerial malicious interference, input the environmental parameters into the TD3 deep reinforcement learning framework for iterative training, and obtain the optimal flight strategy. The specific steps include: S501. The TD3 deep reinforcement learning framework used is suitable for continuous action space. In a two-dimensional plane environment, the drone can adjust the horizontal flight speed and forward angle by itself, so the action space of the drone can be defined as: S502. Communication capacity of drones Distance from drone Directly affects the communication quality of the drone, and the drone needs to monitor the distance to the end point at all times It is guaranteed to arrive at the destination within the Mth time slot, so the state space of the drone communication environment under aerial malicious interference can be defined as: The distance vector that the drone moves in each time slot is The discretization is expressed as Therefore, the update of the drone position can be expressed as: State variables and The update of is related to the position of the UAV, among which the relationship between the UAV communication capacity and the UAV position is expressed as follows: S503. Constructing the reward function requires ensuring that the action decision model after iteration can meet the goal (7a) and constraints (7e), (7f) and (7g): The reward function can be divided into three sub-rewards. For (7a) and (7f), the sub-rewards For (7e), in principle, when Appropriate approach D min When the communication quality is good, the reward should be high. Too close to D min When the risk of drone collision is high, the reward should be very low, and the sub-reward This relationship can be well described, where μ1, μ2, μ3>0, r t (2) (D min )=0,r t (2) (D max )=0; For (7g), the constraint on the time slot of reaching the destination can be converted into a constraint on the distance between the drone and the destination, and the sub-reward So the reward function of this system can be defined as: Among them, ρ1, ρ2, ρ3 are reward factors; S504. Initialize the Actor network π in the TD3 deep reinforcement learning framework φ Parameter φ, Critic1 network With Critic 2 Network Parameters and corresponding Target network π φ′ , and parameter; Based on the network parameters, input the current state s to the Actor network and obtain the action a=π φ (s)+∈,∈~N(0,σ). Input action a into the environment, obtain reward r and next state s′. Store (s,a,r,s′) into the experience replay buffer; input s and a into the dual critic network to obtain Q value Input the expected state s′ to the Target Actor network to obtain the expected action The expected action Input the dual Target Critic network to obtain the target Q value In each cycle, the model extracts N sets of samples from the experience replay buffer, calculates the Q value and the target Q value, and uses Q θ The minimum mean square error between (s, a) and y is used as the loss function, and the gradient descent method is used to update the Critic network parameters θ i ; Periodically, every d cycles, the model extracts N sets of samples from the experience replay buffer, calculates the expectation of the Q value, and uses the gradient ascent method to update the Actor network parameters φ, θ i , the update of φ can be expressed as: Where α, β are learning rates; regularly, every d cycles, update the Target network parameter θ i ′,φ′, the parameter update can be expressed as: i i ′←τθ i +(1-τ)θ i ′(13a)φ′←τφ+(1-τ)φ′(13b) Among them, τ is a hyperparameter less than 1; the above steps are iterated repeatedly to optimize the network parameters until the reward converges, thereby maximizing the communication capacity of the drone under malicious interference.

Images