Anti-interference uav trajectory control method and system based on reinforcement learning

Abstract

This invention discloses a method and system for anti-interference drone trajectory control based on reinforcement learning. The method includes the following steps: Step 1, setting drone data; Step 2, calculating drone energy consumption; Step 3, using a reinforcement learning algorithm to explore and search for the best possible trajectory in an unknown interference environment. This invention provides a method and system for anti-interference drone trajectory control in unknown environments. The drone offloads the collected data to a browser / base (BS) to achieve real-time data transmission, reduce the drone's cache pressure, and ensure that the data is processed in a timely manner.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) trajectory control technology, specifically relating to a trajectory control method and system based on reinforcement learning, for exploring unknown interference environments and achieving autonomous trajectory planning. Background Technology

[0002] Due to their advantages such as rapid deployment and low cost, drones are considered a promising option for data collection and emergency communications. However, drones require the assistance of base stations (BS) during missions. BSs provide access channels to deliver trajectory control information, and drones also need to promptly unload the collected data back to the BS to save their cache space for the next mission. Drones and BSs can coordinate with each other to form a drone-BS network.

[0003] Existing technologies primarily address trajectory optimization through convex optimization techniques and centralized optimization, rather than distributed and automatic adjustments. However, global information is sometimes difficult to obtain, especially in uncertain and disturbing environments, and joint optimization problems considering anti-interference requirements have rarely been studied in existing technologies. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a reinforcement learning-based anti-jamming UAV trajectory control method and system. The purpose of this invention is to enable UAVs to achieve autonomous trajectory planning in unknown interference environments, and to formulate the trajectory control problem of UAVs in anti-jamming satellite-UAV coordination networks as a Markov decision process problem.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for anti-jamming UAV trajectory control based on reinforcement learning includes the following steps:

[0007] Step S1: Set up drone data;

[0008] Step S2: Calculate the drone's energy consumption;

[0009] Step S3: Use reinforcement learning (RL) algorithms to process the exploration and unknown interference environment and search for the best possible trajectory.

[0010] Preferably, in step 1, the UAV data includes the original position and target position parameters of the UAV;

[0011] The UAV possesses environmental awareness capabilities, enabling it to detect its surroundings. The collected reconnaissance data and current geographical location are transmitted to a satellite via an uplink, while the satellite transmits nearby beam information and trajectory control information to the UAV via a downlink. Without loss of generality, this invention considers a three-dimensional coordinate system of the UAV flying at a fixed altitude on the ground, employing a Manhattan network model to model the target area. The Manhattan network model is a well-known network model that primarily divides the task area into multiple grids for ease of modeling. The Manhattan model uses a probabilistic method when selecting node movements, making it suitable for pathfinding models. The coordinates projected onto the horizontal plane at the current and next positions are defined as follows:

[0012] l t =[x t ,y t ]and l t+1 =[x t+1 ,y t+1 ], (1)

[0013] Define the speed v of the drone u ;

[0014] The drone can only choose from four movement options: forward, backward, left, and right. Entering an adjacent grid area (ie,‖l) t -l t+1 ‖≤1, the left side of the equation represents l t to l t+1 The distance to Manhattan can be obtained as follows:

[0015] ||l t -l t+1 || = |x t -x t+1 |+|y t -y t+1 |, (2)

[0016] Preferably, step 2, calculate the drone's energy consumption:

[0017] The UAV moves parallel to the ground. Each grid represents a mission area unit. The UAV's total energy consumption consists of two parts: propulsion energy consumption and data offloading energy consumption, which is mainly related to the offloaded data flow and the corresponding channel status.

[0018] The propulsion energy consumption of the drone is:

[0019] S v =e1L, (3)

[0020] Where e1 is the unit flight energy consumption, and L is the Manhattan distance traveled by the drone. To reach the next data collection point in time, half of the energy reserve is used to ensure the flight distance; therefore, the drone can fly up to L. th .

[0021] Data offloading energy consumption is:

[0022] S c,i =e2D i (4)

[0023] D i =C i T i ,

[0024] Among them, D i Let e2 be the unloaded data traffic of the i-th grid, and e2 be the energy consumption per unit acquisition bit. Due to beam quality degradation or interference, the UAV needs to frequently switch beam grids to enter grids with better channel conditions. The problem studied in this invention is that the UAV needs to maintain an energy reserve E during point-to-point flight. o Under the constraints of trajectory control and beam switching, as much data as possible is unloaded.

[0025]

[0026] st∑ i S c,i +S v ≤E o

[0027] Where ξ=[l s ,l2…l i …,l d [For drones starting from the origin l] s trajectory to the target d .

[0028] Preferably, in step 3, the target area is divided into grids and a path is planned in a discrete manner. In this case, selecting the next grid is a necessary part of trajectory planning. Furthermore, it is only related to the current grid and the available action space, and not to the previous selection; therefore, it can be viewed as a Markov Decision Problem (MDP). The anti-interference trajectory control problem can be modeled as a 4-tuple MDP with a state space S, an operation space A, a reward function R, and dynamic transition probabilities Pr. An anti-interference trajectory control algorithm based on reinforcement learning is used to find the optimal trajectory. The state space is defined as the current position: {s} t ∈S|S=l1,l2,…,L N}, where N = X × Y is the total number of grid cells, and X and Y are the corresponding two-dimensional side lengths. The action space is defined as:

[0029] {a t ∈A|A=a1,a2,a3,a4}, (6)

[0030] Where a1, a2, a3, and a4 represent the actions of "up," "down," "left," and "right," respectively. The reward function is defined as the unloaded traffic data in the current grid.

[0031] r t =T t C t (7)

[0032] Where T t and C t These are the unloading time and unloading rate. The Q function of the drone is represented as Q(s). t ,a t ), updated to Q t+1 as follows:

[0033]

[0034] Where s′ t It is through action a t The state after, and a′ t It is the next possible action to choose.

[0035] π(t) = [π1(t), π2(t)…π4(t)] is a mixed strategy for action selection, where π n (t) represents the action to be selected.

[0036] a n The probability of ∈ [a1, a2, a3, a4]. In particular, π n (t) is given by the following formula:

[0037]

[0038] Where τ is a parameter of the Boltzmann model.

[0039]

[0040]

[0041] Among them, τ0 is related to the exploration time. This represents the termination condition of the exploration phase. ν is a visible variable in the Boltzmann machine that influences the transition from exploration to decision-making.

[0042] A further preferred reinforcement learning algorithm in step S3 is as follows:

[0043] S31: Obtain and input the original and target position parameters;

[0044] The original position is denoted as l s The target location is denoted as l. d ;

[0045] S32: Define the state space S, operation space A, and reward function R;

[0046] Define the action space as {a t ∈A|A=a1,a2,a3,a4}, where a1,a2,a3,a4 represent the actions of "up", "down", "left", and "right";

[0047] Define the state space as the current position: {s t ∈S|S=l1,l2,…,L N}, where N = X × Y is the total number of grid cells, and X and Y are the corresponding two-dimensional side lengths;

[0048] Define the reward function as the traffic data unloaded in the current grid: r t =T t C t T t and C t It refers to uninstallation time and uninstallation rate;

[0049] S33: Initialize Q t and π n (t);

[0050] Q t (s t ,a n ) = 0, π n (t) = 1 / 4;

[0051] S34: Determine the number of iterations K, start the iteration and record t = 1, ξ = [l s ];

[0052] S35: When s t ≠l d And t≤L th At that time, the drone is at its current location s t Obtain a subset of available actions from the state space. And from it based on π n (t) Select action a t Move to the next position s′ t And update the trajectory, denoted as ξ=[l s ,…,s t ,s′ t ];

[0053] S36: The drone is at the next location. t Unload traffic and get a reward. t ;

[0054] S37: Drone Parameter Update Q t+1 (s t ,a n ) and π n (t);

[0055]

[0056] S38: After the drone finishes learning, if a feasible trajectory is found, each selected action on the trajectory will be given a delay to return to its Q value R0; R0 is the reward obtained by the selected result; Q is Q(s,a), which is the expected reward of taking action a (a∈A) in state s (s∈S) at a certain moment.

[0057] S39: Output UAV trajectory ξ and total unload data ∑ i D i .

[0058] This invention also discloses a system based on the above-described anti-interference UAV trajectory control method, which includes the following modules:

[0059] Drone data settings module: Configure drone data;

[0060] Drone energy consumption calculation module: Calculates the energy consumption of drones;

[0061] The optimal possible trajectory acquisition module uses reinforcement learning to explore and search for the optimal possible trajectory in an environment with unknown interference.

[0062] In the context of data collection tasks performed in an air-to-ground coordinated UAV network, the unknown and uncertain environment makes it difficult for UAVs to effectively and dynamically adjust their flight trajectories. Furthermore, the presence of interference further exacerbates the complexity of trajectory control. Therefore, this invention provides a reinforcement learning-based anti-interference UAV trajectory control method and system. Compared to existing technologies, this invention offers the following technical advantages:

[0063] (1) The anti-interference trajectory control method and system in unknown environment provided by the present invention allows the UAV to unload the collected data to the BS, realize the real-time transmission of data, reduce the buffer pressure of the UAV, and ensure that the data is processed in a timely manner.

[0064] (2) A preferred embodiment of the technical solution of the present invention relates to an anti-interference trajectory control method based on RL, which can explore unknown interference environments and realize autonomous trajectory control. Its algorithm complexity is reduced, efficiency is improved, and control is effective. Attached Figure Description

[0065] Figure 1This is a flowchart of a preferred embodiment of the present invention: an anti-interference UAV trajectory control method based on reinforcement learning.

[0066] Figure 2 The Manhattan network model;

[0067] Figure 3 This is a block diagram of a reinforcement learning-based anti-interference drone trajectory control system, which is a preferred embodiment of the present invention. Detailed Implementation

[0068] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other. Embodiment 1

[0069] like Figure 1 As shown in the figure, the anti-interference UAV trajectory control method based on reinforcement learning in this embodiment includes the following steps:

[0070] Step S1: Set up drone data;

[0071] Step S2: Calculate the drone's energy consumption;

[0072] Step S3: Use reinforcement learning algorithms to explore and deal with unknown interference environments and search for the best possible trajectory.

[0073] In step S1 of this embodiment, the UAV data includes the original position and target position parameters of the UAV.

[0074] Considering the three-dimensional coordinate system of the UAV flying at a fixed altitude on the ground, a Manhattan network model is used to model the target area;

[0075] Define the coordinates of the projection onto the horizontal plane at the current and next positions:

[0076] l t =[x t ,y t ]and l t+1 =[x t+1 ,y t+1 (1)

[0077] Among them, l t and l t+1 Let x be the mapping of the coordinate functions at time t and time t+1, respectively. t and x t+1Let y be the horizontal coordinates projected onto the horizontal plane at time t and time t+1, respectively. t and y t+1 These are the ordinates projected onto the horizontal plane at time t and time t+1, respectively.

[0078] Define the speed v of the drone u ;

[0079] The drone can choose from four movement options: forward, backward, left, and right; it can also enter adjacent grid areas. t -l t+1 ‖≤1, the left side of the first expression represents l t to l t+1 The distance to Manhattan is:

[0080] ||l t -l t+1 || = |x t -x t+1 |+|y t -y t+1 | (2).

[0081] Step S2 in this embodiment is as follows: The total energy consumption of the UAV includes: propulsion energy consumption and data offloading energy consumption;

[0082] The propulsion energy consumption of the drone is:

[0083] S v =e1L (3)

[0084] Where e1 is the unit flight energy consumption, and L is the Manhattan distance traveled by the drone;

[0085] Data offloading energy consumption is:

[0086] S c,i =e2D i (4)

[0087] D i =C i T i

[0088] Among them, C i T represents the data transmission rate of the grid calculated using Shannon's formula. i The time for transmitting task data within this grid; D i e2 represents the unloading data flow of the i-th grid, and e2 represents the energy consumption per unit acquisition bit.

[0089] The energy storage E of the drone during point-to-point flight o Under these constraints, trajectory control and beam switching off as much data as possible:

[0090]

[0091] st∑ i S c,i +S v ≤E o

[0092] Where, r i r represents the signal-to-noise ratio of the receiver from the i-th grid. th The signal-to-noise ratio (SNR) indicates the minimum required for normal operation. δ represents the conditional decision: when there are no available channels in a grid cell, or when the channel fading in that grid is very severe, the UAV cannot enter that grid (δ = 0); otherwise, δ = 1. c,i S represents the energy required for the UAV to complete data transmission in the i-th grid. v ξ represents the energy required for the drone's movement; [l] s ,l2…l i …,l d [For drones starting from the origin l] s trajectory to the target d .

[0093] In step S3 of this embodiment, the target area is divided into grids and path planning is performed in a discrete manner. In this case, selecting the next grid is a necessary planning for the trajectory. At the same time, this selection is only related to the current grid and the available action space, and is not related to the previous selection. Therefore, it is regarded as a Markov decision problem. The anti-interference trajectory control problem is modeled as a 4-tuple Markov decision problem, with a state space S, an operation space A, a reward function R, and a dynamic transition probability Pr. The optimal trajectory is found through an anti-interference trajectory control algorithm based on reinforcement learning.

[0094] The state space is defined as the current position: {s t ∈S|S=l1,l2,…,L N}, where N = X × Y is the total number of grid cells, and X and Y are the corresponding two-dimensional side lengths;

[0095] Action space is defined as:

[0096] {a t ∈A|A=a1,a2,a3,a4} (6)

[0097] Among them, a t This indicates the action the drone will choose at time t; a1, a2, a3, a4 represent the actions up, down, left, and right, respectively.

[0098] The reward function is defined as the traffic data unloaded in the current grid:

[0099] r t =Tt C t (7)

[0100] Among them, T t and C t It refers to uninstallation time and uninstallation rate;

[0101] The Q-function of the drone is represented as Q(s) t ,a t ), updated to Q t+1 as follows:

[0102]

[0103] Among them, s t This represents the state at time t, which in the model is the grid where the UAV is located; α is the learning efficiency, (1-α)Q t (s t ,a t ) represents the proportion of the old Q value in the new Q; The rewards learned from this action include the rewards derived from the action itself and potential future rewards, where η is the discount factor; s t ′ is after action a t The state after, and a′ t This is the next possible action to choose;

[0104] π(t) = [π1(t), π2(t)…π4(t)] is a mixed strategy for action selection, where π n (t) represents the action to be selected, a. n The probability of ∈ [a1,a2,a3,a4].

[0105] π n (t) is given by the following formula:

[0106]

[0107] Among them, a n′ This expression represents the joint probability distribution of the model in a Boltzmann machine, indicating the next possible action. This indicates that the partition function is used for normalization;

[0108] τ is a parameter of the Boltzmann model;

[0109]

[0110]

[0111] Where e is the natural constant, t is the current time, and τ0 is the initial probability distribution of the Boltzmann machine, which is related to the exploration time. represents the end condition of the exploration phase; v is the visible variable in the Boltzmann machine, affecting the transition from exploration to decision-making.

[0112] The reinforcement learning algorithm in step S3 of this embodiment is as follows:

[0113] S31: Obtain and input the original and target position parameters;

[0114] The original position is denoted as l s The target location is denoted as l. d ;

[0115] S32: Define the state space S, operation space A, and reward function R;

[0116] Define the action space as {a t ∈A|A=a1,a2,a3,a4}, where a1,a2,a3,a4 represent the actions of "up", "down", "left", and "right";

[0117] Define the state space as the current position: {s t ∈S|S=l1,l2,…,L N}, where N = X × Y is the total number of grid cells, and X and Y are the corresponding two-dimensional side lengths;

[0118] Define the reward function as the traffic data unloaded in the current grid: r t =T t C t T t and C t It refers to uninstallation time and uninstallation rate;

[0119] S33: Initialize Q t and π n (t);

[0120] Q t (s t ,a n ) = 0, π n (t) = 1 / 4;

[0121] S34: Determine the number of iterations K, start the iteration and record t = 1, ξ = [l s ];

[0122] S35: When s t ≠l d And t≤L th At that time, the drone is at its current location s t Obtain a subset of available actions from the state space. And from it based on π n (t) Select action a tMove to the next position s′ t And update the trajectory, denoted as ξ=[l s ,…,s t ,s′ t ];

[0123] S36: The drone is at the next location. t Unload traffic and get a reward. t ;

[0124] S37: Drone Parameter Update Q t+1 (s t ,a n ) and π n (t);

[0125]

[0126] S38: After the drone finishes learning, if a feasible trajectory is found, each selected action on the trajectory will be given a delay to return to its Q value R0; R0 is the reward obtained by the selected result; Q is Q(s,a), which is the expected reward of taking action a (a∈A) in state s (s∈S) at a certain moment.

[0127] S39: Output UAV trajectory ξ and total unload data ∑ i D i .

[0128] Example 2:

[0129] This embodiment applies to satellite coverage models, and the model diagram is referenced below. Figure 2 As shown, in this network, drones and satellite nodes transmit tasks and achieve autonomous trajectory optimization.

[0130] like Figure 3 As shown, this embodiment of the anti-interference UAV trajectory control system based on reinforcement learning includes a UAV data setting module, a UAV energy consumption calculation module, and an optimal possible trajectory acquisition module; specifically:

[0131] Drone data settings module: Sets drone data; drone data includes the drone's original position and target position parameters;

[0132] The drone possesses environmental awareness capabilities, enabling it to detect its surroundings. The collected reconnaissance data and current geographical location are transmitted to a satellite via an uplink, while the satellite transmits nearby beam information and trajectory control information to the drone via a downlink. Without loss of generality, this invention considers a three-dimensional coordinate system of the drone flying at a fixed altitude on the ground, and uses a Manhattan network model to model the target area.

[0133] Define the coordinates of the projection onto the horizontal plane at the current and next positions:

[0134] l t =[x t ,y t ]and l t+1 =[x t+1 ,y t+1 ], (1)

[0135] Define the speed v of the drone u ;

[0136] The drone can only choose from four movement options: forward, backward, left, and right. Entering an adjacent grid area (ie,‖l) t -l t+1 ‖≤1, the left side of the equation represents l t to l t+1 The distance to Manhattan can be obtained as follows:

[0137] ||l t -l t+1 || = |x t -x t+1 |+|y t -y t+1 |, (2)

[0138] Drone energy consumption calculation module: Calculates the energy consumption of drones;

[0139] The UAV moves parallel to the ground. Each grid represents a mission area unit. The UAV's total energy consumption consists of two parts: propulsion energy consumption and data offloading energy consumption, which is mainly related to the offloaded data flow and the corresponding channel status.

[0140] The propulsion energy consumption of the drone is:

[0141] S v =e1L, (3)

[0142] Where e1 is the unit flight energy consumption, and L is the Manhattan distance traveled by the drone. To reach the next data collection point in time, half of the energy reserve is used to ensure the flight distance; therefore, the drone can fly up to L. th .

[0143] Data offloading energy consumption is:

[0144] S c,i =e2D i (4)

[0145] D i =C i T i ,

[0146] Where D i e2 represents the offloaded data traffic for the i-th grid, and e2 represents the power consumption per unit acquisition bit. Due to beam quality degradation or interference, the UAV needs to frequently switch beam grids to enter grids with better channel conditions.

[0147] The problem addressed by this invention is the need for unmanned aerial vehicles (UAVs) to maintain energy reserves (E) during point-to-point flight. o Trajectory control and beam switching under constraints should unload as much data as possible. Modeling this problem yields:

[0148]

[0149] st∑ i S c,i +S v ≤E o

[0150] Where ξ=[l s ,l2…l i …,l d [For drones starting from the origin l] s trajectory to the target d .

[0151] Optimal Possible Trajectory Acquisition Module: Explores and searches for the best possible trajectory in an environment with unknown interference through RL processing;

[0152] The target region is divided into discrete grids and path planning is performed. In this case, selecting the next grid is a necessary part of trajectory planning. Furthermore, it depends only on the current grid and the available action space, and is independent of the previous selection; therefore, it can be viewed as an MDP problem.

[0153] The anti-interference trajectory control problem can be modeled as a 4-tuple MDP, with a state space S, an operation space A, a reward function R, and a dynamic transition probability Pr. An anti-interference trajectory control algorithm based on reinforcement learning is used to find the optimal trajectory.

[0154] The state space is defined as the current position: {s t ∈S|S=l1,l2,…,L N}, where N = X × Y is the total number of grids, and X and Y are the corresponding two-dimensional side lengths.

[0155] Action space is defined as:

[0156] {a t ∈A|A=a1,a2,a3,a4}, (6)

[0157] Where a1, a2, a3, and a4 represent the actions of "up", "down", "left", and "right".

[0158] The reward function is defined as the traffic data unloaded in the current grid:

[0159] r t =T t C t (7)

[0160] Where T t and C t It refers to uninstallation time and uninstallation rate.

[0161] The Q-function of the drone is represented as Q(s) t ,a t ), updated to Q t+1 as follows:

[0162]

[0163] Where s′ t It is through action a t The state after, and a′ t It is the next possible action to choose.

[0164] π(t) = [π1(t), π2(t)…π4(t)] is a mixed strategy for action selection, where π n (t) represents the action to be selected, a. n The probability of ∈ [a1,a2,a3,a4].

[0165] In particular, π n (t) is given by the following formula:

[0166]

[0167] Where τ is a parameter of the Boltzmann model.

[0168]

[0169]

[0170] Among them, τ0 is related to the exploration time. This represents the end condition of the exploration phase. ν affects the transition from exploration to decision-making.

[0171] In the optimal possible trajectory acquisition module of this embodiment, the reinforcement learning algorithm (RL) is as follows:

[0172] Obtain and input the original and target position parameters.

[0173] The original position is denoted as l s The target location is denoted as l. d .

[0174] Define the state space S, the operation space A, and the reward function R.

[0175] Define the action space as {a t Let {a_i ∈ A | A = a_1, a_2, a_3, a_4}, where a_1, a_2, a_3, a_4 represent the actions "up", "down", "left", and "right". Define the state space as the current position: {s} t ∈S|S=l1,l2,…,L N}, where N = X × Y is the number of all grids, and X and Y are the corresponding two-dimensional side lengths. The reward function is defined as the unloaded flow data in the current grid: r t =T t C t T t and C t It refers to uninstallation time and uninstallation rate.

[0176] Initialize Q t and π n (t).

[0177] Q t (s t ,a n ) = 0, π n (t) = 1 / 4.

[0178] Determine the number of iterations K, start the iteration and record t = 1, ξ = [l s ].

[0179] When s t ≠l d And t≤L th At that time, the drone is at its current location s t Obtain a subset of available actions from the state space. And from it based on π n (t) Select action a t Move to the next position s′ t And update the trajectory, denoted as ξ=[l s ,…,s t ,s′ t ].

[0180] The drone is at the next location s′ t Unload traffic and get a reward. t .

[0181] Drone update parameters Q t+1 (s t ,a n ) and π n (t).

[0182]

[0183] After completion, if a feasible trajectory is found, each selected action on the trajectory will be given a delay to return R0 to its Q value.

[0184] Output the drone trajectory ξ and total unload data ∑ i D i .

[0185] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A reinforcement learning-based trajectory control method for anti-jamming unmanned aerial vehicles, characterized by: Includes the following steps: Step S1: Set up drone data; Step S2: Calculate the drone's energy consumption; Step S3: Use reinforcement learning algorithms to process the exploration and unknown interference environment and search for the best possible trajectory; In step S3, the target area is divided into grids and path planning is performed in a discrete manner. Selecting the next grid is a necessary planning for the trajectory. This selection is only related to the current grid and the available action space, and is independent of the previous selection. Therefore, it is regarded as a Markov decision problem. The anti-interference trajectory control problem is modeled as a 4-tuple Markov decision problem, with a state space S, an operation space A, a reward function R, and a dynamic transition probability Pr. The optimal trajectory is found through an anti-interference trajectory control algorithm based on reinforcement learning. The state space is defined as the current position: ,in This represents the total number of grid cells, where X and Y are the corresponding two-dimensional side lengths. Action space is defined as: （6） in, This indicates the action the drone will choose at time t; These represent the actions of going up, down, left, and right, respectively. The reward function is defined as the traffic data unloaded in the current grid: （7） Among them, T t and C t These are uninstallation time and uninstallation rate, respectively. The Q function of the drone is expressed as Updated to Q t+1 as follows: （8） Among them, s t Let represent the state at time t, which in the model is the grid where the drone is located; α is the learning efficiency. It is the proportion of the old Q value in the new Q; The reward learned from this action is the reward brought by the action itself and the potential future reward, where η is the discount factor; It is through action The state after that, This is the next possible action to choose; It is a hybrid strategy for action selection. Indicates the selection action The probability of; It is given by the following formula: （9） in, Indicates the next possible action, and equation (9) represents the joint probability distribution of the model in a Boltzmann machine. This indicates that the partition function is used for normalization; These are the parameters of the Boltzmann model; （10） Where e is the natural constant, and t is the current time; This represents the initial probability distribution of the Boltzmann machine, which is related to the exploration time. represents the end condition of the exploration phase; v is the visible variable in the Boltzmann machine, affecting the transition from exploration to decision-making.

2. The anti-interference UAV trajectory control method based on reinforcement learning as described in claim 1, characterized in that: step In S1, the UAV data includes the UAV's original position and target position parameters; Considering the three-dimensional coordinate system of the UAV flying at a fixed altitude on the ground, a Manhattan network model is used to model the target area; Define the coordinates of the projection onto the horizontal plane at the current and next positions: （1） Among them, l t and l t+1 Let x be the mapping of the coordinate functions at time t and time t+1, respectively. t and x t+1 Let y be the horizontal coordinates projected onto the horizontal plane at time t and time t+1, respectively. t and y t+1 These are the ordinates projected onto the horizontal plane at time t and time t+1, respectively. Define the speed v of the drone u ; The drone has four movement options: forward, backward, left, and right; it can also enter adjacent grid areas. The left side of the first expression represents l t to l t+1 The distance to Manhattan is: （2）。 3. The anti-interference UAV trajectory control method based on reinforcement learning as described in claim 2, characterized in that: Step S2 is as follows: The total energy consumption of the UAV includes: propulsion energy consumption and data offloading energy consumption; The propulsion energy consumption of the drone is: （3） Where e1 is the unit flight energy consumption, and L is the Manhattan distance traveled by the drone; Data offloading energy consumption is: （4） Among them, C i T is the data transmission rate of the grid calculated using Shannon's formula. i The time for transmitting task data within this grid; D i e2 represents the unloading data flow of the i-th grid, and e2 represents the energy consumption per unit acquisition bit. Under the constraint of energy reserve E0 during point-to-point flight, the drone should offload as much data as possible through trajectory control and beam switching. （5） Where, r i r represents the signal-to-noise ratio from the receiver in the i-th grid. th This represents the signal-to-noise ratio at which the system functions properly. This indicates a conditional decision: when there are no available channels in a grid cell, or when the channel fading in that grid is very severe, the drone cannot enter that grid. ;on the contrary ; For drones from the starting point l s trajectory to the target d .

4. The anti-interference UAV trajectory control method based on reinforcement learning as described in any one of claims 1-3, characterized in that: The reinforcement learning algorithm in step S3 is as follows: S31: Obtain and input the original and target position parameters; The original position is denoted as l s The target location is denoted as l. d ; S32: Define the state space S, operation space A, and reward function R; Define the action space as ; Define the state space as the current position: ; Define the reward function as the traffic data unloaded in the current grid: ; S33: Initialization ; S34: Determine the iteration count K, start the iteration and record... ; S35: When At that time, the drone is at its current location s t Obtain a subset of available actions from the state space. and from it based on Select Action Move to the next position And update the trajectory, recorded as ; S36: The drone is at the next location. Unload traffic and get a reward. t ; S37: Drone Parameter Update ; S38: After the drone finishes learning, if a feasible trajectory is found, each selected action on the trajectory will be given a delay to return R0 to its Q value; R0 is the reward obtained by the selection result; Q is Q(s,a), which is the expected reward that can be obtained by taking action a in state s at a certain time, s∈S, a∈A; S39: Output drone trajectory Total uninstallation data .

5. A system based on the anti-interference unmanned aerial vehicle trajectory control method according to any one of claims 1-4, characterized in that: Includes the following modules: Drone data settings module: Configure drone data; Drone energy consumption calculation module: Calculates the energy consumption of drones; The optimal possible trajectory acquisition module uses reinforcement learning to explore and search for the optimal possible trajectory in an environment with unknown interference.

Images