An improved q-learning based unmanned aerial vehicle path planning method

Abstract

In the unmanned aerial vehicle path planning method and device based on improved Q learning and the storage medium provided in the application, the unmanned aerial vehicle flight environment is modeled in three dimensions by a three-dimensional grid method, and the unmanned aerial vehicle flight space position unit is divided into the same cubic shape; based on the three-dimensional environment modeling, a Markov decision process model for solving the unmanned aerial vehicle path planning is constructed; based on the Markov decision process model and the task parameters of the unmanned aerial vehicle flight, the optimal path planning of the unmanned aerial vehicle is obtained through a saturated state search strategy and an improved Q learning algorithm; and the optimal path planning of the unmanned aerial vehicle is input into the three-dimensional environment modeling to obtain the path simulation result of the optimal path of the unmanned aerial vehicle. Thus, in the process of obtaining the optimal path planning of the unmanned aerial vehicle through the saturated state search strategy and the improved Q learning algorithm, the improved Q learning algorithm reduces the amount of parameters to be learned, improves the learning efficiency, accelerates the convergence speed of the algorithm, and thus saves resources.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) trajectory planning technology, and in particular to a UAV trajectory planning method, apparatus and storage medium based on improved Q-learning. Background Technology

[0002] As a novel remote sensing monitoring platform, unmanned aerial vehicles (UAVs) have been widely used in various fields. However, the flight environment during UAV operation is often unknown and complex, necessitating the planning of UAV flight paths to ensure accurate flight in such conditions.

[0003] In related technologies, swarm optimization algorithms (such as worker bee swarm optimization and gray wolf optimization) are used to design UAV trajectories. However, this algorithm requires prior knowledge of environmental information and cannot be applied to unknown environments in UAV trajectory planning. Alternatively, reinforcement learning algorithms can be used to autonomously learn and construct evaluation functions, continuously optimizing them through ongoing interactions with the environment. However, reinforcement learning algorithms have low learning efficiency, typically requiring millions of repeated training iterations to achieve good results, and their convergence speed is slow. Summary of the Invention

[0004] This application provides a UAV trajectory planning method based on improved Q-learning to solve the technical problems mentioned above.

[0005] The first aspect of this application proposes a method for unmanned aerial vehicle (UAV) trajectory planning based on improved Q-learning, the method comprising:

[0006] The UAV flight environment is modeled in three dimensions using a three-dimensional mesh method, and the UAV flight space position unit is divided into the same cube shape.

[0007] Based on the three-dimensional environment modeling, a Markov decision process model for solving the UAV path planning is constructed.

[0008] Based on the Markov decision process model and the mission parameters of the UAV flight, the optimal UAV trajectory planning is obtained through a saturated state search strategy and an improved Q-learning algorithm.

[0009] The optimal flight path plan of the UAV is input into the three-dimensional environment model to obtain the path simulation results of the optimal flight path of the UAV.

[0010] Optionally, constructing the Markov decision process model for solving the UAV path planning includes: constructing a UAV flight action space representation and a reward function.

[0011] Optionally, the reward function includes:

[0012] R(s, a, s′)=R obstale -Rt hreat +R destination ;

[0013] Where s is the current state, s′ represents the next state after performing action a, and R(s, a, s′) is the reward for the drone to reach s′ after performing action a in state s. obstale R represents the obstacle reward function. threat R represents the threat-reward function. destination This indicates the target reward.

[0014] Optionally, the improved Q-learning algorithm includes an SQ table and a Q table; the step of obtaining the optimal UAV trajectory planning based on the Markov decision process model and the UAV flight mission parameters, through a saturation state search strategy and the improved Q-learning algorithm, includes:

[0015] Obtain the mission parameters for the UAV flight, wherein the mission parameters include the target point and the starting point;

[0016] Determine the endpoint state and initial state corresponding to the target point and the starting point, respectively;

[0017] Initialize the SQ table and the Q table, wherein in the initial Q table, the Q value corresponding to each state action is initialized to zero, and in the initial SQ table, the Q value corresponding to each state target action is initialized to zero;

[0018] Determine the current state S current Is it in a saturated state?

[0019] If the S is determined current If the state is not saturated, the exploration strategy generates action a, and the UAV enters the next state S by executing action a. next Update S in the Q table current Execute the Q value of action a, and then set the S value. current Replace the S next ;

[0020] If the S is determined current If it is in a saturated state, then determine the S. current The corresponding target action is determined, and the S in the SQ table is updated. current The corresponding Q value indicates the end of this trajectory training.

[0021] Repeat the above operations until the state corresponding to the starting point is a saturated state, and end the training. Then, determine the target action corresponding to each state in the SQ table as the optimal trajectory planning for the UAV.

[0022] Optionally, the step of determining the current state S currentWhether it is in a saturated state includes:

[0023] If any of the optional actions corresponding to the current state can directly reach the target point, then the current state is determined to be a saturated state; or

[0024] If all the optional actions corresponding to the current state point to the next state, which is a saturated state, then the current state is determined to be a saturated state.

[0025] Optionally, if the S is determined current If it is in a saturated state, then determine the S. current The corresponding target action is determined, and the S in the SQ table is updated. current The corresponding Q values ​​include:

[0026] If the S is determined current If it is in a saturated state, then determine the S. current All corresponding executable actions and the Q value corresponding to each executable action;

[0027] The S current The executable action with the largest Q value among all executable actions is determined as S. current The corresponding target action, and the S in the SQ table current The corresponding Q value is updated to the Q value corresponding to the target action.

[0028] Optionally, inputting the optimal trajectory plan of the UAV into the 3D environment modeling to obtain the path simulation results of the optimal trajectory of the UAV includes:

[0029] In determining the optimal flight path of the UAV, the actions that the UAV performs in each state are as follows:

[0030] Based on the mission parameters of the UAV flight, the execution actions of the UAV in each state are input into the three-dimensional environment model to obtain the path simulation results of the optimal flight path of the UAV.

[0031] A second aspect of this application provides a UAV trajectory planning device based on improved Q-learning, the device comprising:

[0032] The modeling module is used to perform three-dimensional environmental modeling of the UAV flight environment using the three-dimensional mesh method, and to divide the UAV flight space position unit into the same cubic shape;

[0033] A construction module is used to build a Markov decision process model for solving the path planning of the UAV based on the modeling of the three-dimensional environment.

[0034] The processing module is used to obtain the optimal flight path plan for the UAV based on the Markov decision process model and the mission parameters of the UAV flight, through a saturated state search strategy and an improved Q-learning algorithm.

[0035] The output module is used to input the optimal flight path plan of the UAV into the three-dimensional environment modeling to obtain the path simulation results of the optimal flight path of the UAV.

[0036] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0037] The UAV trajectory planning method, apparatus, and storage medium proposed in this application based on improved Q-learning include: 3D environment modeling of the UAV flight environment using a 3D mesh method, dividing the UAV flight space position units into identical cubic shapes; constructing a Markov decision process model for solving the UAV path planning based on the 3D environment model; obtaining the optimal UAV trajectory plan through a saturated state search strategy and an improved Q-learning algorithm based on the Markov decision process model and the UAV flight mission parameters; and inputting the optimal UAV trajectory plan into the 3D environment model to obtain the path simulation results of the optimal UAV trajectory. Therefore, in the process of obtaining the optimal UAV trajectory plan through the saturated state search strategy and the improved Q-learning algorithm in this application, the improved Q-learning algorithm reduces the number of parameters that need to be learned, improves learning efficiency, accelerates algorithm convergence speed, and thus saves resources.

[0038] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0039] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0040] Figure 1 This is a flowchart illustrating an improved Q-learning-based UAV trajectory planning method according to an embodiment of this application.

[0041] Figure 2 This is a schematic diagram of a process for confirming saturation state according to an embodiment of this application;

[0042] Figure 3 This is a schematic diagram of the structure of an unmanned aerial vehicle (UAV) trajectory planning device based on improved Q-learning according to an embodiment of this application. Detailed Implementation

[0043] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0044] The following description, with reference to the accompanying drawings, describes an embodiment of the UAV trajectory planning method and apparatus based on improved Q-learning.

[0045] Example 1

[0046] Figure 1 This is a flowchart illustrating a UAV trajectory planning method based on improved Q-learning according to an embodiment of this application, as shown below. Figure 1 As shown, the method may include:

[0047] Step 101: Model the UAV flight environment in three dimensions using the three-dimensional mesh method, and divide the UAV flight space position unit into the same cube shape.

[0048] In the embodiments of this disclosure, the unknown three-dimensional environment of the UAV flight environment is modeled using a three-dimensional mesh method. This divides the UAV's flight space into identical cubic units, and the UAV can only perform flight maneuvers within one unit at a time. Furthermore, the three-dimensional mesh method is the same as that in the prior art, and will not be described in detail here.

[0049] Step 102: Based on 3D environment modeling, construct a Markov decision process model for solving UAV path planning.

[0050] In one of the embodiments of this disclosure, after obtaining the three-dimensional environment model, a Markov decision process model for solving the UAV path can be constructed based on the three-dimensional environment model.

[0051] Specifically, in the embodiments of this disclosure, constructing a Markov decision process model for solving UAV path planning includes: constructing a UAV flight action space representation and a reward function.

[0052] In the embodiments of this disclosure, the drone differs from other autonomous mobile robots in that it has steering constraints and cannot move backward. Therefore, the actions performed by the drone within a single position unit at each moment can include: forward, up, down, left, right, upper left, lower left, upper right, lower right, upper front, lower front, upper ... 17} serves as the space representation of the drone's flight actions, representing the executable actions of the aforementioned drones.

[0053] Furthermore, in embodiments of this disclosure, the reward function may include:

[0054] R(s, a, s′)=R obstale -R threat +R destination .

[0055] Where s is the current state, s′ represents the next state after performing action a, and R(s, a, s′) is the reward for the drone to reach s′ after performing action a in state s. obstale R represents the obstacle reward function. threat R represents the threat-reward function. destination This indicates the target reward.

[0056] Furthermore, in embodiments of this disclosure, the aforementioned R obstale It can be represented as:

[0057]

[0058] In the embodiments of this disclosure, n obstale This represents the number of times the drone collided with obstacles. Furthermore, if the drone collided with an obstacle, the obstacle reward function is assigned a negative value (-100); if the drone did not collide with an obstacle, the obstacle reward function is assigned a value of 0.

[0059] Furthermore, in embodiments of this disclosure, the above-mentioned R threat It can be represented as:

[0060]

[0061] In the embodiments of this disclosure, n threat b represents the number of all perceived threats. i This represents the distance between the drone and the i-th threat. Specifically, the threat reward value R increases as the drone gets closer to the threat. threat The smaller the value, the greater the threat reward R becomes as the drone moves further away from the threat. threat The larger.

[0062] Furthermore, in embodiments of this disclosure, the above-mentioned R destination It can be represented as:

[0063] R destination =β(d t-1 -d t )

[0064] In the embodiments of this disclosure, the above-mentioned d t d represents the distance between the UAV and the target point at time t. t-1This represents the distance between the drone and the target point at the current time t (the previous time t-1). Also, it represents the target reward value R when the drone approaches the target point. destination It increases; as the drone moves away from the target point, the target reward value R increases. destination It will become smaller.

[0065] Furthermore, in the embodiments of this disclosure, α and β are two weighting parameters that can be set as needed. Specifically, if α > β, it indicates a greater emphasis on the threat posed by obstacles; if α < β, it indicates a greater emphasis on path distance cost.

[0066] Step 103: Based on the Markov decision process model and the mission parameters of the UAV flight, the optimal UAV trajectory plan is obtained through the saturation state search strategy and the improved Q-learning algorithm.

[0067] In the embodiments of this disclosure, the Q-learning algorithm in the prior art is a model-free temporal difference reinforcement learning algorithm. The Q-value function in the Q-learning algorithm is shown in the following equation:

[0068] Q(s,a)=Q(s,a)+ρ[R(s,a,s′)+γmaxQ(s′,a′)-Q(s,a)]

[0069] Where s represents the current state; s′ represents the next state after performing action a; ρ∈(0,1) represents the learning rate, the value of which directly affects the speed at which the Q-learning algorithm learns environmental information; a represents the current action; a′ represents all possible actions that the UAV can perform in the next state s′; γ∈(0,1) is the discount factor, the value of which represents the impact of future planning on the current strategy, and the larger the value of γ, the greater the impact of future planning; R(s,a,s′) represents the reward function. The feedback values ​​obtained by the Q-learning algorithm from the environment are stored in a two-dimensional table Q. The basic unit of the Q-table is the Q value. Similar to the reward function, the Q-table is also determined by (the current state s and the performed action a), with a dimension of m×n, where m is the number of states and n is the number of actions. Q(s,a) represents the feedback value obtained by the UAV from the environment after performing action a in state s. After the Q-learning algorithm is trained, the action corresponding to the maximum Q value of the corresponding state can be found in the Q-table, thus determining the best action in the current state. Table 1 is a schematic diagram of a Q-table proposed in this embodiment.

[0070]

[0071] As shown in Table 1, each row in Table 1 represents the Q-value corresponding to the state after performing an executable action. For example, Q(1,1) is the Q-value obtained by the UAV in state s1 after performing action a1. Furthermore, in the embodiments of this disclosure, during the training process of the above-mentioned Q-learning algorithm, the surrounding environment can be learned by randomly selecting actions using a greedy strategy ε-greedy, where each action has an equal probability of being selected. The agent needs to traverse the entire environmental space as much as possible during the search process to find the global optimal solution; this operation is defined as the Q-learning algorithm exploration operation. Alternatively, the Q-learning algorithm may not use random exploration but instead query the values ​​of the Q-table to perform actions. During continuous exploration and development, the Q-learning algorithm traverses all states to ensure the learning of all environmental information, completing the Q-table corresponding to the Q-learning algorithm, so that the action corresponding to the maximum Q-value of the state in the Q-table is determined as the best action in the current state.

[0072] Furthermore, in the embodiments of this disclosure, when the conventional Q-learning algorithm employs a greedy exploration strategy, excessive development in the early stages of training can lead to the continuous learning of suboptimal actions, thus polluting the Q-table and requiring significant learning resources for correction in the later stages of training. Additionally, in the later stages of training, the entire environment has been thoroughly explored, rendering further exploration meaningless, resulting in low learning efficiency and slow convergence. Therefore, by employing a saturation-state search strategy and an improved Q-learning algorithm, the optimal trajectory planning for the UAV can be obtained, thereby improving learning efficiency, accelerating algorithm convergence, and conserving resources.

[0073] Specifically, in the embodiments of this disclosure, the method for obtaining the optimal UAV trajectory planning based on the Markov decision process model and the UAV flight mission parameters, through a saturation state search strategy and an improved Q-learning algorithm, may include the following steps:

[0074] Step 1031: Obtain the mission parameters for the UAV flight;

[0075] In the embodiments disclosed herein, the task parameters include the target point and the starting point.

[0076] Step 1032: Determine the endpoint state and initial state corresponding to the target point and the starting point, respectively;

[0077] Step 1033: Initialize the SQ and Q tables;

[0078] In the embodiments of this disclosure, the improved Q-learning algorithm includes an SQ table and a Q table.

[0079] Furthermore, in the embodiments of this disclosure, the action corresponding to the maximum value of each state in the Q table is taken as the unique action corresponding to that state in the SQ table. Table 2 is a schematic diagram of an SQ table proposed in the embodiments of this disclosure.

[0080]

[0081] As shown in Table 2, each row in Table 2 corresponds to the Q value of the corresponding state after performing the optimal action.

[0082] In the embodiments of this disclosure, the SQ table is an improved Q table with a dimension of m×1. The amount of data stored in the SQ table is significantly reduced compared to the amount of data in the Q table of the traditional Q-learning algorithm. This results in shorter training and learning times for the UAV each time, and fewer training iterations required for the overall convergence of the algorithm. Furthermore, unlike the traditional algorithm, the improved Q-learning algorithm does not need to query the Q table information, but instead queries the SQ table information. Based on this, the number of parameters that the improved Q-learning algorithm needs to learn is greatly reduced, accelerating the convergence speed of the algorithm and thus saving resources.

[0083] Furthermore, in the embodiments of this disclosure, in the above-mentioned initialized Q table, the Q value corresponding to each state action is initialized to zero; in the above-mentioned initialized SQ table, the Q value corresponding to each state target action is initialized to zero.

[0084] Step 1034: Determine the current state S current Is it in a saturated state?

[0085] In the embodiments of this disclosure, the above-mentioned determination of the current state S current The method for determining whether a state is saturated can include: if there is an action among the optional actions corresponding to the current state that can directly reach the target point, then the current state is determined to be saturated; or, if the next state pointed to by all the optional actions corresponding to the current state is a saturated state, then the current state is determined to be saturated.

[0086] For example, in an embodiment of this disclosure, Figure 2 This is a schematic diagram of a process for confirming saturation state according to an embodiment of this disclosure. Figure 2 As shown, the starting point and ending point of the flight mission are determined to be (1,1) and (4,4) respectively. According to the above method for determining the saturation state, there is an action among the optional actions in (3,4) that can directly reach the target point (4,4). Therefore, (3,4) is marked as the saturation state in this training.

[0087] Step 1035, if S is determined current If the state is not saturated, the exploration strategy generates action 'a', and the drone enters the next state S by executing action 'a'. next Update S in the Q table current Execute the Q value of action a, and set S current Replace S next ;

[0088] Step 1036, if S is determined current If it is in a saturated state, then determine S. current The corresponding target action is determined, and the SQ table is updated. current The corresponding Q value indicates the end of this trajectory training.

[0089] In the embodiments of this disclosure, if S is determined as described above... current If it is in a saturated state, then determine S. current The corresponding target action is determined, and the SQ table is updated. current The method for determining the corresponding Q value may include the following steps:

[0090] Step 1: If S is determined current If it is in a saturated state, then determine S. current All corresponding executable actions and the Q value corresponding to each executable action;

[0091] Step 2, S current The executable action with the largest Q value among all executable actions is determined as S. current The corresponding target action is determined, and the SQ table is updated. current The corresponding Q value.

[0092] In the embodiments of this disclosure, the above-mentioned update of the SQ table S current The specific methods for determining the corresponding Q value may include: if S in the SQ table current If a Q-value exists, then compare the Q-value corresponding to the target action with the current SQ value in the SQ table. current For the corresponding Q value, if the Q value corresponding to the target action is greater than the S value in the current SQ table... current The corresponding Q value indicates that the target action is better than the corresponding action in the current SQ table, so the S value in the SQ table is changed. current The corresponding Q value is updated to the Q value corresponding to the target action; if the Q value corresponding to the target action is less than or equal to the current S in the SQ table... current The corresponding Q value indicates that the action to be performed in the current SQ table is better than the target action. Therefore, the S value in the SQ table should be adjusted accordingly. current The corresponding Q value is not updated; if S in the SQ table current If no corresponding Q value exists, then directly add S to the SQ table. current The corresponding Q value is updated to the Q value corresponding to the target action.

[0093] Step 1037: Repeat the above operation until the state corresponding to the starting point is a saturated state to end the training, and determine the target action corresponding to each state in the SQ table as the optimal trajectory planning for the UAV.

[0094] Step 104: Input the optimal flight path plan of the UAV into the three-dimensional environment model to obtain the path simulation results of the optimal flight path of the UAV.

[0095] In the embodiments of this disclosure, the method for inputting the optimal UAV trajectory planning into a 3D environment model to obtain the path simulation results of the optimal UAV trajectory may include the following steps:

[0096] Step 1041: Determine the actions to be performed by the UAV in each state along its optimal flight path;

[0097] Step 1042: Based on the mission parameters of the UAV flight, input the execution actions of the UAV in each state into the 3D environment model to obtain the path simulation results of the optimal UAV trajectory.

[0098] In summary, the UAV trajectory planning method based on improved Q-learning proposed in this application includes: 3D environment modeling of the UAV flight environment using a 3D mesh method, and dividing the UAV flight space position units into identical cubic shapes; constructing a Markov decision process model for solving the UAV path planning based on the 3D environment model; obtaining the optimal UAV trajectory plan through a saturated state search strategy and an improved Q-learning algorithm based on the Markov decision process model and the UAV flight mission parameters; and inputting the optimal UAV trajectory plan into the 3D environment model to obtain the path simulation results of the optimal UAV trajectory. Therefore, in the process of obtaining the optimal UAV trajectory plan through the saturated state search strategy and the improved Q-learning algorithm in this application, the improved Q-learning algorithm reduces the number of parameters that need to be learned, improves learning efficiency, accelerates algorithm convergence speed, and thus saves resources.

[0099] Figure 3 This is a schematic diagram of a UAV trajectory planning device based on improved Q-learning according to an embodiment of this application, as shown below. Figure 3 As shown, the device may include:

[0100] Modeling module 301 is used to perform three-dimensional environment modeling of the UAV flight environment using the three-dimensional mesh method, and to divide the UAV flight space position unit into the same cubic shape;

[0101] Module 302 is used to build a Markov decision process model for solving UAV path planning based on 3D environment modeling.

[0102] The processing module 303 is used to obtain the optimal trajectory planning of the UAV based on the Markov decision process model and the mission parameters of the UAV flight, through a saturated state search strategy and an improved Q-learning algorithm.

[0103] Output module 304 is used to input the optimal trajectory planning of the UAV into the 3D environment modeling to obtain the path simulation results of the optimal trajectory of the UAV.

[0104] In summary, the UAV trajectory planning device based on improved Q-learning proposed in this application includes: 3D environment modeling of the UAV flight environment using a 3D mesh method, dividing the UAV flight space position units into identical cubic shapes; constructing a Markov decision process model for solving the UAV path planning based on the 3D environment model; obtaining the optimal UAV trajectory plan through a saturated state search strategy and an improved Q-learning algorithm based on the Markov decision process model and the UAV flight mission parameters; and inputting the optimal UAV trajectory plan into the 3D environment model to obtain the path simulation results of the optimal UAV trajectory. Therefore, in the process of obtaining the optimal UAV trajectory plan through the saturated state search strategy and the improved Q-learning algorithm in this application, the improved Q-learning algorithm reduces the number of parameters that need to be learned, improves learning efficiency, accelerates algorithm convergence speed, and thus saves resources.

[0105] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0106] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0107] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for UAV trajectory planning based on an improved Q-learning algorithm, characterized in that, The method includes: The UAV flight environment is modeled in three dimensions using a three-dimensional mesh method, and the UAV flight space position unit is divided into the same cube shape. Based on the three-dimensional environment modeling, a Markov decision process model for solving the UAV path planning is constructed. Based on the Markov decision process model and the mission parameters of the UAV flight, the optimal UAV trajectory planning is obtained through a saturated state search strategy and an improved Q-learning algorithm. The optimal flight path plan of the UAV is input into the three-dimensional environment modeling to obtain the path simulation results of the optimal flight path of the UAV. The improved Q-learning algorithm includes an SQ table and a Q table, wherein the Q table is a two-dimensional table that stores feedback values. Based on the Markov decision process model and the mission parameters of the UAV flight, the optimal UAV trajectory planning is obtained through a saturated state search strategy and an improved Q-learning algorithm, including: Obtain the mission parameters for the UAV flight, wherein the mission parameters include the target point and the starting point; Determine the endpoint state and initial state corresponding to the target point and the starting point, respectively; Initialize the SQ table and the Q table, wherein in the initial Q table, the Q value corresponding to each state action is initialized to zero, and in the initial SQ table, the Q value corresponding to each state target action is initialized to zero; Determine the current state Is it in a saturated state? If the above is determined If it is not in a saturated state, then the exploration strategy generates actions. The drone performs actions Enter the next state Update the Q table Execute action The Q value, and the Replace the ; If the above is determined If it is in a saturated state, then determine the... The corresponding target action is then updated in the SQ table. The corresponding Q value indicates the end of this trajectory training. Repeat the above operation until the state corresponding to the starting point is a saturated state to end the training, and determine the target action corresponding to each state in the SQ table as the optimal trajectory planning for the UAV. Determining the current state Whether it is in a saturated state includes: If any of the optional actions corresponding to the current state can directly reach the target point, then the current state is determined to be a saturated state; or If all the optional actions corresponding to the current state point to the next state, which is a saturated state, then the current state is determined to be a saturated state. If the determination is made If it is in a saturated state, then determine the... The corresponding target action is then updated in the SQ table. The corresponding Q values ​​include: If the above is determined If it is in a saturated state, then determine the... All corresponding executable actions and the Q value corresponding to each executable action; The The executable action with the largest Q value among all executable actions is determined as the [specific action]. The corresponding target action is determined, and the SQ table is updated. The corresponding Q value.

2. The method according to claim 1, characterized in that, The construction of the Markov decision process model for solving the UAV path planning includes: constructing the UAV flight action space representation and reward function.

3. The method according to claim 2, characterized in that, The reward function includes: ； in, This is the current state. Indicates the execution of an action The next state after that, For the drone to perform actions in state s Arrive after The reward Represents the obstacle reward function, This represents the threat-reward function. This indicates the target reward.

4. The method according to claim 1, characterized in that, The step of inputting the optimal flight path plan of the UAV into the three-dimensional environment modeling to obtain the path simulation results of the optimal flight path of the UAV includes: In determining the optimal flight path of the UAV, the actions that the UAV performs in each state are as follows: Based on the mission parameters of the UAV flight, the actions to be performed by the UAV in each state are input into the three-dimensional environment model to obtain the path simulation results of the optimal flight path of the UAV.

5. A UAV trajectory planning device based on improved Q-learning, characterized in that, The device includes: The modeling module is used to perform three-dimensional environmental modeling of the UAV flight environment using the three-dimensional mesh method, and to divide the UAV flight space position unit into the same cubic shape; A construction module is used to build a Markov decision process model for solving the path planning of the UAV based on the modeling of the three-dimensional environment. The processing module is used to obtain the optimal trajectory planning of the UAV based on the Markov decision process model and the mission parameters of the UAV flight, through a saturated state search strategy and an improved Q-learning algorithm. The improved Q-learning algorithm includes an SQ table and a Q table, wherein the Q table is a two-dimensional table that stores feedback values. The output module is used to input the optimal flight path plan of the UAV into the three-dimensional environment modeling to obtain the path simulation results of the optimal flight path of the UAV. The processing module is also used to acquire mission parameters for the UAV flight, wherein the mission parameters include the target point and the starting point; Determine the endpoint state and initial state corresponding to the target point and the starting point, respectively; Initialize the SQ table and the Q table, wherein in the initial Q table, the Q value corresponding to each state action is initialized to zero, and in the initial SQ table, the Q value corresponding to each state target action is initialized to zero; Determine the current state Is it in a saturated state? If the above is determined If it is not in a saturated state, then the exploration strategy generates actions. The drone performs actions Enter the next state Update the Q table Execute action The Q value, and the Replace the ; If the above is determined If it is in a saturated state, then determine the... The corresponding target action is then updated in the SQ table. The corresponding Q value indicates the end of this trajectory training. Repeat the above operation until the state corresponding to the starting point is a saturated state to end the training, and determine the target action corresponding to each state in the SQ table as the optimal trajectory planning for the UAV. Determining the current state Whether it is in a saturated state includes: If any of the optional actions corresponding to the current state can directly reach the target point, then the current state is determined to be a saturated state; or If all the optional actions corresponding to the current state point to the next state, which is a saturated state, then the current state is determined to be a saturated state. If the determination is made If it is in a saturated state, then determine the... The corresponding target action is then updated in the SQ table. The corresponding Q values ​​include: If the above is determined If it is in a saturated state, then determine the... All corresponding executable actions and the Q value corresponding to each executable action; The The executable action with the largest Q value among all executable actions is determined as the [specific action]. The corresponding target action is determined, and the SQ table is updated. The corresponding Q value.

6. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it is able to implement the method as described in any one of claims 1-4.

7. A computer storage medium, wherein, The computer storage medium stores computer-executable instructions; when executed by a processor, the computer-executable instructions can implement the method described in any one of claims 1-4.

Images