Method, system and equipment for planning full-coverage flight path of unmanned aerial vehicle in time-varying wind field environment based on reinforcement learning

Abstract

The invention discloses a time-varying wind field environment unmanned aerial vehicle full-coverage flight path planning method and system based on reinforcement learning, and mainly solves the problem that an existing unmanned aerial vehicle coverage flight path planning method cannot plan a full-coverage flight path in a dynamic wind field environment. According to the implementation scheme, the method comprises the steps that two-dimensional environment modeling is conducted on an unmanned aerial vehicle task area through a grid method, and the task area is discretely segmented into grids of the same size; according to the two-dimensional environment model, constructing a Markov decision model comprising an action space, a state transfer function and a reward function through a reinforcement learning algorithm; according to the time-varying characteristics of the wind field, designing an initialized time-sharing Q table capable of planning the full-coverage flight path of the time-varying wind field environment, and training the initialized time-sharing Q table; according to the trained time-sharing Q table and the Markov decision model, the short-endurance full-coverage flight path of the unmanned aerial vehicle in the time-varying wind field environment is obtained.For the dynamic time-varying wind field environment, an effective coverage path completely covering a task area can be planned, complete reconnaissance of the unmanned aerial vehicle on the task area is guaranteed, and the method can be used for unmanned aerial vehicle reconnaissance tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of UAV trajectory planning, and further relates to a method and system for planning full-coverage trajectory of UAV in a time-varying wind environment, which can be used for regional reconnaissance. Background Art

[0002] In recent years, with the development of unmanned aerial vehicle (UAV) aerospace technology, UAVs (Unmanned Aerial Vehicles) have been widely used in both military and civilian fields. UAVs, with their numerous advantages such as small size, light weight, low cost, and high maneuverability, are widely used in regional reconnaissance missions. When performing reconnaissance missions, UAVs typically first need to plan an offline track and then complete a coverage scan of the mission area by tracking this offline track. Therefore, coverage track planning is a key step in the execution of reconnaissance missions. In UAV coverage track planning, time-varying wind fields can easily cause the UAV to deviate from its track or fly backwards. Accounting for the impact of time-varying wind fields on UAVs is also a key issue in UAV coverage track planning. Therefore, it is crucial to use reinforcement learning to design an intelligent coverage track planning algorithm that can track the time-varying nature of wind fields and output a full coverage track.

[0003] In their paper "Coverage path planning for maritime search and rescue using reinforcement learning," Bo Ai, Maoxin Jia, Hanwen Xu, et al. employed reinforcement learning to plan autonomous coverage paths for maritime vessels by designing a multi-objective optimization reward function. They planned a search path that minimized time and prioritized coverage of high-probability areas, while ensuring complete coverage of the maritime SAR area and avoiding maritime obstacles. First, they established a marine SAR environment model using field data and electronic nautical charts. Second, they proposed a reinforcement learning-based model for autonomous maritime SAR coverage path planning. This model employed a multi-constraint reward function to guide the navigation behavior of the vessel agent. During the iterative training of the path planning model, a nonlinear action selection strategy dynamically adjusted the random action selection probabilities to ensure stable model convergence. Finally, they conducted experimental validation in various simulated small-scale maritime SAR scenarios. Because the constructed reward function prioritized coverage of high-probability areas, this method could not guarantee that the planned path would fully cover the mission area. In actual reconnaissance missions, this method's planned path could result in the UAV failing to fully survey the mission area.

[0004] In their journal article, "Complete Coverage Path Planning of an Unmanned Surface Vehicle Based on a Complete Coverage Neural Network Algorithm," PF Xu, YX Ding, and JC Luo proposed a full coverage neural network algorithm. This algorithm, combined with coverage direction, establishes a next-optimal position decision formula, making the path more regular and coverage more efficient. Furthermore, by adding movement directions to the path within the grid map, it further reduces turning angles and makes the path smoother. However, this method can only be used to plan coverage paths in static environments. When the environment changes over time, it cannot plan an effective coverage path that fully covers the mission area.

[0005] Patent document CN202310730071.6 discloses a UAV trajectory planning method based on improved Q-learning. It first uses a three-dimensional grid method to model the UAV flight environment in three dimensions, then constructs a Markov decision process model for solving the UAV path planning, and finally obtains the optimal trajectory of the UAV through a saturated search strategy and an improved Q-learning algorithm based on the Markov decision process model. Although this method can reduce the number of parameters required to learn, improve learning efficiency, and accelerate the convergence of the algorithm, it can only be used to plan point-to-point paths in static environments and cannot plan full-coverage paths in dynamic wind field environments.

[0006] The above-mentioned defects of the existing technology will result in the drone being unable to fully survey the mission area. Summary of the Invention

[0007] The purpose of the present invention is to address the deficiencies of the above-mentioned existing technologies and propose a method and system for full-coverage trajectory planning of UAVs in a time-varying wind field environment based on reinforcement learning, so as to plan an effective coverage path that fully covers the mission area in a dynamic wind field environment and ensure that the UAV can fully reconnaissance the mission area.

[0008] To achieve the above objectives, the technical solutions of the present invention include the following:

[0009] 1. A method for full-coverage UAV trajectory planning in a time-varying wind environment based on reinforcement learning, characterized by comprising:

[0010] 1) Use the grid method to perform two-dimensional environment modeling of the UAV mission area and discretely divide the mission area into grids of the same size;

[0011] 2) Based on the two-dimensional environment model, a Markov decision model including action space, state transition function and reward function is constructed through reinforcement learning algorithm;

[0012] 3) Based on the time-varying characteristics of the wind farm, an initial time-sharing Q table is designed to plan the full coverage trajectory in the time-varying wind farm environment and then trained;

[0013] 4) Based on the trained time-sharing Q-table and Markov decision model, the UAV's short-flight full-coverage trajectory is planned in a time-varying wind environment.

[0014] Furthermore, in (3), according to the time-varying characteristics of the wind field, a time-sharing Q table that can plan the full coverage trajectory of the time-varying wind field environment is designed and trained, including:

[0015] (3a) With δ as the time interval, the two-dimensional Q tables of several reinforcement learning Q learning algorithms are arranged in time order and all Q values ​​are set to 0 to form an initialized time-sharing Q table;

[0016] (3b) Based on the Markov decision model constructed in (2), a single-step update formula Q′(s, a, t) of the time-varying Q table Q value is defined by the Q learning algorithm;

[0017] (3c) Using the Q-value single-step update formula to update the initialized time-sharing Q table, that is, using the Q-value single-step update formula to update the Q-value corresponding to the UAV taking action a at position s in the time interval from 0 to t in the time-sharing Q table through the Q-learning algorithm, the current updated time-sharing Q table is obtained and used to plan the UAV trajectory in the time interval from 0 to t;

[0018] (3d) Let t = t + 1;

[0019] (3e) Repeat (3c)-(3d) until the UAV trajectories planned in all time intervals can fully cover the mission area, and obtain a trained time-sharing Q table that can be used to plan UAV short-flight full-coverage trajectories in time-varying wind field environments.

[0020] 2. A reinforcement learning-based UAV full-coverage trajectory planning system for time-varying wind environments, characterized by including:

[0021] The environmental modeling module is used to discretely divide the UAV's mission area into several grids of equal size and store the time-varying wind field information of each grid area;

[0022] A model building module, which is used to construct the UAV flight action space, the UAV state transition function and the reward function to form a Markov decision model;

[0023] The training module is used to construct and train an initial time-sharing Q table based on the Markov decision model to obtain a time-sharing Q table that can be used to plan the short-flight full-coverage trajectory of the UAV in a time-varying wind field environment;

[0024] The trajectory planning module is used to plan the UAV's short-flight full-coverage trajectory in a time-varying wind environment based on the trained time-sharing Q table.

[0025] 3. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for full-coverage trajectory planning of a UAV in a time-varying wind environment based on reinforcement learning as claimed in any one of claims 1 to 5 is implemented.

[0026] 4. A non-transitory computer-readable storage medium, characterized in that the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the reinforcement learning-based UAV full-coverage trajectory planning method in a time-varying wind field environment as claimed in any one of claims 1 to 5.

[0027] Compared with the existing technology, the present invention has the following advantages:

[0028] First, the present invention fully considers the time-varying characteristics of the wind field and designs a time-sharing Q table to improve the Q learning algorithm. Compared with the existing Q learning algorithm that can only plan the trajectory in a static environment, the present invention can plan the trajectory of the mission area in a dynamic wind field environment.

[0029] Second, the present invention plans a full-coverage trajectory based on the reinforcement learning method. Compared with the existing methods that can only prioritize the coverage of high-probability areas in the mission area under dynamic conditions or fully cover the mission area under static conditions, the present invention can plan a full-coverage trajectory of the mission area under dynamic wind field conditions, ensuring that the UAV can fully reconnaissance the mission area. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of the method for implementing the full coverage trajectory planning of UAVs in a time-varying wind environment based on reinforcement learning of the present invention;

[0031] Figure 2 for Figure 1 Schematic diagram of using grid method to model two-dimensional environment;

[0032] Figure 3 for Figure 1 Schematic diagram of the optimal yaw angle of a UAV using a Markov decision model constructed using a two-dimensional environment model;

[0033] Figure 4 This is a block diagram of the UAV full coverage trajectory planning system for time-varying wind environment based on reinforcement learning of the present invention;

[0034] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present invention;

[0035] Figure 6A comparison chart of the full coverage of the mission area planned by the present invention and the Q learning algorithm under the same time-varying wind field environment;

[0036] Figure 7 The present invention randomly generates a full coverage track for the mission area of ​​time-varying wind site planning within the wind speed range of 5-15m / s. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions and effects of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0038] Example 1: UAV full coverage trajectory planning method in time-varying wind environment based on reinforcement learning

[0039] Reference Figure 1 The implementation steps of this example include the following:

[0040] Step 1: Use the grid method to perform two-dimensional environment modeling of the UAV mission area and discretely divide the mission area into grids of the same size.

[0041] 1.1) Assume the drone reconnaissance radius is r p The drone can scan 9 grids centered on itself each time and calculate the side length L of the unit grid. grid :

[0042]

[0043] 1.2) According to the side length L of the unit grid grid , the entire reconnaissance area is discretely divided into several grids of the same size, such as Figure 2 As shown, the position of each grid is represented by P i,j It indicates that the grid is in the i-th row and j-th column of the entire grid environment. Calculate the total number of rows n and columns m of the grid environment:

[0044]

[0045] Among them, width is the width of the task area, and length is the length of the task area;

[0046] 1.3) To represent the location of each grid and the wind field data within the grid area, let P i,j The wind field data of the grid area at the tth time interval δ is W t, there are N hours of wind field data, and the two-dimensional environmental model ENV is expressed as:

[0047] ENV={(P i,j ,W0,W1,…,W t )|i∈{0,1,2,…,n-1},j∈{0,1,2,…,m-1},t∈{0,1,2,…,N}};

[0048] 1.4) To represent the coverage of each grid by UAV reconnaissance, let P i,j The reconnaissance coverage of the grid area is C i,j , and according to P i,j The value of a grid is determined by whether it is covered by UAV reconnaissance. The two-dimensional coverage model is expressed as:

[0049] COV={C i,j |i∈{0,1,2,…,n-1},j∈{0,1,2,…,m-1},C i,j ∈{0,1}}.

[0050] The model shows that if P i,j The grid area has been covered by drone reconnaissance, then C i,j The value is 1; otherwise, the value is 0.

[0051] Step 2: Construct a state transfer function based on the two-dimensional environment model.

[0052] 2.1) Establishing the action space:

[0053] The action space is the set of actions that the drone can take. At each grid, the drone can choose one of the four directions: up, down, left, or right, and move one grid. Therefore, the action space A is expressed as:

[0054] A={up,down,left,right};

[0055] Among them, up means the drone moves upward, down means the drone moves downward, left means the drone moves to the left, and right means the drone moves to the right;

[0056] 2.2) Constructing the state transfer function:

[0057] The state transfer function is used to calculate the UAV's state in P i,j The new position P′ of the drone after taking an action a in the action space i,j :

[0058]

[0059] Among them, Pi,j Indicates that the drone is currently located at the i-th row and j-th column of the grid environment, P i-1,j Indicates that the drone is currently located at the i-1th row and jth column of the grid environment, P i+1,j Indicates that the drone is currently located at the i+1th row and jth column of the grid environment, P i,j-1 Indicates that the UAV is currently located at the i-th row and j-1-th column of the grid environment, P i,j+1 Indicates that the drone is currently located at the i-th row and j+1-th column of the grid environment, n represents the total number of rows in the grid environment, and m represents the total number of columns in the grid environment;

[0060] Step 3: Establish a reward function.

[0061] The reward function evaluates the actions taken by the drone and consists of four parts: coverage reward, inverted flight and off-track penalties, flight time reward, and out-of-bounds penalty. The specific implementation is as follows:

[0062] 3.1) Calculate coverage rewards:

[0063] According to the fact that each action of the drone can cover a new grid area, the coverage rate of the mission area continues to increase, which is conducive to achieving full coverage of the mission area. When the drone selects an action to complete the state transfer, the coverage reward R it receives is set to c is 10 times the number of newly covered grid areas N new_cover :

[0064] R c =10N new_cover ;

[0065] 3.2) Set penalties for inverted flight and off-track flight:

[0066] Since drones are prone to flying backwards or off-track when flying in time-varying wind fields, it is necessary to design penalties for these phenomena to guide drones to avoid them during the planning process.

[0067] like Figure 3 As shown, v g is the ground speed vector of the UAV, v a is the UAV airspeed vector and its magnitude |v a | constant, α is v g The angle of the vector, γ is v g Vector and v a The angle between the vectors, β is v g Vector and v w The angle between the vectors; v w is a known wind field vector, the magnitude and angle of which are |v w | and ω wind .

[0068] Drones in P i,j Position flies to the new position P′ i,j During the flight, the wind field will change due to crossing the grid, that is, v w The magnitude and angle of the vector change due to v w Changes in the vector will cause changes in the γ and β angles. Therefore, the flight process needs to be divided into two sections according to the grid area to determine whether inverted flight or deviation from the track occurs. The following operations must be performed for any flight process:

[0069] 3.2.1), calculate the angles of angle α and angle γ:

[0070] Assume the current position of the drone is P i,j The coordinates of (x P ,y P ), new position P′ i,j The coordinates of (x P′ ,y P′ ), calculate the angle α of the drone's ground speed vector:

[0071]

[0072] Using wind farm v w 、UAV ground speed v g 、UAV airspeed v a The triangle formed by three vectors and the law of sines is used to calculate v g Vector and v a The angle γ between the vectors:

[0073] Where β = ω wind -α;

[0074] Then, based on whether γ has a solution, we can determine whether the drone will deviate from its track:

[0075] If γ has a solution, the drone can be transferred to P′ along a straight line i,j ;

[0076] Otherwise, the drone will deviate from its track;

[0077] 3.2.2) Calculate the drone ground speed vector v g With the drone airspeed vector v a The angle of angle is less than v g ,v a >, determine whether the drone has reversed flight during this flight g Vector and v a The angle v of the vector g Vector and v a The angle v of the vectorg Vector and v a The angle between the vectors:

[0078] Calculate the drone ground speed vector v g With the drone airspeed vector v a The angle of angle is less than v g ,v a >:

[0079]

[0080] Among them, the UAV airspeed vector v a For (|v a |cos(α-γ),|v a |sin(α-γ)), UAV ground speed vector v g For (|v a |cos(α-γ)+|v w |cosω wind ,|v a |sin(α-γ)+|v w |sinω wind );

[0081] According to <v g ,v a > If the angle is less than 90 degrees, determine whether the drone will fly upside down:

[0082] If < v g ,v a ><90°, the drone will not fly backwards;

[0083] Otherwise, the drone will fly upside down;

[0084] 3.2.3) If any segment of the flight inverts or deviates from the flight path, a fixed penalty of -100 will be applied.

[0085] 3.3) Set flight time rewards:

[0086] The core goal of a drone's coverage trajectory is to safely complete the coverage of the mission area in the shortest possible flight time. Therefore, a flight time reward needs to be set based on the flight time to guide the drone to complete full coverage of the mission area in the shortest possible time. This is achieved as follows:

[0087] 3.3.1) Calculate the drone’s i,j Position flies to the new position P′ i,j Flight time:

[0088] Let P i,j The wind speed in the grid area is |v w |, wind direction is ω wind , then the UAV is in Pi,j v within the grid area g Vector and v a The angle between the vectors is γ, and the ground speed vector v in the grid area is g for:

[0089] (|v a |cos(α-γ)+|v w |cosω wind ,|v a |sin(α-γ)+|v w |sinω wind );

[0090] Let P′ i,j The wind speed in the grid area is |v′ w |, wind direction is ω′ wind , then the UAV is at P′ i,j v′ in the grid area g Vector and v′ a The angle between the vectors is γ′, and the ground speed vector v′ in the grid area is g for:

[0091] (|v a |cos(α-γ′)+|v′ w |cosω′ wind ,|v a |sin(α-γ′)+|v′ w |sinω′ wind );

[0092] According to P i,j and P′ i,j Ground velocity vector v within the grid area g and v′ g , calculate the UAV from P i,j Position flies to the new position P′ i,j Flight time t flytime :

[0093]

[0094] 3.3.2) Normalize the flight time:

[0095] Computing drones in P i,j The flight time of the position in the four flight directions, let the maximum value of the four flight times be t max , calculate t flytime The normalized value T f :

[0096]

[0097] 3.3.3) Setting flight time rewards:

[0098] When the drone is i,j Position flies to the new position P′ i,j When , the flight time bonus is set based on whether there are newly covered grids:

[0099] If there is a new covered grid N new_cover >0 then flight time bonus value: R f =1-T f ,

[0100] Otherwise, R f =-T f ;

[0101] The formula is as follows:

[0102]

[0103] 3.4) Set boundary penalty:

[0104] The drone flies in a grid environment, and the grid environment has a boundary. If the drone attempts to move outside the boundary, it is considered an invalid action P′ i,j =P i,j , and set a fixed penalty value of -50 to guide the drone to avoid moving outside the boundary during flight;

[0105] 3.5) Based on the above four results of coverage reward, inverted flight and off-track penalty, flight time reward and out-of-bounds penalty, we get the following total reward function R:

[0106]

[0107] Step 4: Construct a Markov decision model through reinforcement learning algorithm.

[0108] In the reinforcement learning algorithm, the Markov decision model is used to output the new position and reward value of the drone based on the input current position and action of the drone. The new position and reward value are used to update the initialized time-sharing Q table;

[0109] The Markov decision model is composed of the state transfer function established in step 2 and the reward function established in step 3, that is, the current position of the drone P i,j and action a as input into the state transfer function constructed in step 2 to obtain the new position P′ of the drone i,j ; The new position of the drone P′ i,j Substitute this into the reward function established in step 3 to obtain the reward value R, and finally output the new position P′ of the drone through the model. i,j With reward value R.

[0110] Step 5: Based on the time-varying characteristics of the wind farm, a time-sharing Q table is designed that can plan the full coverage trajectory of the time-varying wind farm environment and is trained.

[0111] 5.1) Design and initialize the time-sharing Q table:

[0112] Assuming that the time interval of wind farm environment information stored in the two-dimensional environment model is δ, the historical experience of the reinforcement learning Q learning algorithm is accumulated at intervals of δ. Based on this, an initial time-sharing Q table is constructed with δ as the time interval. The two-dimensional Q tables of several Q learning algorithms are arranged in chronological order to form an initial time-sharing Q table. All Q values ​​in the initial time-sharing Q table are 0.

[0113] 5.2) Define the single-step update formula of the time-sharing Q table Q value:

[0114] According to the Q-value single-step update formula of the existing Q-learning algorithm, the Q-value single-step update formula of the time-sharing Q table is defined as follows:

[0115] Q′(s,a,t)=Q(s,a,t)+α RL [R+γ RL Q(s′,a′,t)-Q(s,a,t)]

[0116] Among them, Q(s,a,t) represents the Q value corresponding to the drone taking action a at position s in the t-th time interval [tδ,(t+1)δ) in the current time-sharing Q table, Q′(s,a,t) represents the new Q value corresponding to the drone taking action a at position s in the t-th time interval [tδ,(t+1)δ) after the update, α RL represents the learning rate, R represents the reward value of the reward function, γ RL represents the discount factor, Q(s′,a′,t) represents the maximum Q value corresponding to the four actions of the drone at the new position s′ in the t-th time interval [tδ,(t+1)δ) in the current time-sharing Q table;

[0117] 5.3) Train the initialized time-sharing Q table:

[0118] The initialized time-sharing Q table needs to be fully updated to obtain a trained time-sharing Q table. Since the initialized time-sharing Q table has a time sequence, the currently updated time-sharing Q table will affect the next update of the time-sharing Q table. If the Q values ​​at different time intervals are updated separately during the update, the global optimal coverage path cannot be obtained. Therefore, the following process is designed for time-sharing Q table training:

[0119] 5.3.1) Initialize t to 1;

[0120] 5.3.2) Calculate the new Q values ​​Q′(s, a, t) corresponding to the UAV taking different actions a at different positions s during the time interval from 0 to t using the single-step Q-value update formula of the time-sharing Q-table. Replace the original Q values ​​Q(s, a, t) with the new Q values ​​Q′(s, a, t) to obtain the current updated time-sharing Q-table, which is used to plan the UAV's trajectory during the time interval from 0 to t.

[0121] 5.3.3) Determine whether to end training based on the currently updated time-sharing Q table:

[0122] If the UAV tracks in all time intervals planned by the currently updated time-sharing Q table have fully covered the mission area, the training is terminated;

[0123] Otherwise, proceed to step 5.3.4)

[0124] 5.3.4) Set t = t + 1 and repeat steps 5.3.2) to 5.3.3) until the planned UAV tracks in all time intervals can fully cover the mission area, and then the training ends;

[0125] After the training, a time-sharing Q table is obtained, which can be used to plan the short-flight full-coverage trajectory of the UAV in a time-varying wind environment.

[0126] Step 6: Based on the trained time-sharing Q table and Markov decision model, plan the UAV's short-flight full-coverage trajectory in the time-varying wind environment.

[0127] 6.1) Let the starting position of the environment be the current position of the drone s, and the total flight time be t start , t start Initialized to 0;

[0128] 6.2) According to the total flight time t start Assume that the current moment is at the t-th time interval, select the action a that maximizes Q(s, a, t) at the t-th time interval under position s, and use it as the flight direction of the drone;

[0129] 6.3) Input the current position s and action a of the drone into the Markov decision model to obtain the new position s′ of the drone, which is used as the current position of the drone.

[0130] 6.4) Calculate the flight time t of the UAV during this flight flytime ;

[0131] 6.5) Let t start =t start +t flytime Repeat steps 6.2) to 6.4) until the UAV reconnaissance covers the entire mission area and the trajectory planning is completed;

[0132] 6.6) Connect all the obtained new UAV positions s′ in sequence to form a UAV short-flight full-coverage track in a time-varying wind environment.

[0133] The flowchart representations or method representations of the above embodiments can be understood as representing a module, segment, or portion of code that includes one or a group of executable instructions configured to implement a specific logical function or process. The present invention is not limited to the disclosed preferred embodiments, and its implementation may not follow the order presented or discussed. That is, the step numbers in the method of the present invention are only for the purpose of clarifying the embodiment of the present invention and facilitating understanding, and the order of the steps is not limited.

[0134] Example 2: UAV full coverage trajectory planning system in time-varying wind environment based on reinforcement learning.

[0135] Reference Figure 4 This example includes: environment modeling module 1, model building module 2, training module 3, and trajectory planning module 4. Among them, model building module 2 includes: action space submodule 21, state transition submodule 22, and reward function submodule 23. The working principle of the entire system is as follows:

[0136] The environmental modeling module 1 is used to discretely divide the UAV's mission area into a number of grids of equal size, establish a two-dimensional environmental model and coverage model based on the time-varying wind field information and coverage information of each grid area, and transmit the established models to the model construction module 2;

[0137] The model construction module 2 is used to construct the UAV flight action space, the UAV state transfer function and the reward function to form a Markov decision model, wherein the action space submodule 21 is used to construct the UAV flight action space and determine the feasible flight direction of the UAV; the state transfer submodule 22 is used to calculate the new position of the UAV based on the current position of the UAV and the flight direction taken, and transmit the new position to the reward function submodule 23; the reward function submodule 23 is used to calculate the reward value obtained by the UAV based on the change of the UAV position result according to the new position sent by the state transfer submodule 22, and transmit the reward value to the training module 3;

[0138] The training module 3 constructs and trains an initialized time-sharing Q table based on the reward value sent by the model building module 2, obtains a time-sharing Q table that can be used to plan a short-flight, full-coverage trajectory of a UAV in a time-varying wind field environment, and transmits the trained time-sharing Q table to the trajectory planning module 4;

[0139] The trajectory planning module 4 plans a short-flight full-coverage trajectory of the UAV in a time-varying wind field environment according to the trained time-sharing Q table sent by the training module 3.

[0140] It should be noted that the above-mentioned functional modules can be implemented in whole or in part through software, hardware, or any combination thereof. When implemented using software, they can be implemented in whole or in part in the form of a program instruction product. The program instruction product includes one or a group of program instructions. When the program instructions are loaded and executed on a computer, the described process or function is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in a computer-readable and writable storage medium, or transferred from a computer-readable and writable storage medium to another computer-readable and writable storage medium.

[0141] The direct coupling or communication connection between the modules shown or discussed in this embodiment can be achieved through indirect coupling or communication connection of some interfaces, devices or modules. The various functional modules and submodules in this embodiment can be dynamically located in a processing component, or each module can exist physically separately, or two or more modules can be dynamically located in a processing component. When the above-mentioned dynamic components are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable and writable storage medium. The storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0142] Reference Figure 5 The electronic device provided by the embodiment of the present invention includes: a processor, a memory and a bus. The processor and the memory are connected to each other via the bus in the device.

[0143] The processor can be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to load a computer program to execute the technical solution provided by the method and system for full-coverage trajectory planning of unmanned aerial vehicles in a time-varying wind field environment based on reinforcement learning in this embodiment.

[0144] The memory can be implemented in the form of a read-only memory (ROM), a random access memory (RAM), a static storage device, a dynamic storage device, etc. These memories can store operating systems and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory and called and executed by the processor.

[0145] The bus comprises a pathway that transfers information between various components of a device, such as a processor and memory.

[0146] It should be noted that although the above device only shows a processor, a memory, and a bus, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figures.

[0147] The non-transitory computer-readable medium provided in this embodiment stores a plurality of computer instructions, which can be loaded by a processor to execute the steps in the embodiment of the method for full-coverage trajectory planning of a UAV in a time-varying wind environment based on reinforcement learning. The readable medium includes permanent and non-permanent, removable and non-removable media, and the information storage function can be implemented by any other methods and technologies. The information can be a computer-readable instruction, a data structure, a program module or other data. Examples of computer storage media include, but are not limited to, PRAM phase change memory, SRAM static random access memory, DRAM dynamic random access memory, other types of RAM random access memory, ROM read-only memory, EEPROM electrically erasable programmable read-only memory, flash memory or other memory technology, CD-ROM read-only disc read-only memory, DVD digital versatile disc or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0148] The effects of the present invention can be further illustrated by the following simulation results:

[0149] 1. Simulation Experiment Conditions

[0150] The simulation experiment software platform of the present invention is: the experimental computer operating system is Window11 (64-bit), the memory is 16GB, the processor configuration is: Intel(R) Core(TM) i7-8750H CPU@2.20GHz2.20GHz, implemented in Python language, the version is Python3.11, and the experimental operating environment is JetBrains PyCharm Community Edition2022.2.

[0151] The wind field dataset used in the simulation experiment comes from NASA data.

[0152] 2. Simulation content and results analysis:

[0153] Simulation 1: Under the above conditions, the present invention and the existing Q learning algorithm are used to plan full coverage tracks for the same mission area under the same time-varying wind field environment. The track shapes obtained by the two methods are compared. The results are as follows: Figure 6 .

[0154] Depend on Figure 6 It can be seen that the Q learning algorithm cannot plan a full-coverage trajectory of the mission area in a dynamic environment, while the method of the present invention can plan a full-coverage trajectory of the mission area in a dynamic wind field environment, which can ensure that the UAV can fully reconnaissance the mission area.

[0155] Simulation 2, under the above conditions, using the UAV coverage trajectory planning method of the present invention, randomly generating a time-varying wind field environment in the wind speed range of 5-15m / s to plan a full coverage trajectory for the mission area, the results are as follows Figure 7 .

[0156] like Figure 7 It can be seen that in a randomly generated time-varying wind field environment, the method of the present invention can still stably plan a full-coverage trajectory of the mission area, verifying the full coverage performance of the trajectory planned by the present invention.

[0157] The above simulation results show that under the above conditions, the present invention can plan an effective coverage path that completely covers the mission area in a dynamic and time-varying wind field environment, ensuring that the UAV can fully reconnaissance the mission area.

Claims

1. A method for full-coverage UAV trajectory planning in a time-varying wind environment based on reinforcement learning, characterized by: include: (1) The two-dimensional environment modeling of the UAV mission area is carried out using the grid method, and the mission area is discretely divided into grids of the same size; (2) Based on the two-dimensional environment model, a Markov decision model including action space, state transition function and reward function is constructed through reinforcement learning algorithm; (3) According to the time-varying characteristics of the wind farm, a time-varying Q table is designed to plan the full coverage trajectory of the time-varying wind farm environment and to train it; (4) Based on the trained time-sharing Q table and Markov decision model, the UAV's short-flight full-coverage trajectory is planned in a time-varying wind environment.

2. The method according to claim 1, characterized in that In (1), a two-dimensional environment model is performed on the UAV mission area using a grid method, including a two-dimensional environment model and a two-dimensional coverage model, which is implemented as follows: (1a) Calculate the unit grid side length L grid Among them, r p The reconnaissance radius of the drone; (1b) According to the unit grid side length L grid , calculate the number of rows n and columns m after the task area is discretized into a grid environment Among them, width is the width of the task area, and length is the length of the task area; (1c) Based on the results of (1b), a two-dimensional environment model ENV and a two-dimensional coverage model COV are constructed respectively: ENV={(P i,j ,W0,W1,…,W t )|i∈{0,1,2,…,n-1},j∈{0,1,2,…,m-1},t∈{0,1,2,…,N}}, COV={C i,j |i∈{0,1,2,…,n-1},j∈{0,1,2,…,m-1},C i,j ∈{0,1}}, Among them, P i,j Indicates that the grid is in the i-th row and j-th column of the entire grid environment, W t It represents the wind field data in the t-th time interval δ of the grid area. There are N time intervals of wind field data in total. n represents the total number of rows in the grid environment, and m represents the total number of columns in the grid environment. C i,j Indicates P i,j Whether the grid has been detected by drones: If it has been detected by drones, then C i,j The value is 1; otherwise, the value is 0.

3. The method according to any one of claims 1 to 2, characterized in that In (2), based on the two-dimensional environment model, a Markov decision model including action space, state transition function and reward function is constructed by reinforcement learning algorithm, which is implemented as follows: (2a) Based on the grid characteristics of the two-dimensional environment model, the UAV chooses to move in any of the four directions: up, down, left, or right at each grid point, and establishes the Markov decision model action space A: A={up,down,left,right}, Among them, up means the drone moves upward, down means the drone moves downward, left means the drone moves to the left, and right means the drone moves to the right; (2b) According to the result of (2a), the UAV is i,j After taking different actions a in the action space when the position is reached, the new position P′ is obtained i,j : Among them, P i,j Indicates that the drone is currently located at the i-th row and j-th column of the grid environment, P i-1,j Indicates that the drone is currently located at the i-1th row and jth column of the grid environment, P i+1,j Indicates that the drone is currently located at the i+1th row and jth column of the grid environment, P i,j-1 Indicates that the UAV is currently located at the i-th row and j-1-th column of the grid environment, P i,j+1 Indicates that the drone is currently located at the i-th row and j+1-th column of the grid environment, n represents the total number of rows in the grid environment, and m represents the total number of columns in the grid environment; (2c) The drone receives a reward based on the new position change obtained in (2b), and establishes a Markov decision model reward function R: Among them, N new_cover Indicates the number of newly detected grids after the drone moves, T f represents the normalized flight time during the movement of the UAV position, v g Represents the ground speed vector of the UAV during its movement.

4. The method according to claim 1, wherein In (3), a time-sharing Q table that can plan a full-coverage trajectory in a time-varying wind field environment is designed based on the time-varying characteristics of the wind field, and is trained, including: (3a) With δ as the time interval, the two-dimensional Q tables of several reinforcement learning Q learning algorithms are arranged in time order and all Q values ​​are set to 0 to form an initialized time-sharing Q table; (3b) Based on the Markov decision model constructed in (2), the single-step update formula of the time-sharing Q table Q value is defined by the Q learning algorithm: Q′(s,a,t)=Q(s,a,t)+α RL [R+γ RL Q(s′,a′,t)-Q(s,a,t)] Among them, Q(s,a,t) represents the Q value corresponding to the drone taking action a at position s in the t-th time interval [tδ,(t+1)δ) in the current time-sharing Q table, Q′(s,a,t) represents the new Q value corresponding to the drone taking action a at position s in the t-th time interval [tδ,(t+1)δ) after the update, α RL represents the learning rate, R represents the reward value of the reward function, γ RL represents the discount factor, Q(s′,a′,t) represents the maximum Q value corresponding to the four actions of the drone at the new position s′ in the t-th time interval [tδ,(t+1)δ) in the current time-sharing Q table; (3c) Using the Q-value single-step update formula to update the initialized time-sharing Q table, that is, using the Q-value single-step update formula to update the Q-value corresponding to the UAV taking action a at position s in the time interval from 0 to t in the time-sharing Q table through the Q-learning algorithm, the current updated time-sharing Q table is obtained and used to plan the UAV trajectory in the time interval from 0 to t; (3d) Let t = t + 1; (3e) Repeat (3c)-(3d) until the UAV trajectories planned in all time intervals can fully cover the mission area, and obtain a trained time-sharing Q table that can be used to plan UAV short-flight full-coverage trajectories in time-varying wind field environments.

5. The method according to claim 1, wherein In (4), the short-flight full-coverage trajectory of the UAV in a time-varying wind environment is planned based on the trained time-sharing Q table and Markov decision model, including: (4a) The starting position of the environment is taken as the current position s of the UAV; (4b) Calculate the total flight time from the starting position of the UAV to the current position. Based on the total flight time, assume that the current moment is in the tth time interval. Select the action a that maximizes Q(s, a, t) in the tth time interval at position s and use it as the flight direction of the UAV. (4c) Calculate the new position s′ of the UAV based on the Markov decision model and the flight direction of the UAV, and use it as the current position of the UAV; (4d) Repeat (4b)-(4c) until the UAV reconnaissance covers the entire mission area, and then connect all the calculated new UAV positions s′ in sequence to form a UAV short-flight full-coverage track in a time-varying wind field environment.

6. A UAV full coverage trajectory planning system in time-varying wind environment based on reinforcement learning, characterized by ,include: The environmental modeling module is used to discretely divide the UAV's mission area into several grids of equal size and store the time-varying wind field information and coverage information of each grid area; A model building module, which is used to construct the UAV flight action space, the UAV state transition function and the reward function to form a Markov decision model; The training module is used to construct and train an initial time-sharing Q table based on the Markov decision model to obtain a time-sharing Q table that can be used to plan the short-flight full-coverage trajectory of the UAV in a time-varying wind field environment; The trajectory planning module is used to plan the UAV's short-flight full-coverage trajectory in a time-varying wind environment based on the trained time-sharing Q table.

7. The method according to claim 1, characterized in that The model building module includes: The action space submodule is used to construct the UAV's flight action space and determine the UAV's feasible flight direction; The state transfer submodule is used to calculate the new position of the drone based on its current position and the flight direction it takes, and transmit the new position to the training module The reward function submodule is used to calculate the reward value obtained by the drone based on the change of the drone's position results and transmit the reward value to the training module.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for full-coverage trajectory planning of a UAV in a time-varying wind environment based on reinforcement learning is implemented as claimed in any one of claims 1 to 5.

9. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method for full-coverage trajectory planning of a UAV in a time-varying wind environment based on reinforcement learning as claimed in any one of claims 1 to 5.

Images