A Low-Energy Trajectory Planning Method for Airships Based on Inverse Reinforcement Learning

Abstract

The present invention discloses a low-energy trajectory planning method for airships based on inverse reinforcement learning, and belongs to the technical field of machine learning data processing. The low-energy trajectory planning method for airships based on inverse reinforcement learning includes the following steps: modeling of airships and task spaces, establishing the plane kinematic state equation of airships, and performing grid discretization processing on the task space; collecting and organizing environmental and expert driving data; establishing an inverse reinforcement learning model; using an IRL algorithm to learn a reward function; learning a trajectory planning strategy based on RL; and verifying the optimal strategy. The low-energy trajectory planning method for airships based on inverse reinforcement learning described in the present invention is adopted, and by combining inverse reinforcement learning with reinforcement learning, the optimal trajectory planning strategy is automatically derived, so that the airship can autonomously adapt to environmental changes in a complex dynamic environment with uncertain factors, optimize the rationality of the flight trajectory, and improve endurance.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning data processing, and in particular to a low-energy-consumption airship trajectory planning method based on inverse reinforcement learning. Background Art

[0002] An airship is a low-speed aircraft that can fly at low and high altitudes and can rely on gas buoyancy to operate for a long time. Due to its long endurance, strong load-bearing capacity and low cost, it is widely used in aerial tours, earth observation, communication relay, network services and surveillance tasks. As a large, low-powered and high-inertia drone, the operating environment of an airship is more complex than that of a traditional aircraft, resulting in lower controllability and sensitivity to environmental influences in its flight trajectory. The flight trajectory of an airship is easily affected by the wind field. In addition, airships usually carry limited energy. Therefore, under the condition of ensuring safe navigation from the starting point to the target area, developing a suitable trajectory to make the airship consume less energy is of great significance to the long-flight mission planning of the airship.

[0003] Most of the current research on airship trajectory planning focuses on static environments, such as stable wind fields. When the external environment changes or is uncertain, traditional trajectory planning methods are difficult to adapt to these dynamic changes. However, the changes in the environment during actual flight have a great impact on the flight trajectory and flight energy consumption of the airship. It is necessary to find a new method to solve the low-energy trajectory planning problem of stratospheric airships in dynamic environments.

[0004] Reinforcement Learning (RL) is an unsupervised machine learning method that conducts online learning by studying the interaction between the object and the environment. When the environment changes, the RL algorithm can automatically adapt to the new environmental changes through continuous interaction and updating. In recent years, reinforcement learning has become a research hotspot in the field of machine learning and has many applications in aerospace planning, control, and navigation.

[0005] For airships in complex operating environments, low energy consumption requires a reward function related to fuel or electricity consumption, but it is difficult to design an accurate reward function that takes into account energy consumption, flight stability, and time requirements, resulting in complex multi-objective trade-offs. In view of the difficulty of low-energy operation of airships, it is urgent to study the corresponding low-energy trajectory planning method to improve the endurance and autonomous navigation capabilities of airships.

[0006] Inverse Reinforcement Learning (IRL) can analyze the flight data of expert pilots and learn directly from expert behavior. This means that in the absence of clear reward signals, IRL can effectively derive low-energy trajectory planning strategies similar to expert strategies. Compared with traditional optimization methods and reinforcement learning methods, IRL can better utilize expert knowledge, adapt to complex environments, improve sample efficiency, reduce design complexity, and optimize long-term decisions to handle complex decision-making problems. Summary of the invention

[0007] The purpose of the present invention is to provide a low-energy trajectory planning method for airships based on inverse reinforcement learning. By combining inverse reinforcement learning with reinforcement learning, the optimal trajectory planning strategy is automatically derived, so that the airship can autonomously adapt to environmental changes in a complex dynamic environment with uncertain factors, optimize the rationality of the flight trajectory and improve the endurance.

[0008] To achieve the above object, the present invention provides a low-energy airship trajectory planning method based on inverse reinforcement learning, comprising the following steps:

[0009] S1. Modeling of airship and mission space, establishing the plane kinematic state equation of airship, and performing grid discretization processing on mission space;

[0010] S2, collect and organize environmental and expert driving data;

[0011] S3, establish an inverse reinforcement learning model;

[0012] S4. Use IRL algorithm to learn reward function;

[0013] S5, RL-based trajectory planning strategy learning;

[0014] S6. Optimal strategy verification.

[0015] Preferably, in S1, the airship plane kinematic state equation is:

[0016] ;

[0017] in, The airship is in the ground coordinate system Axis coordinates, The airship is in the ground coordinate system Axis coordinates, is the heading angle, is the zonal wind speed, is the meridional wind speed, It's the speed of the airship.

[0018] Preferably, the dynamic equation of the airship is:

[0019] ;

[0020] ;

[0021] in, is the overall mass of the airship, is the overall vector thrust generated by the airship, is the resistance encountered by the airship during flight; is the air density, is the drag coefficient, is the volume of the stratospheric airship, is the propulsion power of the airship, is the propulsion efficiency of the airship, is the wind speed.

[0022] Preferably, in S2, the airship driving trajectory data of the expert pilots under different time periods, different environmental conditions and different tasks are collected to form an expert data set, and the airship driving trajectory data includes several airship positions and , the state corresponding to each airship position includes the airship speed , heading angle , remaining power And the wind field data at the corresponding location , the status data is sorted by location and Match the trajectory points on the discrete map to form a trajectory planning discrete map.

[0023] Preferably, in S3, the airship in the inverse reinforcement learning model t Momentary sports space for:

[0024] ;

[0025] in, is the speed of the airship relative to the ground, is the heading angular velocity of the airship;

[0026] Airship in t The state space at time for:

[0027] ;

[0028] in, is the distance from the current position of the airship to the target position;

[0029] state The eigenvector of for:

[0030] .

[0031] Preferably, in S4, the specific steps of using the IRL algorithm to learn the reward function are:

[0032] S41. Define the reward function.

[0033] ;

[0034] in, is the trajectory in the expert sample, is the state in the trajectory, Yes Status The feature vector, trajectory The eigenvector of is the sum of the eigenvectors of each trajectory point, that is, , is the characteristic vector of each state A vector of corresponding reward weights;

[0035] S42. Solve the inverse reinforcement learning model and obtain the optimal strategy ;

[0036] S43, based on the optimal strategy Estimate each state Frequency of visits ;

[0037] S44, solving the state access frequency;

[0038] Calculating the loss function Relative to reward weight gradient.

[0039] ;

[0040] in, represents the expected feature vector of all demonstration trajectories in the expert sample, is the number of demonstration trajectories, It is The feature vector of the demonstration trajectory; the expression Indicates the probability of transition in a given state T and reward weight θ In the case of The conditional probability of

[0041] S45, reward function parameter update;

[0042] Use the calculated ,right Perform a gradient update and minimize the loss function through the gradient descent method. , is the learning rate.

[0043] S46, loop iterate between S42~S45 to obtain the final reward function .

[0044] Preferably, in S43, each state is estimated Frequency of visits The specific process is:

[0045] S431, initialization state frequency counter

[0046] Initialize frequency counter , for all , is the state space;

[0047] S432, trajectory sampling

[0048] 1) For n Simulation, , N The total number of trajectory simulations

[0049] (1) Initialize trajectory

[0050] Reset the environment to its initial state , and initialize the track visit count record , for all ;

[0051] (2) Trajectory simulation

[0052] In Status Select Action ;

[0053] implement And observe the next state transferred to ;

[0054] Will The number of visits is increased by 1: ;

[0055] Update current status ← ;

[0056] 2) Add the number of visits in this trajectory to the total visit count

[0057] For each state , ;

[0058] S433, calculate the expected access frequency of each state

[0059] After trajectory sampling is completed, the expected access frequency of each state is calculated. : .

[0060] Preferably, in S5, the learned reward function is used , combined with the action space , state space , attenuation coefficient , complete positive reinforcement learning in the training set and obtain the final optimal strategy .

[0061] Preferably, in S6, the optimal strategy is used Complete trajectory planning in the test set, compare the generated trajectory with the actual expert trajectory, and evaluate the degree of expert trajectory imitation of the optimal strategy by calculating the ratio of the length of the common route in the two trajectories to the total route length.

[0062] The advantages and positive effects of the low-energy airship trajectory planning method based on inverse reinforcement learning described in the present invention are:

[0063] 1. The low-energy reward function design of the present invention is based on IRL, which directly learns the low-energy trajectory strategy from the expert flight data without the need for an explicit reward signal, significantly improves the sample efficiency, reduces the trial-and-error process and data requirements, and improves the efficiency of strategy learning.

[0064] 2. The RL-based trajectory planning strategy of the present invention enables the airship to adjust its route in real time and adapt to changes in environmental wind speed. It has stronger robustness than the traditional static method, ensures stable flight under uncertain meteorological conditions and reduces energy loss, thus improving the airship's environmental adaptability.

[0065] 3. The present invention combines inverse reinforcement learning with reinforcement learning to automatically derive the optimal trajectory planning strategy, reduce complex environmental modeling and manual intervention, adapt to different tasks and environmental changes more quickly, and reduce the complexity of planning and design.

[0066] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a flow chart of the low-energy-consumption trajectory planning method of an airship based on inverse reinforcement learning of the present invention;

[0068] Figure 2 The horizontal kinematic model of the airship of the present invention;

[0069] Figure 3Schematic diagram of the discretization of task space of the present invention. DETAILED DESCRIPTION

[0070] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0071] like Figure 1 A low-energy airship trajectory planning method based on inverse reinforcement learning includes the following steps:

[0072] S1. Modeling of airship and mission space, establishing the plane kinematic state equation of airship, and performing grid discretization processing on the mission space.

[0073] When the airship reaches the designated altitude and begins to perform tasks such as relay communication, monitoring, and sightseeing, the motion of the airship can be considered as planar motion. Therefore, the roll, pitch, and other attitude changes of the airship are ignored, and only the planar motion of the airship is considered. Considering the influence of wind speed on the airship, the planar kinematic model of the airship is as follows: Figure 2 shown.

[0074] The plane kinematic state equation of the airship is:

[0075] ;

[0076] in, The airship is in the ground coordinate system Axis coordinates, The airship is in the ground coordinate system Axis coordinates, is the heading angle. is the zonal wind speed, is the meridional wind speed, and is the wind speed The components in the longitude and latitude directions vary with position and time. is the airship speed, the actual speed of the airship is the speed of the airship and wind speed The vector sum of .

[0077] The plane dynamics model of the airship ignores lift, and the dynamic equation is:

[0078] ;

[0079] ;

[0080] in, is the overall mass of the airship, is the overall vector thrust generated by the airship, is the resistance encountered by the airship during flight; is the air density, is the drag coefficient, is the volume of the stratospheric airship, is the propulsion power of the airship, is the propulsion efficiency of the airship, is the wind speed. Assuming the airship is powered by batteries, define the remaining power of the airship This is the basic state of the airship.

[0081] In order to facilitate the trajectory planning of the airship, the mission space plane map is divided into grid-like discrete maps, such as Figure 3 The grid is detailed enough to ensure that each airship position in the expert pilot data and The distance to the nearest grid node is less than the length of the airship.

[0082] S2. Collect and organize environmental and expert driving data.

[0083] The data of airship driving trajectories of expert pilots in different time periods, different environmental conditions, and different tasks are collected to form an expert dataset. The dataset contains a large number of expert driving trajectories, each of which contains multiple airship positions. and , the state corresponding to each airship position includes the airship speed , heading angle , remaining power And the wind field data at the corresponding location . The above data is sorted by location and Match the trajectory points on the discrete map to form a discrete map for trajectory planning. 70% of the trajectory data in the above dataset is used as the training set, and the remaining 30% of the trajectory data is used as the test set.

[0084] S3. Establish an inverse reinforcement learning model.

[0085] In order to facilitate the application of the IRL algorithm in the dataset, the trajectory planning model is defined in the form of a deterministic Markov decision process (MDP) model.

[0086] Definition of airship t Momentary sports space for:

[0087] ;

[0088] in, is the speed of the airship relative to the ground, is the heading angular velocity of the airship;

[0089] In order to describe the state of the airship, the state space of the airship is designed. tThe state space at time for:

[0090] ;

[0091] in, is the distance from the current position of the airship to the target position. It can be calculated by any existing distance calculation formula, including but not limited to Euclidean distance, Manhattan distance, Chebyshev distance, etc. In addition, the decay coefficient in the custom Markov chain .

[0092] In order to facilitate the quantification of each state in the IRL algorithm , predict and explain the driver's behavior, and convert each state , and Make feature counts for this state and form a feature vector .

[0093] state The eigenvector of for:

[0094] .

[0095] S4. Use the IRL algorithm to learn the reward function.

[0096] The maximum entropy-based IRL method is used to complete the learning of the reward function. The specific steps are:

[0097] S41. Define the reward function.

[0098] ;

[0099] in, is the trajectory in the expert sample, is the state in the trajectory, Yes Status The feature vector, trajectory The eigenvector of is the sum of the eigenvectors of each trajectory point, that is, , is the characteristic vector of each state A vector of corresponding reward weights.

[0100] S42. Solve the inverse reinforcement learning model and obtain the optimal strategy .

[0101] Known state space , Action Space , reward function and attenuation coefficient , find the optimal strategy The solution process can use existing dynamic programming methods, Monte Carlo methods, temporal difference methods, etc.

[0102] S43, based on the optimal strategy Estimate each state Frequency of visits .

[0103] Use the Monte Carlo method to estimate each state Frequency of visits , the specific process is:

[0104] S431, initialization state frequency counter

[0105] For each state in the environment , initialize its access count to 0.

[0106] Initialize frequency counter , for all , is the state space.

[0107] S432, trajectory sampling

[0108] Perform multiple trajectory simulations and record the number of visits to each state in each simulation.

[0109] 1) For n Simulation, , N The total number of trajectory simulations

[0110] (1) Initialize trajectory

[0111] Reset the environment to its initial state , and initialize the track visit count record , for all .

[0112] (2) Trajectory simulation

[0113] Before the terminal state, actions are continuously selected and states are transferred to form a complete trajectory. For each step in the trajectory:

[0114] In Status Select Action (usually chosen at random);

[0115] implement And observe the next state transferred to ;

[0116] Will The number of visits is increased by 1: ;

[0117] Update current status ← .

[0118] 2) Accumulate the number of visits to this track

[0119] Add the number of visits in this trace to the total visit count.

[0120] For each state , .

[0121] S433, calculate the expected access frequency of each state

[0122] After trajectory sampling is completed, the expected access frequency of each state is calculated. : .

[0123] S44, solving the state access frequency.

[0124] Calculating the loss function Relative to reward weight gradient.

[0125] ;

[0126] in, represents the expected feature vector of all demonstration trajectories in the expert sample, is the number of demonstration trajectories, It is The feature vector of the demonstration trajectory; the expression Indicates the probability of transition in a given state T and reward weight θ In the case of The conditional probability of .

[0127] S45, reward function parameter update;

[0128] Use the calculated ,right Perform a gradient update and minimize the loss function through the gradient descent method. . is the learning rate, which controls the step size of parameter update.

[0129] S46, iterate between S42 and S45 until the gradient converges or the maximum number of iterations is reached, then stop iterating. Get the final reward function .

[0130] S5. RL-based trajectory planning strategy learning.

[0131] Using the learned reward function , combined with the action space , state space , attenuation coefficient , complete positive reinforcement learning in the training set and obtain the final optimal strategy . Forward reinforcement learning includes but is not limited to existing methods such as Deep Q-Network, Deep Deterministic Policy Gradient, and Soft Actor-Critic.

[0132] S6. Optimal strategy verification.

[0133] Use the best strategy Complete trajectory planning in the test set, compare the generated trajectory with the actual expert trajectory, and evaluate the degree of expert trajectory imitation of the optimal strategy by calculating the ratio of the length of the common route in the two trajectories to the total route length.

[0134] Therefore, the low-energy trajectory planning method for airships based on inverse reinforcement learning described in the present invention is adopted. By combining inverse reinforcement learning with reinforcement learning, the optimal trajectory planning strategy is automatically derived, so that the airship can autonomously adapt to environmental changes in a complex dynamic environment with uncertain factors, optimize the rationality of the flight trajectory and improve the endurance.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A low-energy airship trajectory planning method based on inverse reinforcement learning, characterized in that: The following steps are involved: S1. Modeling of airship and mission space, establishing the plane kinematic state equation of airship, and grid discretization of mission space; S2. Collecting and organizing environmental and expert driving data; S3. Establishing inverse reinforcement learning model; S4. Learning reward function using IRL algorithm; S5. Learning trajectory planning strategy based on RL; S6. Verifying the optimal strategy; In S1, the airship plane kinematic state equation is: ； in, The airship is in the ground coordinate system Axis coordinates, The airship is in the ground coordinate system Axis coordinates, is the heading angle, is the zonal wind speed, is the meridional wind speed, is the airship speed; In S4, the specific steps of using the IRL algorithm to learn the reward function are: S41. Define the reward function. ； in, is the trajectory in the expert sample, is the state in the trajectory, Yes Status The feature vector, trajectory The eigenvector of is the sum of the eigenvectors of each trajectory point, that is, , is the characteristic vector of each state A vector of corresponding reward weights; S42. Solve the inverse reinforcement learning model and obtain the optimal strategy ; S43, based on the optimal strategy Estimate each state Frequency of visits ; S44, solving the state access frequency; Calculating the loss function Relative to reward weight θ Gradient ； in, represents the expected feature vector of all demonstration trajectories in the expert sample, is the number of demonstration trajectories, It is The feature vector of the demonstration trajectory; the expression Indicates the probability of transition in a given state T and reward weight θ In the case of The conditional probability of S45, reward function parameter update; Use the calculated ,right Perform a gradient update and minimize the loss function through the gradient descent method. , is the learning rate; S46, loop iterate between S42~S45 to obtain the final reward function .

2. The method for low-energy airship trajectory planning based on inverse reinforcement learning according to claim 1, characterized in that: The dynamic equation of the airship is: ； ； in, is the overall mass of the airship, is the overall vector thrust generated by the airship, is the resistance encountered by the airship during flight; is the air density, is the drag coefficient, is the volume of the stratospheric airship, is the propulsion power of the airship, The propulsion efficiency of the airship, is the wind speed.

3. The method for low-energy airship trajectory planning based on inverse reinforcement learning according to claim 2, characterized in that: In S2, the airship driving trajectory data of the expert pilots under different time periods, different environmental conditions, and different tasks are collected to form an expert data set. The airship driving trajectory data includes several airship positions. and , the state corresponding to each airship position includes the airship speed , heading angle , remaining power And the wind field data at the corresponding location , the status data is sorted by location and Match the trajectory points on the discrete map to form a trajectory planning discrete map.

4. The method for low-energy airship trajectory planning based on inverse reinforcement learning according to claim 3 is characterized in that: In S3, the airship in the inverse reinforcement learning model t Momentary sports space for: ； in, is the speed of the airship relative to the ground, is the heading angular velocity of the airship; Airship in t The state space at time for: ； in, is the distance from the current position of the airship to the target position; state The eigenvector of for: 。 5. The method for low-energy airship trajectory planning based on inverse reinforcement learning according to claim 4, characterized in that: In S43, each state is estimated Frequency of visits The specific process is: S431, initialization state frequency counter Initialize frequency counter , for all , is the state space; S432, trajectory sampling 1) For Simulation, , N The total number of trajectory simulations (1) Initialize trajectory Reset the environment to its initial state , and initialize the track visit count record , for all ; (2) Trajectory simulation In Status Select Action ; implement And observe the next state transferred to ; Will The number of visits is increased by 1: ; Update current status ← ; 2) Add the number of visits in this trajectory to the total visit count For each state , ; S433, calculate the expected access frequency of each state After trajectory sampling is completed, the expected access frequency of each state is calculated. : .

6. The method for low-energy airship trajectory planning based on inverse reinforcement learning according to claim 5, characterized in that: In S5, the learned reward function is used , combined with the action space , state space , attenuation coefficient , complete positive reinforcement learning in the training set and obtain the final optimal strategy .

7. The method for low-energy airship trajectory planning based on inverse reinforcement learning according to claim 6, characterized in that: In S6, the optimal strategy is used Complete trajectory planning in the test set, compare the generated trajectory with the actual expert trajectory, and evaluate the degree of expert trajectory imitation of the optimal strategy by calculating the ratio of the length of the common route in the two trajectories to the total route length.

Images