A Dynamic Mission Planning Method for Aircraft Based on Reinforcement Learning and Constraint Optimization

By modeling aircraft mission planning as a Markov decision process and combining reinforcement learning and constraint optimization, the problems of real-time performance and constraint fusion in aircraft mission planning in dynamic environments are solved, achieving efficient and safe mission execution.

CN123113304APending Publication Date: 2026-11-17SHENYANG AIRCRAFT DESIGN INST AVIATION IND CORP OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202511425982.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-09-30
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing aircraft mission planning methods fail to respond to dynamic changes in real time and fail to effectively integrate aircraft performance limitations, resulting in low mission execution efficiency and insufficient safety.

Method used

The method employs reinforcement learning and constraint optimization to model mission planning as a Markov decision process. Mission planning is performed through a deep Q-learning network, and constraint optimization modules for speed, energy, and safe distance are set to adjust flight paths and mission scheduling in real time.

Benefits of technology

It enables real-time optimization of mission planning in dynamic environments, improves mission completion efficiency, ensures that the aircraft can complete missions safely and efficiently in complex environments, and reduces energy waste and accident risks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123113304A_ABST
    Figure CN123113304A_ABST
Patent Text Reader

Abstract

This application belongs to the field of aircraft mission planning technology, and specifically relates to a dynamic mission planning method for aircraft based on reinforcement learning and constraint optimization. The method generates flight missions through the aircraft's mission execution model, uses these missions as input, and performs reinforcement learning through a deep Q-learning network algorithm. Based on experience replay and a target network mechanism, it trains an optimal mission planning strategy to obtain the optimal flight path and mission scheduling strategy for the aircraft. A constraint optimization module is set up; when the mission scheduling strategy exceeds the constraint optimization strategy, the current mission scheduling strategy is executed according to the corresponding constraint value. By combining reinforcement learning and constraint optimization, the mission planning strategy can be flexibly adjusted in a dynamic environment, significantly improving the aircraft's mission execution efficiency, reducing energy consumption, and ensuring the safe completion of the mission.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of aircraft mission planning technology, and specifically relates to an aircraft dynamic mission planning method based on reinforcement learning and constraint optimization. Background Technology

[0002] With the rapid development of aircraft technology, particularly its broad application prospects in fields such as aerial logistics, urban air transportation, and unmanned delivery, aircraft mission planning has increasingly become a key technology. Currently, traditional mission planning methods mostly rely on pre-set flight paths and fixed scheduling strategies, failing to respond in real-time to dynamic changes occurring during flight, such as weather changes, wind speed fluctuations, sudden mission demands, and interactions with other aircraft. Furthermore, existing aircraft scheduling methods fail to effectively integrate aircraft performance constraints (such as flight speed, minimum safe distance, and energy consumption), which are crucial for the feasibility and safety of flight missions. Therefore, how to optimize mission planning in real-time in dynamic environments and efficiently complete missions without violating aircraft physical performance constraints has become a major challenge in the widespread application of aircraft.

[0003] Reinforcement learning, as an adaptive decision-making method based on interactive learning, has been widely used in complex task scheduling and intelligent control in recent years. Its superior dynamic decision-making capabilities enable it to adjust task planning strategies based on real-time feedback, making it particularly suitable for scenarios requiring rapid response to changes. However, pure reinforcement learning methods may not meet the physical constraints of aircraft performance in practical applications. Therefore, it is necessary to introduce constraint optimization techniques into task planning to ensure that the task planning process meets multiple constraints such as flight speed, energy consumption, and safe distance during reinforcement learning.

[0004] Constraint optimization methods can explicitly consider the physical constraints of the aircraft during mission planning, such as flight speed, minimum safe distance, and remaining energy limits. Through optimization algorithms, the aircraft can maintain a safe and efficient flight state when performing missions.

[0005] However, existing constraint optimization techniques still face the challenge of integrating with task scheduling in dynamic environments. In particular, how to adjust flight paths in real time to ensure mission continuity and execution efficiency in the face of emergencies or environmental changes remains a technical challenge. Summary of the Invention

[0006] The purpose of this application is to provide a dynamic mission planning method for aircraft based on reinforcement learning and constraint optimization, so as to solve the problem that existing constraint optimization techniques are difficult to achieve real-time adjustment in dynamic environments.

[0007] The technical solution of this application is: a dynamic mission planning method for aircraft based on reinforcement learning and constraint optimization, comprising:

[0008] The mission planning problem is modeled as a Markov decision process, defining the state space, action space, reward function, and state transition probability, and a mission execution model for the aircraft is established.

[0009] Flight missions are generated by the aircraft's mission execution model. The flight missions are used as inputs and reinforcement learning is performed through a deep Q-learning network algorithm. The optimal mission planning strategy is trained based on experience replay and target network mechanism to obtain the optimal flight path and mission scheduling strategy for the aircraft.

[0010] Set up a constraint optimization module, which sets speed constraints, energy consumption constraints, and minimum safe distance constraints for the aircraft. When the task scheduling strategy exceeds the constraint optimization strategy, execute the current task scheduling strategy according to the corresponding constraint values.

[0011] Preferably, the state space S defines all possible states s of the aircraft. t ; state s t Represented as:

[0012] s t =[x t y t , z t v t θ t E t ];

[0013] Where, x t y t , z t v represents the position coordinates of the aircraft in three-dimensional space. t θ represents the current velocity of the aircraft. t E represents the heading angle. t Remaining energy;

[0014] Action space A defines the control actions that the aircraft can take; each action a t Represents the flight decision of the aircraft; State A t It can be represented as:

[0015] A t =[Δv t , Δθ t Δz t ];

[0016] Where, Δv t Δθ represents the change in the velocity of the aircraft. t The change in heading angle, Δz tThis refers to the change in flight altitude.

[0017] Reward function R(s) t a t The reward function R(s) is used to measure the feedback after an aircraft performs a certain action. t a t The design is as follows:

[0018] R(s t a t )=α1.T c (s t a t )+α2.F e (s t a t )-α3.E c (s t a t )-α4.C v (s t a t );

[0019] Among them, T c (s t a t ) is a reward given based on the degree of task completion; F e (s t a t E is a reward given based on the shortest flight path and flight efficiency. c (s t a t C is the penalty imposed based on the aircraft's energy consumption; v (s t a t It is used to punish behaviors that violate the constraints of the aircraft.

[0020] Preferably, when performing reinforcement learning using a deep Q-learning network algorithm, the Q-value is approximated.

[0021] The Q-value represents the expected total reward for choosing a certain action in a given state. The Q-value is updated using the Bellman equation:

[0022]

[0023] Where α is the learning rate, γ is the discount factor, and r t For instant rewards, The maximum Q value in the future state;

[0024] The input to the deep Q-learning network algorithm is the current state s. tThe output is the Q-value corresponding to each action; during this process, the aircraft continuously updates the network weights through interaction with the environment.

[0025] After each interaction with the environment, the aircraft stores a quadruple of state, action, reward, and next state in an experience pool. During each update, a batch is randomly selected from the experience pool for training until the optimal task planning strategy is obtained based on the Q-value.

[0026] Preferably, the constraint optimization module is designed as follows:

[0027] Define the penalty function P(s) for constraint optimization. t a t )for:

[0028] P(s t a t )=λ1.S c (s t a t )+λ2.D c (s t a t )+λ3.e c (s t a t );

[0029] Among them, the speed constraint S c (s t a t (v) represents the speed of the aircraft. t Cannot exceed the maximum flight speed v max or below the minimum flight speed v min ;

[0030] Among them, the speed constraint S c (s t a t )for:

[0031] S c (s t a t ) = max(0, v t -v max )+max(0, v min -v t );

[0032] Safe distance D c (s t a t The constraint is that the aircraft needs to maintain a minimum safe distance d from obstacles or other aircraft. s ,for:

[0033]

[0034] Energy consumption constraint e c (s t a t E represents the energy consumption of the aircraft. t Cannot exceed the maximum energy E max ,for:

[0035] e c (s t a t ) = max(0, E t -E max ).

[0036] Preferably, a residual energy threshold for the aircraft is set. If the residual energy of the aircraft is less than the residual energy threshold after the mission scheduling strategy has been executed, the mission scheduling strategy is deemed to have failed.

[0037] Preferably, a ring-shaped sensing module is provided, which can collect other environmental parameters, including wind speed, weather conditions, and the positions of other aircraft; when the positions of other aircraft are detected, the flight heading is adjusted according to the angle between the current heading and the positions of other aircraft; different altitude coefficients are set for wind speed and weather conditions respectively, and when wind speed and weather conditions change, the current flight speed is adjusted according to the corresponding altitude coefficient.

[0038] This application presents a dynamic mission planning method for aircraft based on reinforcement learning and constraint optimization. Through adaptive optimization using reinforcement learning algorithms, the aircraft can automatically adjust its path planning and mission scheduling according to mission requirements and environmental changes, thereby reducing the total mission execution time and significantly improving mission completion efficiency. Especially in complex environments, the aircraft can adjust its decisions based on real-time feedback, avoiding inefficient paths and unnecessary delays. Attached Figure Description

[0039] To more clearly illustrate the technical solutions provided in this application, the accompanying drawings will be briefly described below. Obviously, the drawings described below are merely some embodiments of this application.

[0040] Figure 1 This is a schematic diagram of the overall process of this application. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] A dynamic mission planning method for aircraft based on reinforcement learning and constraint optimization, such as... Figure 1 As shown, it includes the following steps:

[0043] Step S100: Model the mission planning problem as a Markov Decision Process (MDP), define the state space, action space, reward function and state transition probability, and establish the mission execution model of the aircraft.

[0044] Preferably, the state space S defines all possible states s of the aircraft. t Each state s t This includes information such as the aircraft's position, speed, heading, altitude, and remaining energy. State s t Represented as:

[0045] Where, x t y t , z t v represents the position coordinates of the aircraft in three-dimensional space. t θ represents the current velocity of the aircraft. t E represents the heading angle. t This represents the remaining energy.

[0046] Action space A defines the control actions that the aircraft can take; each action a t Represents the aircraft's flight decisions, such as adjusting flight speed, heading angle, or altitude. Each action a t Represents the flight decision of the aircraft; State A t It can be represented as:

[0047] A t =[Δv t , Δθ t Δz t ];

[0048] Where, Δv t Δθ represents the change in the velocity of the aircraft. t The change in heading angle, Δz t This represents the change in flight altitude.

[0049] Reward function R(s) t a t The reward function (R(s)) is used to measure the feedback after an aircraft performs a certain action. Its design considers multiple factors, including mission completion, flight efficiency, and energy consumption. t a t The design is as follows:

[0050] R(st a t )=α1.T c (s t a t )+α2.F e (s t a t )-α3.E c (s t a t )-α4.C v (s t a t );

[0051] Among them, T c (s t a t ) is a reward given based on the degree of task completion; F e (s t a t E is a reward given based on the shortest flight path and flight efficiency. c (s t a t C is the penalty imposed based on the aircraft's energy consumption; v (s t a t It is used to punish behaviors that violate the constraints of the aircraft.

[0052] In step S200, a flight mission is generated through the mission execution model of the aircraft. The flight mission is used as input, and reinforcement learning is performed through the Deep Q-Learning Network (DQN) algorithm. The optimal mission planning strategy is trained based on experience replay and target network mechanism to obtain the optimal flight path and mission scheduling strategy of the aircraft.

[0053] Preferably, when performing reinforcement learning using a deep Q-learning network algorithm, the Q-value is approximated.

[0054] The Q-value represents the expected total reward for choosing a certain action in a given state. The Q-value is updated using the Bellman equation:

[0055]

[0056] Where α is the learning rate, γ is the discount factor, and r t For instant rewards, The maximum Q value in the future state;

[0057] The input to the deep Q-learning network algorithm is the current state s. t The output is the Q-value corresponding to each action; during this process, the aircraft continuously updates the network weights through interaction with the environment.

[0058] To improve training stability, DQN employs an experience replay mechanism. After each interaction with the environment, the aircraft stores a quadruple of state, action, reward, and next state in an experience pool. During each update, batches are randomly selected from the experience pool for training until the optimal task planning strategy based on the Q-value is obtained.

[0059] Preferably, a residual energy threshold for the aircraft is set. If the residual energy of the aircraft is less than the residual energy threshold after the mission scheduling strategy has been executed, the mission scheduling strategy is deemed unqualified and a new mission planning strategy needs to be obtained.

[0060] Step S300: Set the constraint optimization module, and set the speed constraint, energy consumption constraint and minimum safe distance constraint of the aircraft in the constraint optimization module; when the task scheduling strategy exceeds the constraint optimization strategy, execute the current task scheduling strategy according to the corresponding constraint value.

[0061] Preferably, the constraint optimization module ensures that the aircraft does not violate constraints such as flight speed, minimum safe distance, and energy consumption when performing a mission. A penalty mechanism is introduced in mission planning to ensure that these constraints are met. The constraint optimization module is designed as follows:

[0062] Define the penalty function P(s) for constraint optimization. t a t )for:

[0063] P(s t ,a t )=λ1·S c (s t ,a t )+λ2·D c (s t ,a t )+λ3·e c (s t ,a t );

[0064] Among them, the speed constraint s c (s t ,a t (v) represents the speed of the aircraft. t Cannot exceed the maximum flight speed v max or below the minimum flight speed v min ;

[0065] Among them, the speed constraint S c (s t ,a t )for:

[0066] S c (s t,a t ) = max(0, v t -v max )+max(0,v min -v t );

[0067] Safe distance D c (s t ,a t The constraint is that the aircraft needs to maintain a minimum safe distance d from obstacles or other aircraft. s ,for:

[0068]

[0069] Energy consumption constraint e c (s t ,a t E represents the energy consumption of the aircraft. t Cannot exceed the maximum energy E max ,for:

[0070] e c (s t a t ) = max(0, E t -E max ).

[0071] In the process of combining reinforcement learning and constraints, the aircraft must not only select the optimal mission planning strategy, but also ensure that all physical constraints of the aircraft are followed during mission execution. By introducing constraint optimization, the aircraft will avoid selecting actions that violate constraints during the learning process, thereby achieving the optimal mission planning strategy.

[0072] Preferably, a ring-shaped sensing module is provided, which can collect other environmental parameters, including wind speed, weather conditions, and the positions of other aircraft; when the positions of other aircraft are detected, the flight heading is adjusted according to the angle between the current heading and the positions of other aircraft; different altitude coefficients are set for wind speed and weather conditions respectively, and when wind speed and weather conditions change, the current flight speed is adjusted according to the corresponding altitude coefficient.

[0073] For example, an aircraft can adjust its altitude when encountering changes in wind speed to avoid the effects of adverse weather conditions. When encountering other aircraft, it can change its course in time to avoid collisions. In this way, the aircraft can flexibly respond to environmental changes and ensure the successful completion of its mission.

[0074] During mission execution, the aircraft can perceive its surrounding environment in real time. If a new obstacle is detected, the flight path will be recalculated and the mission execution strategy will be adjusted.

[0075] In a specific example, suppose an aircraft needs to transport goods from destination A to destination B. The system first acquires the aircraft's initial state information, such as its current position, speed, heading, and remaining energy. Based on this information, an initial mission plan is generated using a reinforcement learning algorithm. During mission execution, the aircraft dynamically adjusts its mission plan by sensing real-time changes in the environment (such as wind speed and weather conditions). The constraint optimization module monitors the aircraft's flight speed, energy consumption, and altitude in real time to ensure that the aircraft does not violate constraints during flight and ultimately completes the mission successfully.

[0076] In a specific example, when multiple aircraft collaborate on a mission, the aircraft autonomously optimize their mission planning using reinforcement learning algorithms and dynamically adjust their flight paths based on mission priorities and environmental changes. By introducing constraint optimization techniques, a safe distance is maintained between the aircraft, while effectively controlling their energy consumption and flight speed, thereby achieving efficient multi-mission collaborative execution.

[0077] In summary, this application has the following advantages:

[0078] By combining reinforcement learning with constraint optimization, the mission planning strategy is dynamically adjusted according to real-time environmental changes during mission execution, ensuring that the aircraft can optimally complete the mission in complex environments and improving the mission success rate.

[0079] Through adaptive optimization using reinforcement learning algorithms, the aircraft can automatically adjust path planning and task scheduling based on mission requirements and environmental changes, thereby reducing the total mission execution time and significantly improving mission completion efficiency. Especially in complex environments, the aircraft can adjust decisions based on real-time feedback, avoiding inefficient paths and unnecessary delays.

[0080] By incorporating constraint optimization, the aircraft can optimize energy consumption while ensuring mission success. Through optimization of flight speed, path selection, and flight strategy, the aircraft can minimize energy waste and ensure efficient mission completion with limited battery capacity.

[0081] By introducing multiple constraints and employing reinforcement learning and constraint optimization, the aircraft can adjust its mission execution strategy in real time within a dynamic environment, ensuring its safety in complex conditions. The aircraft can also avoid violating safety constraints during mission execution, reducing the risk of flight accidents.

[0082] Finally, it should be noted that the accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other.

[0083] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic mission planning method for aircraft based on reinforcement learning and constraint optimization, characterized in that, include: The mission planning problem is modeled as a Markov decision process, defining the state space, action space, reward function and state transition probability, and establishing a mission execution model for the aircraft. Flight missions are generated by the aircraft's mission execution model. The flight missions are used as inputs and reinforcement learning is performed through a deep Q-learning network algorithm. The optimal mission planning strategy is trained based on experience replay and target network mechanism to obtain the optimal flight path and mission scheduling strategy for the aircraft. Set up a constraint optimization module, and set the speed constraint, energy consumption constraint and minimum safe distance constraint of the aircraft within the constraint optimization module; When the task scheduling strategy exceeds the constraint optimization strategy, the current task scheduling strategy is executed according to the corresponding constraint value.

2. The aircraft dynamic mission planning method based on reinforcement learning and constraint optimization as described in claim 1, characterized in that, The state space S defines all possible states of the aircraft. t ; state s t Represented as: Where, x t y t , z t v represents the position coordinates of the aircraft in three-dimensional space. t θ represents the current velocity of the aircraft. t E represents the heading angle. t Remaining energy; Action space A defines the control actions that the aircraft can take; each action a t Represents the flight decision of the aircraft; State A t It can be represented as: Where, Δv t Δθ represents the change in the velocity of the aircraft. t The change in heading angle, Δz t This refers to the change in flight altitude. Reward function R(s) t a t The reward function R(s) is used to measure the feedback after an aircraft performs a certain action. t a t The design is as follows: R(s t ,a t )=α1·T c (s t ,a t )+α2·Fe(s t ,a t )-α3·E c (s t ,a t )-α4·Cv(s t ,a t ); Among them, T c (s t a t ) is a reward given based on the degree of task completion; F e (s t a t (F) is a reward given based on the shortest flight path and flight efficiency; c (s t a t C is the penalty imposed based on the aircraft's energy consumption; v (s t a t It is used to punish behaviors that violate the constraints of the aircraft.

3. The aircraft dynamic mission planning method based on reinforcement learning and constraint optimization as described in claim 2, characterized in that, When performing reinforcement learning using deep Q-learning network algorithms, the Q-values ​​are approximated. The Q-value represents the expected total reward for choosing a particular action in a given state. The Q-value is updated using the Bellman equation: Where α is the learning rate, γ is the discount factor, and r t For instant rewards, The maximum Q value in the future state; The input to the deep Q-learning network algorithm is the current state s. t The output is the Q-value corresponding to each action; during this process, the aircraft continuously updates the network weights through interaction with the environment. After each interaction with the environment, the aircraft stores a quadruple of state, action, reward, and next state in the experience pool. Each time it is updated, a batch is randomly selected from the experience pool for training until the optimal task planning strategy is obtained based on the Q value.

4. The aircraft dynamic mission planning method based on reinforcement learning and constraint optimization as described in claim 3, characterized in that, The constraint optimization module is designed as follows: Define the penalty function P(s) for constraint optimization. t a t )for: P(s t ,a t )=λ1·S c (s t ,a t )+λ2·D c (s t ,a t )+λ3·e c (s t ,a t ); Among them, the speed constraint S c (s t a t (v) represents the speed of the aircraft. t Cannot exceed the maximum flight speed v max or below the minimum flight speed v min ; Among them, the speed constraint S c (s t a t )for: S c (s t ,a t )=max(0,v t -v max )+max(0,v min -v t ); Safe distance D c (s t a t The constraint is that the aircraft needs to maintain a minimum safe distance d from obstacles or other aircraft. s ,for: Energy consumption constraint e c (s t a t E represents the energy consumption of the aircraft. t It cannot exceed the maximum energy E max ,for: and c (s t ,the t ) = max(0, E t -AND max ) 5. The aircraft dynamic mission planning method based on reinforcement learning and constraint optimization as described in claim 4, characterized in that, Set a threshold for the remaining energy of the aircraft. If the remaining energy of the aircraft is less than the threshold after the mission scheduling strategy has been executed, the mission scheduling strategy is deemed to have failed.

6. The aircraft dynamic mission planning method based on reinforcement learning and constraint optimization as described in claim 1, characterized in that, A ring-shaped sensing module is set up, which can collect other environmental parameters, including wind speed, weather conditions, and the positions of other aircraft. When the positions of other aircraft are detected, the flight heading is adjusted according to the angle between the current heading and the positions of other aircraft. Different altitude coefficients are set for wind speed and weather conditions. When wind speed and weather conditions change, the current flight speed is adjusted according to the corresponding altitude coefficient.