Fixed-wing aircraft high-maneuver flight control method based on course-based reinforcement learning

Abstract

The application relates to the technical field of flight control, in particular to a fixed-wing aircraft high-maneuver flight control method based on course reinforcement learning, a closed-loop learning system containing a course scheduler, an intelligent agent, a simulation environment and an experience buffer is constructed, and the intelligent agent has a strategy network; a state space of an aircraft and a normalized rudder and throttle action space output by the strategy network are defined. A course difficulty measurement model is established, the task difficulty is quantified by weighting the state deviation and the envelope penalty term; based on the current course level and the model, a training task set that is difficult to match and safe is dynamically generated. In the simulation environment, the strategy network is iteratively updated through double-loop training, and a high-maneuver flight control strategy is obtained after all courses are completed; the strategy is deployed to the flight control system to realize high-maneuver flight control based on real-time state. Efficient, safe and adaptive flight control strategy training and deployment are realized.

Description

Technical Field

[0001] This application relates to the field of flight control technology, specifically to a high-maneuverability flight control method for fixed-wing aircraft based on curriculum-based reinforcement learning. Background Technology

[0002] Traditional flight control methods, such as proportional-integral-derivative (PID) controllers or linear quadratic regulators (LQRs), typically rely on precise mathematical models and parameter tuning. However, these methods exhibit significant limitations when handling high-maneuverability missions. High-maneuverability flight involves operations at the flight envelope boundaries, such as sharp turns, steep climbs, or roll maneuvers, requiring the control system to adapt quickly to nonlinear, coupled dynamic characteristics. Traditional methods, lacking learning and adaptability, often require complex gain scheduling or manual adjustments, making it difficult to achieve optimal control in unknown or dynamic environments, and are prone to control command jitter or instability.

[0003] To overcome the limitations of traditional methods, deep reinforcement learning (DRL) has been introduced into the field of flight control. This method autonomously learns control strategies through a trial-and-error mechanism, theoretically possessing the potential to handle highly nonlinear and coupled systems. However, directly applying DRL to high-maneuverability flight control faces numerous challenges. The process of an agent exploring a vast state space from scratch is typically inefficient, with slow convergence, and prone to policy oscillations during high-risk maneuvers, leading to discontinuous control commands and directly impacting flight quality. Existing flight control methods employing reinforcement learning often focus on high-level tactical decision-making or trajectory tracking, while their underlying control often prioritizes stability, with conservative reward function designs that fail to fully utilize the high-maneuverability potential of the aircraft at its envelope boundaries.

[0004] Curriculum learning, as a progressive training paradigm, has shown promise in the field of machine learning. It accelerates the learning process and improves stability by breaking down complex tasks into increasingly difficult sub-tasks. In aviation applications, curriculum learning has been attempted for path planning, but its application to underlying flight control systems remains in the exploratory stage. Existing methods often lack close integration with flight dynamics; for example, they lack mechanisms for quantifying the difficulty of state transitions, leading to subjective or ineffective curriculum design. Furthermore, some methods focus on directly generating control surface commands, which limits their versatility to the aerodynamic parameters of a specific aircraft.

[0005] In summary, existing technologies present a significant contradiction between training efficiency, control performance, and system versatility when addressing the high-maneuverability flight control problem of fixed-wing aircraft. Summary of the Invention

[0006] To address the aforementioned issues, this invention provides a high-maneuverability flight control method for fixed-wing aircraft based on curriculum-based reinforcement learning. By deeply integrating curriculum-based learning and reinforcement learning, and introducing an objective state difficulty metric, it achieves safe, efficient, and universal high-maneuverability flight control.

[0007] The present invention provides a high-maneuverability flight control method for fixed-wing aircraft based on curriculum-based reinforcement learning, comprising the following steps: S1. Construct a curriculum-based reinforcement learning closed-loop system, the system including a curriculum scheduler, a reinforcement learning agent, a flight simulation environment and an experience buffer, the reinforcement learning agent including a policy network for generating control commands; S2. Define the flight state of the fixed-wing aircraft in the flight simulation environment as the state space, and define the output of the reinforcement learning agent, i.e. the instructions generated by the policy network according to the state space, as the action space. The action space is the normalized control surface and throttle control instructions. S3. Based on the state space, establish a course difficulty measurement model for quantifying task difficulty. The model calculates the deviation between each state dimension and the target state by weighted calculation and introduces an envelope constraint penalty term to comprehensively output the task difficulty coefficient. S4. Based on the current course level and the course difficulty measurement model, dynamically generate a series of training tasks that match the difficulty and comply with the flight envelope safety constraints, forming the training task set for the current level. S5. Based on the training task set, training is performed in the flight simulation environment. The policy network of the reinforcement learning agent is iteratively updated through the outer course level loop and the inner training round loop until the training of all course levels is completed, and the final high-maneuverability flight control strategy implemented by the trained policy network is obtained. S6. Deploy the final high-maneuverability flight control strategy to the flight control system of the fixed-wing aircraft, so that the system can generate high-maneuverability flight control commands based on the strategy and the real-time flight status.

[0008] By integrating a course scheduler, difficulty metric, and safety constraint verification, a progressive and measurable training framework was constructed, systematically solving the technical challenges of low exploration efficiency, unstable training, and the generation of dangerous commands in complex flight control tasks using traditional reinforcement learning. The final flight control strategy is derived from thorough learning of full-envelope, high-difficulty maneuvering tasks in a simulation environment, thus achieving precise adaptation to the dynamics of fixed-wing aircraft. In real flight, it exhibits faster state convergence, smoother command output, and stronger disturbance resistance.

[0009] As a further limitation of the technical solution of the present invention, the state space For a multidimensional vector, its elements must include at least: position in a given coordinate system. Three-axis velocity components Pitch angle Roll angle Yaw angle , corresponding angular velocity Altitude and target heading angle deviation Target roll angle deviation Deviation from target speed .

[0010] The state vector comprehensively represents the aircraft's six degrees of freedom motion state and error information with the target, providing the agent with all the environmental perception information needed to make accurate decisions. Including the target deviation allows reward function design and policy learning to be directly geared towards reducing tracking error, fundamentally ensuring the accuracy of the control strategy.

[0011] As a further limitation of the technical solution of the present invention, the action space A four-dimensional vector ,in For normalized elevator commands, For normalized aileron commands, For normalized rudder commands, The throttle commands are normalized, and each command value is limited to the range [-1, 1].

[0012] The commands directly correspond to the aircraft's four main control channels, ensuring that the policy network output can directly drive the aircraft, covering all basic requirements for attitude and speed control. Normalization masked the differences in specific actuator parameters across different aircraft, making the trained policy easier to adapt to different aircraft models through scaling, thus improving the method's versatility.

[0013] As a further limitation of the technical solution of the present invention, the course difficulty measurement model calculates the task difficulty coefficient using the following formula. :

[0014] in, and These represent the current state of the aircraft and the state of the target, respectively. Indicates the first The deviation of each state dimension; For the first Weight coefficients for each state dimension; This is the corresponding normalization function; To constrain the penalty coefficient; This is a penalty term for envelope constraint;

[0015] In the formula, To measure the target state Violation of Article The instantaneous risk item of the degree of constraint of the envelope, To measure from state arrive The gradient risk term of the transfer trend risk; , and To adjust the weighting coefficients, the instantaneous risk item The specific definition is:

[0016] In the formula, and They represent the first The safety envelope range of each state quantity. For the target state of the first Each parameter value.

[0017] The gradient risk term The expression is:

[0018] in, The current state of the first Parameter values, This represents the amount of change in the state parameters; The envelope safety boundary value closest to the current state; For safety reasons, This represents the maximum value of the parameter change.

[0019] Transforming subjective task difficulty into a computable metric based on state deviation provides an objective basis for the automated and scientific design of courses. Gradient risk terms can preemptively penalize control objectives that require drastic state changes and rapid depletion of safety margins, thus guiding the agent to learn smooth and gradual approaches to the target during the course task generation phase. This suppresses drastic jitter in control commands and envelope overstepping risks from the source, improving the safety of the training process and the final strategy.

[0020] As a further limitation of the technical solution of the present invention, in S4, the step of dynamically generating the training task set includes: S41. Determine the target difficulty range based on the current course level; S42. Randomly sample candidate target states from the state space; S43. Calculate the difficulty coefficient of the candidate target state using the course difficulty measurement model; S44. Determine whether the difficulty coefficient is within the target difficulty range. If yes, determine that the difficulty verification has been passed and execute S45; otherwise, return to S42 to resample. S45. Verify the flight envelope safety of candidate target states that have passed the difficulty verification. S46. Add the candidate target states that have passed security verification to the training task set, and repeat S42 to S46 until a specified number of training tasks are generated or the maximum number of samplings is reached.

[0021] The system can dynamically generate task sets of appropriate difficulty based on the current course level, accelerating the training convergence process. Using flight envelope constraints as a hard filter condition for task generation ensures that all target states explored in the training environment are physically flyable and safe, preventing the agent from learning dangerous or ineffective strategies and guaranteeing the quality of training data.

[0022] As a further limitation of the technical solution of the present invention, the flight envelope safety verification in S45 specifically includes: Obtain a predefined set of flight envelope constraints, wherein the set of constraints at least limits the range of values ​​for flight speed, angle of attack, normal overload, and roll angle; Extract the state parameter values ​​from the candidate target states that correspond to the constraint set; Determine whether each extracted state parameter value is within its corresponding range; if so, the security verification is passed; otherwise, it is failed.

[0023] As a further limitation of the technical solution of the present invention, the steps in S5 include: S51. Setting Course Levels Initial values ​​and maximum course level The outer course level is iterated in a loop as follows: S52, at the current course level Next, proceed Each inner training round consists of steps S521-S522: S521. Sample a training task from the current level's training task set and initialize the state of the flight simulation environment; S522, Maximum allowed step size in a single training round Within, the following steps a)-e) are executed in a loop. a) The reinforcement learning agent, based on its policy network Based on the current environmental status Generate Actions ; b) Perform actions in the flight simulation environment To obtain the next state and instant rewards ; c) Transfer sample ( , , , Stored in the experience buffer; d) When the amount of data in the experience buffer reaches the predetermined batch size B, sample data from the experience buffer and update the policy network parameters using a reinforcement learning algorithm. ; e) If the task is completed or the step size is reached... If the current time step loop terminates, the loop continues; otherwise, it proceeds to the next time step. S53, after completion After training rounds, based on the reinforcement learning agent's recent performance... The average performance metrics in each training round are used by the course scheduler to determine whether the conditions for course advancement or backtracking have been met; among them, the average performance metrics include the average success rate and / or average number of convergence steps of the reinforcement learning agent in the training rounds. S54. Based on the course scheduler's decision, upgrade, maintain, or downgrade the current course level. ; S55. Repeat steps S52 to S54 until the course level is reached. achieve And the training completion conditions are met, thus obtaining the final high-maneuverability flight control strategy realized by the trained policy network.

[0024] As a further limitation of the technical solution of the present invention, a hybrid reward function is adopted during the training process. The reward signal for each time step is calculated using the following expression:

[0025] In the formula, Indicates sparse rewards; , indicating dense reward elements; and These are the weighting coefficients for sparse and dense rewards, respectively;

[0026] In the formula, This is the base reward value for the current course level; This is the time reward coefficient; This represents the maximum allowed step size for the task. This is the current time step; Indicates the current state With the target state The Euclidean distance between them; Course Level The success threshold is as follows;

[0027] in, and These are the threshold and basic reward for the initial course level, respectively; The threshold attenuation coefficient, As a reward growth coefficient, The current course level;

[0028] In the formula, This represents the change in state distance between adjacent time steps; This is the distance reward coefficient. The distance to the maximum possible state;

[0029] In the formula, and These represent the action vectors of the current and previous time steps, respectively. This is the smoothness penalty coefficient;

[0030] In the formula, Indicates the first Each envelope constraint variable; This represents the upper limit of the corresponding constraint; These are the constraint weighting coefficients.

[0031] As a further limitation of the technical solution of the present invention, The dynamic adjustment during the course-based learning process is expressed as follows:

[0032] in, This is the initial sparse reward weight; Adjust the total amount for weighting. This represents the current course level.

[0033] This approach addresses the challenge of exploration under sparse rewards while directly suppressing instruction jitter through smooth action rewards. As the course level increases, the weight of sparse rewards is gradually increased, meaning that more emphasis is placed on precise task completion in advanced stages, while relying more on the gradual guidance of dense rewards in beginner stages. This dynamic adjustment ensures that reward signals are always optimally matched to the learning stage, further optimizing learning efficiency and the accuracy of the final strategy.

[0034] As can be seen from the above technical solutions, this application has the following advantages: This application integrates components such as a course scheduler, a reinforcement learning agent, a flight simulation environment, and an experience buffer to form a closed-loop learning system. The course scheduler dynamically generates progressive training tasks based on state distance metrics, the agent learns control strategies through interaction with the simulation environment, the experience buffer supports strategy updates, and the performance feedback mechanism ensures that the course progress matches the learning ability. This achieves a smooth transition from basic flight to high-maneuverability scenarios and improves training efficiency.

[0035] The state distance metric objectively quantifies task difficulty by weighted calculation of differences in key parameters such as heading angle, roll angle, and velocity. Based on this, the course scheduler sequentially trains tasks, gradually transitioning from simple maneuvers like fine-tuning heading to high-maneuverability tasks such as large-angle turns. This progressive approach reduces exploration randomness, accelerates convergence, and ensures training stability.

[0036] The training of the reinforcement learning agent adopts a proximal policy optimization algorithm, the experience buffer adopts a priority sampling mechanism, focuses on the replay of high-value experiences, and dynamically adjusts the course progress through performance indicators such as average convergence time. Attached Figure Description

[0037] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the method provided in an embodiment of the present invention.

[0039] Figure 2 A flowchart for the task generation algorithm.

[0040] Figure 3 This is a flowchart of the course scheduler's workflow. Detailed Implementation

[0041] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this application and in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0043] like Figure 1 As shown, this embodiment of the invention provides a high-maneuverability flight control method for fixed-wing aircraft based on curriculum-based reinforcement learning, including the following steps: S1. Construct a curriculum-based reinforcement learning closed-loop system. The system includes a curriculum scheduler, reinforcement learning agents, a flight simulation environment, and an experience buffer. The reinforcement learning agents contain a policy network for generating control commands. The flight simulation environment simulates the dynamics of a fixed-wing aircraft, employing a six-degree-of-freedom motion model to approximate the real environment. The reinforcement learning agents are trained using the Proximal Policy Optimization (PPO) algorithm. The experience buffer uses a priority sampling mechanism, emphasizing the replay of high-value experiences, and dynamically adjusts the curriculum progress based on performance metrics such as average convergence time.

[0044] S2. Define the flight state of the fixed-wing aircraft in the flight simulation environment as the state space, and define the output of the reinforcement learning agent, i.e. the instructions generated by the policy network according to the state space, as the action space. The action space is the normalized control surface and throttle control instructions. state space The complete dynamic characteristics of an aircraft are characterized using multidimensional vectors, and their mathematical expression is as follows:

[0045] The physical meanings of each parameter are shown in Table 1: Table 1. Definition of State-Space Parameters

[0046] Action space The design employs normalized lever input commands, and its mathematical expression is as follows: ,

[0047] The specific action parameters are defined as shown in Table 2: Table 2 Definition of motion space parameters

[0048] State space and action space are connected through a policy network. Establish a mapping relationship. This mapping must satisfy flight dynamics constraints:

[0049] in Representing the equations of motion of an aircraft, This design addresses environmental disturbances and ensures the physical consistency of the state-action space, providing a mathematical model foundation for subsequent lessons.

[0050] S3. Based on the state space, establish a course difficulty measurement model for quantifying task difficulty. The model calculates the deviation between each state dimension and the target state by weighted calculation and introduces an envelope constraint penalty term to comprehensively output the task difficulty coefficient. The course difficulty measurement model is a multi-dimensional difficulty measurement function established based on state space. This function forms a unified difficulty assessment standard by weightedly fusing the deviations of each state dimension. Its mathematical expression is as follows:

[0051] in, and These represent the current state and the desired state of the aircraft, respectively. Indicates the first The deviation of each state dimension; For the first Weight coefficients for each state dimension; A dimension-specific normalization function; To constrain the penalty coefficient; This is a penalty term for envelope constraint. The physical meaning of quantifies the risk of violating the flight envelope during state transition, and its mathematical expression is:

[0052] In the formula, This is a transient risk factor that measures the target state. The The degree to which each parameter violates the envelope; As a gradient risk term, evaluate from state arrive The risk of a shifting trend; , and To adjust the weighting coefficients. Among them, the instantaneous risk item... The specific definition is:

[0053] In the formula, and They represent the first The safety envelope range of each state quantity. For the target state of the first Each parameter value.

[0054] Gradient risk term The expression is:

[0055] in, The current state of the first Parameter values, This represents the amount of change in the state parameters; The envelope safety boundary value closest to the current state; This represents the safety margin between the current state and the nearest envelope boundary. For safety, the value is set to 0.5; This represents the maximum value of the parameter variation, used for normalization. This penalty applies to control targets that change too aggressively and consume too much safety margin.

[0056] The envelope constraint has four terms. Taking a manned fixed-wing aircraft as an example, the corresponding constraint parameter settings and typical boundary sizes are shown in Table 3: Table 3 Definition of Envelope Constraint Parameters

[0057] Weight coefficients for each state dimension and normalization function Its specific configuration parameters are shown in Table 4.

[0058] Table 4. State Dimension Weight Allocation Table

[0059] S4. Based on the current course level and the course difficulty measurement model, dynamically generate a series of training tasks that match the difficulty and comply with the flight envelope safety constraints, forming the training task set for the current level. The task generation algorithm is the execution component of the curriculum-based learning mechanism. Its function is to dynamically generate a series of training tasks of appropriate difficulty that comply with flight safety constraints, based on the current course level and difficulty metric model. This algorithm ensures that the agent can gradually transition from simple maneuvers to complex high-maneuver scenarios during training, thereby optimizing learning efficiency and ensuring training stability. Following the principles of difficulty matching and safety priority, it utilizes the difficulty range provided by the course scheduler and the calculation results of the course difficulty metric model to sample target states from the state space. A dual verification mechanism ensures that the generated tasks meet both the current course difficulty requirements and the flight envelope constraints. Its core idea is to construct a set of training tasks covering the target difficulty range through random sampling and iterative filtering. The task generation algorithm's flow is as follows: Figure 2 As shown, it specifically includes: S41. Determine the target difficulty range based on the current course level; S42. Randomly sample candidate target states from the state space; S43. Calculate the difficulty coefficient of the candidate target state using the course difficulty measurement model; S44. Determine whether the difficulty coefficient is within the target difficulty range. If yes, determine that the difficulty verification has been passed and execute S45; otherwise, return to S42 to resample. S45. Perform flight envelope safety verification on the candidate target states that have passed the difficulty verification; in this step, the flight envelope safety verification specifically includes: Obtain a predefined set of flight envelope constraints, wherein the set of constraints at least limits the range of values ​​for flight speed, angle of attack, normal overload, and roll angle; Extract the state parameter values ​​from the candidate target states that correspond to the constraint set; Determine whether each extracted state parameter value is within its corresponding range; if so, the security verification is passed; otherwise, it is failed.

[0060] S46. Add the candidate target states that have passed security verification to the training task set, and repeat S42 to S46 until a specified number of training tasks are generated or the maximum number of samplings is reached.

[0061] During the initialization phase, the target state set is set to empty, and the difficulty range corresponding to the current course level is obtained. Then, cyclic sampling begins, randomly sampling candidate target states from the state space as training tasks. Each candidate state must have its difficulty determined using the course difficulty metric model, ensuring it falls within the target range for the current course level; otherwise, it is resampled. After passing difficulty verification, the candidate state enters the safety verification phase, where the envelope constraint detection module ensures it conforms to flight dynamics constraints. Only after passing both verifications is the candidate state added to the final task set. The process terminates when a specified number of qualified tasks are generated or the maximum sampling limit is reached.

[0062] S5. Based on the training task set, training is performed in the flight simulation environment. The policy network of the reinforcement learning agent is iteratively updated through outer course level loops and inner training round loops until training for all course levels is completed, obtaining the final high-maneuverability flight control strategy implemented by the trained policy network. The training steps include: S51. Setting Course Levels Initial values ​​and maximum course level The outer course level is iterated in a loop as follows: S52, at the current course level Next, proceed Each inner training round consists of steps S521-S522: S521. Sample a training task from the current level's training task set and initialize the state of the flight simulation environment; S522, Maximum allowed step size in a single training round Within, the following steps a)-e) are executed in a loop. a) The reinforcement learning agent, based on its policy network Based on the current environmental status Generate Actions ; b) Perform actions in the flight simulation environment To obtain the next state and instant rewards ; c) Transfer sample ( , , , Stored in the experience buffer; d) When the amount of data in the experience buffer reaches the predetermined batch size B, sample data from the experience buffer and update the policy network parameters using a reinforcement learning algorithm. ; e) If the task is completed or the step size is reached... If the current time step is not reached, the loop terminates; otherwise, it proceeds to the next time step. S53, after completion After training rounds, based on the reinforcement learning agent's recent performance... The average performance metrics in each training round are used by the course scheduler to determine whether the conditions for course advancement or backtracking have been met; among them, the average performance metrics include the average success rate and / or average number of convergence steps of the reinforcement learning agent in the training rounds. S54. Based on the course scheduler's decision, upgrade, maintain, or downgrade the current course level. ; S55. Repeat steps S52 to S54 until the course level is reached. achieve And the training completion conditions are met, thus obtaining the final high-maneuverability flight control strategy realized by the trained policy network.

[0063] During training, a hybrid reward strategy combining sparse and dense rewards is employed. The reward structure is dynamically adjusted through a course adaptation mechanism to balance exploration efficiency and learning stability. The dense reward function provides continuous, fine-grained feedback signals at each time step, guiding the agent to gradually approach the target behavior through reward shaping techniques. A hybrid reward function is used. The reward signal for each time step is calculated using the following expression:

[0064] In the formula, Indicates sparse rewards; , indicating dense reward elements; and These represent the weighting coefficients for sparse and dense rewards, respectively. The sparse reward function provides a one-time reward signal when the agent achieves a specific goal; its design focuses on defining the criteria for task success and the incentive strength for the final goal. The definition of the reward for successful achievement is as follows: The mathematical expression is:

[0065] In the formula, This is the base reward value for the current course level; This is the time reward coefficient; This represents the maximum allowed step size for the task. This is the current time step; Indicates the current state With the target state The Euclidean distance between them; Course Level The success threshold is as follows;

[0066] in, and These are the threshold and basic reward for the initial course level, respectively; , which is the threshold decay coefficient, makes the criteria for judging higher-level courses more stringent; To reward the growth factor, ensuring that high-difficulty tasks yield correspondingly high rewards; This represents the current course level.

[0067] Status Distance Reward The agent is encouraged to converge toward the target state, expressed as:

[0068] In the formula, This represents the change in state distance between adjacent time steps; This is the distance reward coefficient, which controls the scale of the reward signal; The maximum possible state distance is used for normalizing the reward value.

[0069] Motion smoothness reward The expression for drastic changes in punishment control instructions is:

[0070] In the formula, and These represent the action vectors of the current and previous time steps, respectively. This is the smoothness penalty coefficient; The envelope constraint reward penalizes violations of the flight envelope, expressed as:

[0071] In the formula, Indicates the first Individual envelope constraint variables (velocity, angle of attack, normal overload, roll angle); This represents the upper limit of the corresponding constraint; These are the constraint weighting coefficients.

[0072] The weights are dynamically adjusted during the course-based learning process, increasing continuously as the course progresses. The expression is:

[0073] in, This is the initial sparse reward weight (which can be set to 0.3). Adjust the total amount for weighting; This represents the total number of course levels. This represents the current course level.

[0074] This reward function design defines the ultimate goal through sparse rewards, provides progressive guidance through dense rewards, and ensures that the reward signals match the learning stages through a course adaptive mechanism, together forming a complete learning incentive system.

[0075] The training algorithm process is as follows: Input: Initial course level; Maximum course level; Performance evaluation period (number of training rounds); : Batch size for strategy updates; Success rate threshold for course advancement; : Success rate threshold for course review; : The size of the sliding window for performance evaluation; Flight simulation environment model; Output: : Parameters of the trained policy network; Initialize system components: Initialize the experience replay buffer ; Initialize policy network With value network ; Initialize the course scheduler state ; Course level iteration: for current course level to do : Based on course level Difficulty parameters are used to generate training task sets.

[0076] while the advanced condition is not met and the callback condition is not met, do: for training rounds to do : from Mid-sampling task Initialize environment state

[0077] for time step to do : The agent, according to the policy Select Action The environment performs an action and obtains a state. and rewards

[0078] Storage and transfer samples To the experience buffer

[0079] if experience quantity then: from Mid-sample batch data, using the PPO algorithm to update network parameters.

[0080] end if if the task terminates then break end for (time step loop) end for (training cycle) Performance evaluation and course scheduling: Based on recent Calculate the average success rate for each round. and average number of convergence steps

[0081] Update course scheduler status

[0082] end while Get the next course level

[0083] end for (course level cycle) The course scheduler dynamically manages the training process based on a performance feedback mechanism. By monitoring real-time metrics such as the agent's task success rate, convergence time, and control stability, the system establishes closed-loop control logic to adaptively adjust the training difficulty. The scheduler's workflow is as follows: Figure 3 As shown, the specific steps include: Step 1: Initialize course difficulty D Determine the initial difficulty level of the course, denoted as D. This initial value can be set based on factors such as the basic requirements of the course design and the basic level of the target audience.

[0084] Step 2: Generate the task set corresponding to difficulty D Based on the currently set course difficulty D, generate a set of tasks that match it. These tasks should reflect the knowledge and skill requirements of this difficulty level and be used for subsequent training and testing of the agent.

[0085] Step 3: Train the agent using the task set The agent is trained on this generated task set. During training, the agent attempts to complete tasks, learns relevant patterns or rules, etc.

[0086] Step 4: Collect performance metrics P After the agent completes training, performance metrics P are collected to measure the agent's performance. Performance metrics can take various forms, such as accuracy, completion time, score, and error rate, and need to be determined according to the task type and evaluation objectives.

[0087] Step 5: Performance Evaluation The collected performance metric P is compared with two pre-set thresholds. and By comparing the different ranges, we can determine the appropriate strategy for adjusting the difficulty of the course. If the performance metric P is greater than or equal to the high performance threshold This indicates that the agent is performing well enough at the current difficulty level. It is advisable to increase the difficulty of the course to make subsequent tasks more challenging in order to further explore the potential of the agent.

[0088] If the performance metric P is between the low performance threshold and the high performance threshold, that is... This indicates that the current difficulty level is appropriate for the agent, so we choose to keep the course difficulty unchanged and continue training and evaluation at the current difficulty level.

[0089] If the performance metric P is less than or equal to the low performance threshold This means that the current difficulty level is too high for the intelligent agent, making it difficult for the agent to complete the corresponding tasks well. Therefore, it is necessary to reduce the difficulty of the course to make the tasks more in line with the agent's current level, so as to help the agent consolidate its foundation and gradually improve.

[0090] Step 6: Update Difficulty D Based on the conclusions obtained from the performance evaluation in step 5, the course difficulty D is updated accordingly (increased, maintained, or decreased) to obtain a new difficulty level. .

[0091] Step 7: Determine if the termination condition has been met. Check if the preset termination conditions are met. Termination conditions may include reaching the preset maximum number of training rounds, performance metrics remaining within a certain stable range for a number of consecutive times, or reaching the target performance level.

[0092] If the termination condition is not met: Return to step 2, based on the updated difficulty. Regenerate the task set, and then repeat the process of training → collecting metrics → evaluating → updating difficulty, continuously iterating and optimizing.

[0093] If the termination condition is met, proceed to the next step.

[0094] Step 8: Course adjustments completed Once the termination conditions are met, the entire process of dynamically adjusting the course difficulty based on performance ends. At this point, the course difficulty has been adjusted to a relatively suitable (or target) state. Subsequently, it can be decided whether to stop training or proceed to other stages (such as formal application, results acceptance, etc.) based on actual needs.

[0095] The scheduler dynamically calculates advancement conditions based on historical performance data: when the success rate exceeds a set threshold for multiple consecutive training cycles. The system automatically upgrades course levels; it also has an adaptive rollback function that detects a continuous performance decline that falls below a threshold. The system automatically backtracks to adjust course difficulty to ensure training stability. The parameter adjustment mechanism dynamically changes the evaluation window size to adapt to different training stages. Initially, a smaller window is used to quickly respond to performance changes, while a larger window is used later to ensure evaluation stability. Performance thresholds are optimized based on the learning curve; initially, lower thresholds are set to encourage exploration, while requirements are gradually increased later.

[0096] The course scheduling strategy establishes a complete closed-loop control system of "state monitoring - performance evaluation - decision execution - feedback optimization", which can be applied to different types of fixed-wing aircraft control tasks.

[0097] S6. Deploy the final high-maneuverability flight control strategy to the flight control system of the fixed-wing aircraft, enabling the system to generate high-maneuverability flight control commands based on the strategy and real-time flight status. The final high-maneuverability flight control strategy is encapsulated as a software module executable on the flight control computer; the software module is integrated into the control loop of the flight control system and data connections are established with flight status sensors and flight actuators to form a closed-loop control system. The specific implementation process is as follows: Phase 1: Strategy Encapsulation and Consolidation The trained policy network is converted into a format suitable for embedded system deployment. Model quantization and pruning are performed to reduce computational load and memory consumption, meeting the real-time requirements of airborne computers.

[0098] The transformed policy network model is encapsulated into a standalone, callable flight control software module. The input interface of this module must strictly correspond to the state space vector defined during training. Its output interface must strictly correspond to the action space vector defined during training. .

[0099] Phase Two: Flight Control System Integration The flight control system's state construction module receives raw data in real time from various airborne sensors, as well as target commands from the mission management system. Following an algorithm identical to that used in the training phase, this data is calculated and combined into the complete state vector required by the policy network. .

[0100] The flight control system is configured to receive the normalized action vector output by the strategy network from the command calculation module. = Based on the actual physical travel range of the flight servo and the thrust curve of the engine, the normalized [-1,1] value is mapped to the actual control surface deflection angle command and throttle opening percentage command.

[0101] The encapsulated policy network module, state construction module, and command calculation module are integrated into the flight control main loop. This ensures that the policy network module can be stably invoked within a defined control cycle.

[0102] Phase 3: Real-time Control Loop After the system is powered on, the flight control system enters real-time operation. Within each control cycle, it executes the following sequentially: (1) Collect data and generate the current state .

[0103] (2) The input policy network module calculates the action in the forward computation. .

[0104] (3) The calculations are translated into physical commands and sent to the servo system and engine controller.

[0105] When high-maneuverability maneuvers are required, the mission management system transforms the maneuver objectives into target deviations in the state space. The state construction module then incorporates these target values. The calculation is performed on the policy network model, which automatically generates a series of smooth, fast control commands that do not exceed the control envelope based on the difference between the current state and the target, driving the aircraft to complete the maneuver.

[0106] Phase 4: Security and Validation While integrating the policy network, traditional law control must be retained and activated as a backup security layer. A safety monitoring module should be deployed to continuously check for anomalies in the policy network's output commands and whether the aircraft's status is approaching its envelope boundary. Upon detecting a hazard, the system should automatically switch back to traditional law control immediately to ensure flight safety.

[0107] Before the official flight, it must undergo rigorous hardware-in-the-loop simulation testing and progressive test flights to verify whether the stability, reliability and performance of the deployed system meet expectations.

[0108] This invention also provides a high-maneuverability flight control system for fixed-wing aircraft based on curriculum-based reinforcement learning, comprising: The system building module is used to construct the framework of a closed-loop system for curriculum-based reinforcement learning. The space definition module is used to define the flight state of a fixed-wing aircraft in a flight simulation environment as the state space and the output of the reinforcement learning agent as the action space, wherein the action space is the normalized control surface and throttle control commands. The difficulty measurement module is used to establish a course difficulty measurement model based on the state space. The model calculates the deviation between each state dimension and the target state by weighting and introduces an envelope constraint penalty term to comprehensively output the task difficulty coefficient. The task generation module is used to dynamically generate a series of training tasks that match the difficulty level and comply with the flight envelope safety constraints based on the current course level and the course difficulty measurement model, thus forming the training task set for the current level. The course training module is used to perform course-based reinforcement learning training in a flight simulation environment based on the training task set. Through the outer course level loop and the inner training round loop, the policy network of the reinforcement learning agent is iteratively updated until the training of all course levels is completed, and the final high-maneuverability flight control strategy implemented by the trained policy network is obtained. The application deployment module is used to deploy the final high-maneuverability flight control strategy to the flight control system of the fixed-wing aircraft, so that the system can generate high-maneuverability flight control commands based on the strategy and the real-time flight status. The curriculum-based reinforcement learning closed-loop system framework includes a curriculum scheduler, a reinforcement learning agent, a flight simulation environment, and an experience buffer. The reinforcement learning agent includes a policy network for generating control commands.

[0109] In some embodiments, the task generation module includes: The difficulty range determination unit is used to determine the target difficulty range based on the current course level. A state sampling unit is used to randomly sample candidate target states from the state space; The difficulty verification unit is used to calculate the difficulty coefficient of the candidate target state using the course difficulty measurement model, and to determine whether it is within the target difficulty range. The safety verification unit is used to verify the flight envelope safety of candidate target states that have passed the difficulty verification. The task set construction unit is used to add candidate target states that have passed security verification to the training task set and control the loop execution until a specified number of training tasks are generated or the maximum number of samplings is reached.

[0110] In some embodiments, the security verification unit is specifically used for: Obtain a predefined set of flight envelope constraints, wherein the set of constraints at least limits the range of values ​​for flight speed, angle of attack, normal overload, and roll angle; Extract the state parameter values ​​from the candidate target states that correspond to the constraint set; The system determines whether each extracted state parameter value is within its corresponding range to verify whether the security verification has been passed.

[0111] In some embodiments, the course training module includes: The loop control unit is used to set the initial course level. and maximum course level And control the cycling of outer course levels; Round execution unit, used in the current course level Next, execute Each inner training round consists of: sampling tasks from the current level's training task set and initializing the environment, and performing training within the maximum allowed step size. The inner loop executes strategy network action generation, environment interaction, experience storage, and strategy network parameter updates; The scheduling decision unit is used to complete the task. After training rounds, based on the agent's recent... The average performance metrics over each round are used by the course scheduler to determine whether the conditions for course advancement or rollback have been met, and based on this, a decision is made to upgrade, maintain, or downgrade the current course level. ; Strategy output unit, used at course level achieve When the training completion conditions are met, the final high-maneuverability flight control strategy implemented by the trained policy network is output.

[0112] The system is integrated into a computing device in the form of software, or is embedded in the flight control computer of a fixed-wing aircraft in the form of hardware.

[0113] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or other media capable of storing program code. It includes several instructions to cause a computer terminal (which may be a personal computer, server, or a second terminal, network terminal, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0114] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0115] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0117] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A high-maneuverability flight control method for fixed-wing aircraft based on curriculum-based reinforcement learning, characterized in that, Includes the following steps: S1. Construct a curriculum-based reinforcement learning closed-loop system, the system including a curriculum scheduler, a reinforcement learning agent, a flight simulation environment and an experience buffer, the reinforcement learning agent including a policy network for generating control commands; S2. Define the flight state of the fixed-wing aircraft in the flight simulation environment as the state space, and define the output of the reinforcement learning agent, i.e. the instructions generated by the policy network according to the state space, as the action space. The action space is the normalized control surface and throttle control instructions. S3. Based on the state space, establish a course difficulty measurement model for quantifying task difficulty. This model calculates the deviation between each state dimension and the target state using weighted averages, and introduces a boundary constraint penalty term to comprehensively output the task difficulty coefficient. The course difficulty measurement model calculates the task difficulty coefficient using the following formula. : in, and These represent the current state of the aircraft and the state of the target, respectively. Indicates the first The deviation of each state dimension; For the first Weight coefficients for each state dimension; This is the corresponding normalization function; To constrain the penalty coefficient; This is a penalty term for envelope constraint; In the formula, To measure the target state Violation of Article The instantaneous risk item of the degree of constraint of the envelope, To measure from state arrive The gradient risk term of the risk of the transfer trend; , and To adjust the weighting coefficients, the instantaneous risk item The specific definition is: In the formula, and They represent the first The safety envelope range of each state quantity. For the target state of the first S4. Based on the current course level and the course difficulty measurement model, dynamically generate a series of training tasks that match the difficulty and comply with the flight envelope safety constraints, forming the training task set for the current level. S5. Based on the training task set, training is performed in the flight simulation environment. The policy network of the reinforcement learning agent is iteratively updated through the outer course level loop and the inner training round loop until the training of all course levels is completed, and the final high-maneuverability flight control strategy implemented by the trained policy network is obtained. S6. Deploy the final high-maneuverability flight control strategy to the flight control system of the fixed-wing aircraft, so that the system can generate high-maneuverability flight control commands based on the strategy and the real-time flight status.

2. The high-maneuverability flight control method for fixed-wing aircraft based on curriculum-based reinforcement learning according to claim 1, characterized in that, state space For a multidimensional vector, its elements must include at least: position in a given coordinate system. Three-axis velocity components Pitch angle Roll angle Yaw angle , corresponding angular velocity Altitude and target heading angle deviation Target roll angle deviation Deviation from target speed .

3. The high-maneuverability flight control method for fixed-wing aircraft based on curriculum-based reinforcement learning according to claim 1, characterized in that, Action space A four-dimensional vector ,in For normalized elevator commands, For normalized aileron commands, For normalized rudder commands, The throttle commands are normalized, and each command value is limited to the range [-1, 1].

4. The high-maneuverability flight control method for fixed-wing aircraft based on curriculum-based reinforcement learning according to claim 3, characterized in that, The gradient risk term The expression is: in, The current state of the first Parameter values, This represents the amount of change in the state parameters; The envelope safety boundary value closest to the current state; For safety reasons, This represents the maximum value of the parameter change.

5. The high-maneuverability flight control method for fixed-wing aircraft based on curriculum-based reinforcement learning according to claim 1, characterized in that, In S4, the steps for dynamically generating the training task set include: S41. Determine the target difficulty range based on the current course level; S42. Randomly sample candidate target states from the state space; S43. Calculate the difficulty coefficient of the candidate target state using the course difficulty measurement model; S44. Determine whether the difficulty coefficient is within the target difficulty range. If yes, determine that the difficulty verification has been passed and execute S45; otherwise, return to S42 to resample. S45. Verify the flight envelope safety of candidate target states that have passed the difficulty verification. S46. Add the candidate target states that have passed security verification to the training task set, and repeat S42 to S46 until a specified number of training tasks are generated or the maximum number of samplings is reached.

6. The high-maneuverability flight control method for fixed-wing aircraft based on curriculum-based reinforcement learning according to claim 5, characterized in that, The flight envelope safety verification in S45 specifically includes: Obtain a predefined set of flight envelope constraints, wherein the set of constraints at least limits the range of values ​​for flight speed, angle of attack, normal overload, and roll angle; Extract the state parameter values ​​from the candidate target states that correspond to the constraint set; Determine whether each extracted state parameter value is within its corresponding range; if so, the security verification is passed; otherwise, it is failed.

7. The high-maneuverability flight control method for fixed-wing aircraft based on curriculum-based reinforcement learning according to claim 1, characterized in that, The steps in S5 include: S51. Setting Course Levels Initial values ​​and maximum course level The outer course level is iterated in a loop as follows: S52, at the current course level Next, proceed Each inner training round consists of steps S521-S522: S521. Sample a training task from the current level's training task set and initialize the state of the flight simulation environment; S522, Maximum allowed step size in a single training round Within, the following steps a)-e) are executed in a loop. a) The reinforcement learning agent, based on its policy network Based on the current environmental status Generate Actions ; b) Perform actions in the flight simulation environment To obtain the next state and instant rewards ; c) Transfer sample ( , , , Stored in the experience buffer; d) When the amount of data in the experience buffer reaches the predetermined batch size B, sample data from the experience buffer and update the policy network parameters using a reinforcement learning algorithm. ; e) If the task is completed or the step size is reached... If the current time step is not reached, the loop terminates; otherwise, it proceeds to the next time step. S53, after completion After training rounds, based on the reinforcement learning agent's recent performance... The average performance metrics in each training round are used by the course scheduler to determine whether the conditions for course advancement or backtracking have been met; among them, the average performance metrics include the average success rate and / or average number of convergence steps of the reinforcement learning agent in the training rounds. S54. Based on the course scheduler's decision, upgrade, maintain, or downgrade the current course level. ; S55. Repeat steps S52 to S54 until the course level is reached. achieve And the training completion conditions are met, thus obtaining the final high-maneuverability flight control strategy realized by the trained policy network.

8. The high-maneuverability flight control method for fixed-wing aircraft based on curriculum-based reinforcement learning according to claim 7, characterized in that, A hybrid reward function is used during the training process. The reward signal for each time step is calculated using the following expression: In the formula, Indicates sparse rewards; , indicating dense reward elements; and These are the weighting coefficients for sparse and dense rewards, respectively; In the formula, This is the base reward value for the current course level; This is the time reward coefficient; This represents the maximum allowed step size for the task. This is the current time step; Indicates the current state With the target state The Euclidean distance between them; Course Level The success threshold is as follows; in, and These are the threshold and basic reward for the initial course level, respectively; The threshold attenuation coefficient, As a reward growth coefficient, The current course level; In the formula, This represents the change in state distance between adjacent time steps; This is the distance reward coefficient. The distance to the maximum possible state; In the formula, and These represent the action vectors of the current and previous time steps, respectively. This is the smoothness penalty coefficient; In the formula, Indicates the first Each envelope constraint variable; This represents the upper limit of the corresponding constraint; These are the constraint weighting coefficients.

9. The high-maneuverability flight control method for fixed-wing aircraft based on curriculum-based reinforcement learning according to claim 8, characterized in that, The dynamic adjustment during the course-based learning process is expressed as follows: in, This is the initial sparse reward weight; Adjust the total amount for weighting. This represents the current course level.

Images