An adversarial reinforcement learning training method for robust control of fixed-wing aircraft

Abstract

This invention discloses an adversarial reinforcement learning training method for robust control of fixed-wing aircraft, comprising: constructing a nonlinear simulation environment and task set for the fixed-wing aircraft, and constructing several fixed-wing aircraft agents in parallel within the simulation environment. An adversarial agent module based on the AdvNet neural network is introduced to generate environmental disturbances. An Actor-Critic structure with near-end policy optimization is adopted, and the agents and adversarial agent modules are jointly trained through the task set to gradually obtain a robust control strategy. During training, based on a course learning mechanism, the disturbance constraint parameters of the adversarial agent are dynamically adjusted according to the completion of flight tasks. After training convergence, the obtained control strategy parameters are applied to the fixed-wing aircraft to achieve robust flight control. This invention improves the robustness and control accuracy of aircraft when facing complex environmental disturbances and is applicable to various flight missions with uncertainties.

Description

Technical Field

[0001] This invention belongs to the field of aircraft technology, specifically relating to an adversarial reinforcement learning training method for robust control of fixed-wing aircraft. Background Technology

[0002] Aircraft are widely used in both civilian and military fields, including unmanned aerial vehicles (UAVs), manned aircraft, and various vertical takeoff and landing (VTOL) aircraft. Among these, quadcopter UAVs are widely used due to their simple structure and high maneuverability. However, fixed-wing aircraft have significant advantages in long-range, high-speed missions due to their longer flight range, greater endurance, and higher speed. However, fixed-wing aircraft exhibit complex nonlinear dynamic characteristics during flight, including strongly coupled attitude and position control. The design of their control systems becomes particularly complex and challenging, especially when encountering uncertainties such as wind shear, gusts, and load variations.

[0003] Traditional control methods for fixed-wing aircraft are mostly based on classical control theory, such as PID control and state feedback control. These methods typically rely on the experience and knowledge of domain experts and require precise modeling and parameter tuning of the aircraft. While these methods demonstrate good stability and local performance, they suffer from poor adaptability and flexibility, long development cycles, and high costs, making them unsuitable for complex environments and dynamic tasks. Especially when facing complex environmental conditions, such as large-scale airflow variations or load disturbances, traditional control methods often struggle to cope.

[0004] In recent years, Deep Reinforcement Learning (DRL), as an end-to-end control method, has gradually emerged as a powerful alternative by automatically learning control policies from interactions. DRL does not rely on precise physical modeling but optimizes policies through interaction with the environment, thus enabling it to automatically learn control methods for complex tasks. For fixed-wing aircraft control, DRL has been successfully applied to attitude control, landing missions, and flight formation. However, DRL faces several challenges in practical applications, particularly in acquiring training data and handling environmental uncertainties.

[0005] In the prior art, Chinese patent CN119024878A discloses a method and apparatus for controlling a drone based on robust reinforcement learning and adversarial reinforcement learning. The method includes: initializing the drone system and obtaining the drone's current state; calculating the robust partial control quantity of the drone based on the current state, and determining whether reinforcement learning control needs to be triggered to output an accurate control quantity based on the robust partial control quantity; inputting the accurate control quantity into the drone system to obtain a reward; updating the reinforcement learning controller and the adversarial network, and determining whether the reinforcement learning controller and the adversarial network have converged, saving the network parameters, and completing the drone control based on robust reinforcement learning and adversarial reinforcement learning.

[0006] However, due to the high cost and risks of collecting flight data in real-world environments, most reinforcement learning methods are initially trained in simulation environments. While simulation environments can significantly reduce training costs and provide abundant training data, in real-world applications, aircraft are frequently affected by various disturbances, such as payload variations, airflow fluctuations, engine parameter mismatches, and gusts. These disturbances are often difficult to perfectly reproduce in existing simulation platforms, leading to potential performance degradation or failure of strategies trained in simulation environments in real-world applications.

[0007] To address this issue, domain randomization, as a common method, has been applied to the training process of reinforcement learning. Domain randomization improves the generality of policies by randomizing environmental parameters (such as wind speed and aircraft mass), enabling them to adapt to more varied environmental conditions. However, the limitation of domain randomization lies in its reliance on expert knowledge to define the range and distribution of perturbation parameters. This limits its coverage of complex perturbations and leaves it with insufficient adaptability to unknown perturbations.

[0008] In recent years, Adversarial Reinforcement Learning (ARL) has emerged to address this problem. ARL introduces adversarial agents to generate perturbations, exposing the vulnerability of the normal agent in the face of these perturbations, thus forcing it to learn a more robust control strategy. The adversarial agent continuously adjusts the strength and distribution of perturbations during training, exposing the weaknesses of the normal agent and prompting it to improve. However, existing ARL methods face several key challenges, such as high computational cost, uneven perturbation distribution, and unstable training processes, which limit their effectiveness in practical control tasks.

[0009] Therefore, how to ensure the efficiency of adversarial reinforcement learning while addressing issues such as uneven perturbation generation, training instability, and high computational costs has become a pressing technical challenge in adversarial reinforcement learning training for robust control of fixed-wing aircraft. Existing technologies have not adequately and effectively solved these challenges, thus adversarial reinforcement learning still faces significant technical bottlenecks and room for improvement in its application to robust control of fixed-wing aircraft. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the existing technology and provide an adversarial reinforcement learning training method for robust control of fixed-wing aircraft.

[0011] The objective of this invention can be achieved through the following technical solutions:

[0012] This invention provides an adversarial reinforcement learning training method for robust control of fixed-wing aircraft, comprising:

[0013] Construct a nonlinear simulation environment and task set for fixed-wing aircraft;

[0014] Several fixed-wing aircraft agents are constructed in parallel in the nonlinear simulation environment, and an adversarial agent module based on the neural network AdvNet is constructed; the adversarial agent module interacts with each agent and provides environmental perturbations.

[0015] Based on the near-end policy optimization Actor-Critic structure, the normal agent control strategy is obtained by joint training of the several fixed-wing aircraft agents and the adversarial agent module through the task set.

[0016] During training, based on the course learning mechanism, the disturbance constraint parameters of the adversarial agent module are dynamically adjusted according to the completion of the flight mission.

[0017] After training convergence, the control strategy parameters of the normal agent are used as the control strategy of the fixed-wing aircraft to achieve training of the fixed-wing aircraft.

[0018] Furthermore, several fixed-wing aircraft intelligent agents are constructed in parallel within the nonlinear simulation environment, specifically including:

[0019] In the nonlinear simulation environment, each fixed-wing aircraft agent is assigned an independent observation space and action space;

[0020] Initialize an Actor-Critic structure based on near-end policy optimization for each fixed-wing aircraft agent.

[0021] Furthermore, the observation space includes the current state and mission characteristics. The current state includes the aircraft's state variables, including target distance information, speed, attitude angle, and angular velocity. The mission characteristics are the deviation between the current state and the target state. The action space includes the control inputs of throttle, elevator, aileron, and rudder, and each control input has physical boundaries and normalization rules.

[0022] Furthermore, the Actor-Critic structure includes an Actor network and a Critic network; the Actor network generates control actions by outputting a continuous action distribution; the Critic network is used to estimate the value function of the current state; the Actor network and the Critic network adopt a neural network structure with three fully connected layers, each layer having 128 nodes, and a gated recurrent unit module is embedded in the Actor network and the Critic network.

[0023] Furthermore, the Actor-Critic structure based on near-end policy optimization, jointly trained by the several fixed-wing aircraft agents and the adversarial agent module through the task set, specifically includes:

[0024] In the nonlinear simulation environment, the strategy parameters of the current fixed-wing aircraft agent are used. The task set is executed, and trajectory data of the fixed-wing aircraft agent is generated through interaction with the adversarial agent module. ,in, For a moment t The observation state corresponding to the observation space. For a moment t The selected action space controls the actions; For a moment t The reward value; For a moment The observation status corresponding to the observation space; The length of the trajectory represents the number of time steps from the initial state to the final state; The current policy parameters represent the agent's policy parameters at the current moment, including the parameters of the Actor and Critic networks;

[0025] The return is calculated based on the obtained trajectory data using a discount and sum method. The formula is:

[0026]

[0027] in, As a discount factor, Expressing expectations, Indicates time tCounteracting environmental disturbances generated by the proxy module; For the reward function, Indicates at time t state Next action and suffered disturbance The reward given by the environment afterwards;

[0028] Utilize the rewards and dominance function The parameters of the Actor network are optimized using a proximal policy optimization algorithm. The objective function is:

[0029]

[0030] in, This represents the probability ratio between the current strategy and the old strategy. Let be the optimization objective function of the Actor network; The desired operation to be performed on the sample represents averaging over all trajectory samples; This is a cut operation; For shearing parameters;

[0031] The Critic network is based on the acquired observation status and returns Predictive value function According to the predicted value function The parameters of the Critic network are optimized using the loss function of the Critic network, as shown in the formula:

[0032]

[0033] in, Let be the loss function of the Critic network. This represents an estimated return, based on the return. Sure; For the Critic network to observe the state Value prediction;

[0034] In each training step, the network parameters are updated by optimizing the objective functions of the Actor and Critic networks.

[0035] Furthermore, the reward function is obtained based on the current state of the fixed-wing aircraft, the target state in the task set, control actions, and counter-disturbances, and includes the following parts:

[0036] Attitude tracking error penalty is used to measure the deviation between the current attitude of the aircraft and the attitude of the mission target, and it is derived from the observation state and the target state in the mission set;

[0037] Attitude stability penalty is used to constrain the rate of change of the aircraft's attitude angle, and it is derived from the state changes in continuous time steps;

[0038] The control input penalty is used to limit the control amplitude of throttle, elevator, aileron and rudder, and is derived from the control action output by the Actor network;

[0039] The control change penalty is used to constrain the changes in control actions at consecutive time steps. It is derived from the difference in control actions between the current time step and the previous time step.

[0040] Furthermore, during the process of the agent's task set, the adversarial agent module adjusts its response based on the current state. and the control actions output by the intelligent agent Generate disturbance Specifically, it includes the following steps:

[0041] At any moment t The adversarial agent module obtains the current state of the fixed-wing aircraft's intelligent agent. and its control actions Among them, the observation status Includes the current state ;

[0042] Control actions As the initial mixing action:

[0043]

[0044] in, The initial blending action for iterative optimization;

[0045] In the n During the next iteration, based on the mixed actions of the current iteration, the AdvNet neural network is used to predict future... H The near-term returns at each time step are estimated using the following formula:

[0046]

[0047] in, For the estimated time t The near-term returns, representing the returns from time 1 to 2. t arrive The cumulative sum of rewards; These are the parameters of the AdvNet neural network; The length of the time window for near-end reporting; For the reward function, Indicates time Next state Execute action The reward received later; Indicates the first n The mixed actions of the next iteration;

[0048] Based on the estimated near-term reward, the hybrid action is iteratively optimized using gradient descent, as follows:

[0049]

[0050] in, Indicates the first The mixed action of the next iteration This is the gradient descent step size; This represents the gradient of the estimated proximal reward with respect to the mixed action;

[0051] go through N The next iteration yields the final blend action. ;

[0052] From the final mixed action Extract the initial perturbation :

[0053]

[0054] in, For a moment t The initial disturbance;

[0055] Initial disturbance Project to Nodal-constrained sphere, generating perturbations :

[0056]

[0057] in, For a moment t The disturbance constraint parameter represents the disturbance constraint radius; Representing vectors Norm.

[0058] Furthermore, the loss function of the adversarial proxy module is expressed as:

[0059]

[0060] in, To counteract the loss function of the proxy module, Parameters to counter the proxy module; This serves as a rollback buffer, storing the state during training. Mixed actions and a tuple of the corresponding reward data; Expressing expectations; For near-term returns, it means the time step from the current time. t Starting the Future H The cumulative sum of rewards for each time step; This represents AdvNet's prediction of near-end returns.

[0061] Furthermore, the normal agent control strategy includes the parameters of the Actor network and Critic network in the trained Actor-Critic structure.

[0062] Furthermore, during the training process, based on the course learning mechanism, the disturbance constraint parameters of the adversarial agent module are dynamically adjusted according to the completion of the flight mission, specifically including:

[0063] Before training begins, a completion criterion corresponding to each training task in the task set is pre-set. The completion criterion is a binary judgment rule used to determine whether the agent performing the training task has met the minimum completion requirements specified in the task set. The completion criterion includes a trajectory deviation threshold, an attitude error threshold, or an arrival time limit.

[0064] In a parallel simulation environment, for each parallel fixed-wing aircraft agent, the completion status of the agent on the training task of the task set in the current round is evaluated according to the preset completion criteria. If the corresponding completion criteria are met, the task is judged to be completed; otherwise, the task is judged to be failed.

[0065] The disturbance constraint parameters are updated based on the judgment result, using the following formula:

[0066]

[0067] in, , Representing time respectively t The disturbance constraint parameters, Adjust the step size for the preset disturbance constraints;

[0068] In a parallel simulation environment, several agents are divided into groups, and different perturbation limits are set for each group. The updated data is then adjusted based on these perturbation limits. Amplitude limiting is applied.

[0069] Compared with the prior art, the present invention has the following advantages:

[0070] (1) Traditional adversarial reinforcement learning methods often exhibit strong randomness and uneven distribution when generating perturbations, and cannot adaptively adjust the perturbation intensity, which can easily lead to unstable training or overfitting to specific perturbation patterns. Furthermore, existing adversarial reinforcement learning methods typically require training the adversarial agent using reinforcement learning algorithms to learn and generate environmental perturbations. This approach not only incurs high computational costs and slow convergence speed, but also easily falls into training instability problems in high-dimensional control tasks (such as robust control of fixed-wing aircraft), resulting in poor generalization of the final policy. This invention introduces an adversarial agent module based on the AdvNet neural network, directly performing gradient attacks on near-end rewards and generating perturbations through the AdvNet network. This avoids the high computational cost of independent reinforcement learning training for the adversarial agent in traditional methods, thereby significantly improving the efficiency and stability of adversarial perturbation generation, and enhancing the overall training speed and policy convergence quality. This design enables perturbations to more closely resemble the dynamic characteristics of real-world environments, allowing the agent to maintain stable control performance under various perturbation conditions, significantly improving the training convergence speed and policy robustness.

[0071] (2) In adversarial reinforcement learning, improper perturbation intensity settings can lead to unstable training processes: excessively strong perturbations make it difficult for the policy to converge, while excessively weak perturbations fail to improve robustness. This invention achieves adaptive adjustment of perturbation intensity by dynamically adjusting perturbation constraint parameters based on the completion of flight missions using a course-based learning mechanism. When a mission is successful, the perturbation intensity is increased to enhance the challenge; when a mission fails, the perturbation intensity is decreased to consolidate the policy. Furthermore, different perturbation upper limit groups are set in a parallel environment to ensure comprehensive coverage of the perturbation range. This mechanism effectively balances the stability and exploratory nature of training, enabling the policy to gradually transition from simple to complex tasks, significantly improving the convergence efficiency of adversarial reinforcement learning and the generalization ability of the control policy. This mechanism allows for adaptive adjustment of perturbation intensity, ensuring that the training process progresses gradually from easy to difficult, thereby obtaining more evenly distributed and wider-coverage perturbation samples, significantly improving the generalization ability and training stability of the policy.

[0072] (3) Existing reinforcement learning models for fixed-wing aircraft control mostly adopt fully connected neural network structures, which cannot effectively capture the temporal correlation of flight states and have weak perception capabilities for dynamic disturbances and hysteresis characteristics, resulting in lag in the response of control strategies when faced with rapidly changing disturbances. This invention embeds a gated recurrent unit (GRU) module into the Actor-Critic structure, which can fully extract the temporal dynamic features of the aircraft under the action of disturbances, giving the model stronger memory capabilities and temporal dependency modeling capabilities, thereby improving the sensitivity to changes in flight states and the foresight of control actions, enabling the aircraft to maintain more stable attitude control in complex disturbance environments.

[0073] (4) Existing reinforcement learning training for fixed-wing aircraft is usually conducted in a single environment or on a low-concurrency simulation platform, resulting in long training times and insufficient sample diversity, making it difficult to cover various aerodynamic disturbances and environmental uncertainties. This invention uses a GPU parallel simulation platform to construct a nonlinear flight environment and task set, supporting the parallel training and evaluation of hundreds of fixed-wing aircraft agents. This design not only significantly shortens the policy training cycle but also enables the model to perform robust learning under multiple tasks and multiple disturbances simultaneously, thus maintaining a high mission success rate and attitude stability when dealing with complex environments such as wind field changes, sensor noise, engine parameter mismatch, and gusts.

[0074] (5) Existing reinforcement learning methods for fixed-wing aircraft missions often focus on a single optimization index of attitude or trajectory, neglecting the constraints of control input smoothness and stability. This leads to frequent oscillations in flight control actions, affecting the lifespan of actuators and flight safety. In the design of the reward function, this invention comprehensively considers attitude tracking error penalties, attitude stability penalties, control input penalties, and control change penalties. By setting weighting coefficients before training to balance the weights of each part, the agent can maintain stable control input and attitude while optimizing the accuracy of task completion, thereby achieving a robust control effect with high precision and high smoothness. Attached Figure Description

[0075] Figure 1 This is a flowchart of the adversarial reinforcement learning training method according to an embodiment of the present invention;

[0076] Figure 2 This is a schematic diagram of the training process according to an embodiment of the present invention. Detailed Implementation

[0077] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0078] Example 1:

[0079] This embodiment provides an adversarial reinforcement learning training method for robust control of fixed-wing aircraft, such as... Figure 1 As shown, it includes:

[0080] Step S1: Construct a nonlinear simulation environment and task set for the fixed-wing aircraft. The nonlinear simulation environment for the fixed-wing aircraft adopts a six-degree-of-freedom dynamic model, with state variables including at least 12 dimensions such as position, velocity, attitude angle, and angular velocity. Control variables include four types of control inputs: throttle, elevator, aileron, and rudder, covering the complete longitudinal and lateral control space of the aircraft. The evaluation metric is the mission success rate.

[0081] Step S2: Construct several fixed-wing aircraft agents in parallel in a nonlinear simulation environment, and construct an adversarial agent module based on the AdvNet neural network; the adversarial agent module interacts with each agent to provide environmental disturbances;

[0082] In an ideal simulation environment, several fixed-wing aircraft agents are constructed in parallel based on the computing power to train the control strategy of a normal agent. First, an observation and action space is constructed: the observations are denoted as... (May include task-related features such as the deviation between the current state and the target state), actions It covers control variables such as throttle, elevator, aileron, and rudder, and sets the physical boundaries and normalization rules for each channel to match the actuator stroke. The normal agent adopts a PPO-based Actor-Critic structure, where the Actor outputs a continuous action distribution, and the Critic outputs state values. The Actor-Critic structure comprises an Actor network and a Critic network. The Actor network generates control actions by outputting a continuous action distribution. The Critic network estimates the value function of the current state. Both the Actor and Critic networks employ a neural network structure with three fully connected layers, each with 128 nodes. To enhance the perception of temporal perturbations, GRUs are embedded in both the Actor and Critic networks to extract temporal features. The Adam optimizer is typically used for network training. The initialization parameters for the adversarial reinforcement learning algorithm include: a discount factor γ set to 0.99 to ensure the stability of long-term reward calculation; an initial learning rate of 5e-4 for the PPO algorithm to balance convergence speed and training stability; and the Adam optimizer. The initial value of the perturbation constraint δ is set to 0.02 to ensure moderate perturbation intensity in the early stages of training.

[0083] Step S3: Based on the Actor-Critic structure optimized for near-end policies, several fixed-wing aircraft agents and adversarial agent modules are jointly trained through a task set to obtain normal agent control policies, specifically including:

[0084] PPO training is implemented in the following steps. First, sampling and caching: using the old policy parameters. Operating environment, obtain trajectory Value estimation with Critic The returns are defined in the form of discounts and other terms:

[0085]

[0086] in, As a discount factor, Expressing expectations, Indicates time t Counteracting environmental disturbances generated by the proxy module; For the reward function, Indicates at time t state Next action and suffered disturbance The reward given by the environment afterwards;

[0087] Policy Processing (PPO) is a trust region-based approach. Its core idea is to adjust the policy by optimizing the objective function, avoiding excessively large policy updates. The Actor in PPO is optimized using the following objective function:

[0088]

[0089] in, This represents the probability ratio between the current strategy and the old strategy. Let be the optimization objective function of the Actor network; The desired operation to be performed on the sample represents averaging over all trajectory samples; This is a cut operation; For shearing parameters;

[0090] Advantage function It is used to measure the quality of the current action relative to the average level, and is calculated using GAE (Generalized Advantage Estimation):

[0091]

[0092] in, It is a moment TD residuals, It is a discount factor. It is a hyperparameter used to balance bias and variance.

[0093] The Critic network is based on the acquired observation status and returns Predictive value function According to the predicted value function The parameters of the Critic network are optimized using the loss function of the Critic network, as shown in the formula:

[0094]

[0095] in, Let be the loss function of the Critic network. This represents an estimated return, based on the return. Sure; For the Critic network to observe the state Value prediction;

[0096] Based on the ARL algorithm grounded in the PGD method, this invention establishes an adversarial agent. Unlike the classic ARL algorithm, this adversarial agent employs an asymmetric architecture. The optimization objective of the adversary agent is no longer the opposite of that of the normal agent, but rather focuses on attacking the near-reward of the normal agent, defined as:

[0097]

[0098]

[0099] This invention will It is called the approximate reward, which represents the cumulative sum of rewards over the most recent H time steps. Indicating the opponent's actions - Norm. A value greater than or equal to 0 indicates a constraint that causes a disturbance. It is a hybrid action, consisting of the actions of a normal intelligent agent. Disruptions by the opponent composition.

[0100] Compared to the total reward of a round, the estimation of proximal rewards is easier to compute and has smaller errors. Attacking proximal rewards can help the adversary agent directly generate the most effective perturbations under the current conditions. Therefore, this invention abandons the traditional DRL algorithm to train the adversary agent and instead uses the neural network AdvNet to reduce optimization time. AdvNet employs a data-driven approach to fit the algorithm. Its loss function is defined as:

[0101]

[0102] in, To counteract the loss function of the proxy module, Parameters to counter the proxy module; This serves as a rollback buffer, storing the state during training. Mixed actions and a tuple of the corresponding reward data; Expressing expectations; For near-term returns, it means the time step from the current time. t Starting the Future H The cumulative sum of rewards for each time step; This represents AdvNet's prediction of near-end returns.

[0103] After obtaining a reasonable estimate of near-term returns, this invention can apply the PGD method to solve the problem. Specifically, at each time step, the present invention will As the initial hybrid action And randomly sample one The nearby actions are the first optimized hybrid actions. The present invention then uses the gradient descent algorithm to optimize the hybrid action:

[0104]

[0105] in, Indicates the first The mixed action of the next iteration This is the gradient descent step size; This represents the gradient of the estimated proximal reward with respect to the mixed action;

[0106] The gradient can be represented as:

[0107]

[0108] After n iterations, this invention yields a hybrid action that effectively attacks near-end rewards. According to the definition of hybrid action, the present invention can be achieved by... Subtract To easily obtain unconstrained actions from adversary agents .

[0109] Ultimately, the perturbation of the adversary agent By Projected onto a radius of We obtain the following from the sphere:

[0110]

[0111] in, For a moment t The disturbance constraint parameter represents the disturbance constraint radius; Representing vectors Norm; the perturbation generated by the proxy module of this invention Through actions with intelligent agents Superimposed, it enables interaction with the intelligent agent and achieves perturbation.

[0112] In order for the control strategy of a normal intelligent agent to detect the actions of an adversary intelligent agent in a timely manner This invention incorporates a GRU module into the Actor and Critic networks of a normal agent. The temporal feature extraction mechanism of the GRU module can accurately represent the state changes of a fixed-wing aircraft under uncertain disturbances. This enables the normal agent to predict potential disturbances in the current environment and thus take appropriate actions. This helps to cope with these uncertainties. Furthermore, the policy training of normal agents converges faster.

[0113] Step S4: During the training process, based on the course learning mechanism, the disturbance constraint parameters of the adversarial agent module are dynamically adjusted according to the completion of the flight mission;

[0114] During training, the course-based learning method adaptively adjusts the perturbation constraints according to the task completion status, thereby solving two shortcomings of adversarial agents: First, it is difficult to directly train a robust control policy that can cope with extreme environmental perturbations; second, the trained policy is only effective when the environmental perturbation matches the perturbation generated by the adversarial agent, otherwise it may degenerate.

[0115] At each time step, the present invention evaluates the mission performance of each aircraft in the environment, thereby appropriately adjusting the disturbance constraints. :

[0116]

[0117] in, , Representing time respectively t The disturbance constraint parameters, Adjust the step size for the preset disturbance constraints;

[0118] Upon completion of a flight mission, the agent will move on to learning more challenging missions; upon failure, the agent will consolidate previous tasks. To ensure that perturbations generated by the adversary agent are evenly distributed, covering all possible perturbation ranges in the environment, this invention sets five different course upper limits for the aircraft in a parallel environment. These upper limits adjust the perturbation constraints. This setup prevents the control policy from focusing solely on responding to a specific perturbation, thereby improving the policy's generalization and robustness.

[0119] Step S5: After training convergence, the control strategy parameters of the normal agent are used as the control strategy of the fixed-wing aircraft to achieve training of the fixed-wing aircraft.

[0120] The trained control strategy was used to conduct performance evaluations, examining its stability and mission success rate compared to other algorithms under uncertainties including wind field changes, sensor noise, and engine parameter mismatches.

[0121] Example 2:

[0122] like Figure 2 As shown, the specific embodiments of the present invention include the following main steps:

[0123] 1) Construct a nonlinear simulation environment and task set for fixed-wing aircraft, establish an aircraft dynamics model, clarify state variables and control variables, and set task objectives and evaluation indicators.

[0124] 2) Initialize the variables and parameters required for the adversarial reinforcement learning algorithm.

[0125] 3) Initialize the agent in the environment as a normal agent, and use the Actor network and Critic network in the Proximal Policy Optimization (PPO) algorithm to generate control actions and perform state value evaluation for the agent, respectively.

[0126] 4) Establish an adversarial agent, with the optimization objective being to generate effective perturbations against normal agent actions by using the near-end reward fitted through the AdvNet network.

[0127] 5) Perform adversarial reinforcement learning training. During the training process, the perturbation constraints are adaptively adjusted based on the course learning method.

[0128] 6) Use the trained control strategy to evaluate its performance. Under multiple uncertainties, including wind field changes, sensor noise, and engine parameter mismatch, test the stability and task success rate of the strategy compared with other algorithms.

[0129] The following embodiments of the present invention are based on the following tasks: the implementation of heading control task and the implementation of target tracking control task.

[0130] The implementation of heading control includes the following steps:

[0131] 1) Construct a simulation model of a fixed-wing aircraft: Establish a six-degree-of-freedom dynamic model of a fixed-wing aircraft on the simulation platform. The state variables include position, velocity, attitude angle and angular velocity, and the control variables are throttle, elevator, aileron and rudder.

[0132] 2) Mission Setting: The heading mission requires the aircraft to yaw, turn, and climb while maintaining a given airspeed; for example, yaw changes by 120 degrees, climbs 300 meters, and maintains cruise speed. During the training period, start in an environment without additional uncertain disturbances.

[0133] 3) Initialize the adversarial reinforcement learning (C-RPARL) and PPO hyperparameters: Set the discount factor to 0.99 and the learning rate to 5e-4.

[0134] 4) Initialize the network structure: The Actor and Critic networks each contain three fully connected layers, each with 128 units and embedding a GRU module to extract temporal features. In addition, a network is built to fit the proximal reward.

[0135] 5) Initialize normal agent and initial state: Set the initial position, speed and attitude of the aircraft.

[0136] 6) Adversarial generator establishment: Perturbations are generated using the PGD method, and optimization is performed to make the perturbations reduce the rewards of normal agents.

[0137] 7) Disturbance generation and optimization: In each iteration, a disturbance is generated near the current action, and several optimization steps are performed to ensure that the disturbance meets the constraints.

[0138] 8) Forming hybrid actions and interacting with the environment: The generated disturbances are added to the control actions of the normal agent, interact with the environment, and record feedback.

[0139] 9) Curriculum-based adaptive perturbation constraints: Adjust perturbation constraints based on task completion. Increase constraints when a task succeeds, and decrease them when a task fails.

[0140] 10) Upper limit of perturbation constraint: The initial perturbation constraint is 0.02, and the upper limit of courses for all parallel entities is uniformly set to 0.05.

[0141] 11) Trigger Update: When the replay pool is full, update the Actor, Critic, and the network that fits the near-end reward simultaneously.

[0142] 12) Training round control: Determine whether to continue training based on the number of weight updates.

[0143] 13) Evaluation and Analysis: The success rate of the task was evaluated under different wind speed conditions, and the performance of different algorithms was compared. C-RPARL showed better robustness under most wind speed conditions.

[0144] The target tracking and control task includes the following steps:

[0145] 1) Task construction: Randomly generate the target direction in the simulation environment, and the target point is located 500 meters away in that direction.

[0146] 2) Target control: Reach the target point within a specified time using four-channel control (throttle, elevator, aileron, rudder).

[0147] 3) Initialize hyperparameters: Set the discount factor to 0.99 and the learning rate to 5e-4.

[0148] 4) Initialize the network structure: The Actor and Critic network structures are the same as in Example 1, including the GRU module. Establish a network for fitting proximal rewards.

[0149] 5) Initialize normal agent and initial state: Set the initial position, speed, attitude and target position of the aircraft.

[0150] 6) Antibody Generator: Perturbs antibody generation to reduce the reward of normal agents.

[0151] 7) Disturbance generation and update: Disturbances are generated using the PGD method, and the disturbances are ensured to meet the constraints.

[0152] 8) Normal agent interaction with the environment: Execute mixed control actions, interact with the environment, and record status and rewards.

[0153] 9) Curriculum-based perturbation constraint adjustment: Adjust perturbation constraints based on task success and failure.

[0154] 10) Parallel course upper limit grouping: Set multiple values ​​for the course upper limit of parallel agents to cover perturbations of different intensities.

[0155] 11) Trigger and Update: When the replay pool is full, update the Actor, Critic, and the network that fits the near-end reward.

[0156] 12) Robustness assessment: Test the success rate of target tracking tasks under multiple types of uncertain perturbations.

[0157] 13) Analysis and optimization: Comparing the task success rates of different algorithms, C-RPARL shows more stability under various perturbation conditions.

[0158] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for adversarial reinforcement learning training for robust control of fixed-wing aircraft, characterized in that, The application relates to a method for training a fixed-wing aircraft, and belongs to the technical field of artificial intelligence. The method comprises the following steps: Constructing a nonlinear simulation environment and a task set for a fixed-wing aircraft; Parallelly constructing a plurality of fixed-wing aircraft agents in the nonlinear simulation environment, and constructing an adversarial agent module based on a neural network AdvNet; the adversarial agent module interacts with each agent to provide environmental disturbance; Based on an actor-critic structure of proximal policy optimization, the plurality of fixed-wing aircraft agents and the adversarial agent module are jointly trained through the task set to obtain a normal agent control strategy; During the training process, the disturbance constraint parameters of the adversarial agent module are dynamically adjusted according to the completion of the flight task based on a curriculum learning mechanism; During the agent's task set, the adversarial agent module adjusts according to the current state. and the control actions output by the intelligent agent Generate disturbance Specifically, it includes the following steps: At time t , the counter-agent module obtains the current state of the fixed-wing aircraft agent and its control action , wherein the observation state contains the current state ; , the control action is selected from the action space at time t ; , the observation state corresponds to the observation space at time t ; Control actions will be As initial mixing action: wherein, is the initial mixture of actions for the iterative optimization; In the first n iteration process, the neural network AdvNet is used to estimate the proximal return of the future H time step according to the mixed action of the current iteration, and the formula is: wherein, is an estimated time t of a proximal return, representing the cumulative sum of rewards from time t to ; are parameters of a neural network AdvNet; is a length of a time window of proximal returns; is a reward function, represents a state at time after performing an action and obtaining a reward; represents a mixed action at the n th iteration; After the training converges, the control strategy parameters of the normal agent are used as the control strategy of the fixed-wing aircraft, and the training of the fixed-wing aircraft is realized; wherein, denotes the mixed action of the is the gradient descent step size; denotes the estimated gradient of the proximal return with respect to the mixed action; After N sub-iterations, obtain final mixed action ; extracting initial perturbations from final mixed action :​ wherein is the initial disturbance at time t t = 0; initial perturbation projected to norm-constrained sphere, generating perturbation : in, For a moment t The disturbance constraint parameter represents the disturbance constraint radius; Representing vectors Norm.

2. The method of claim 1, wherein, According to the estimated proximal return, the hybrid action is iteratively optimized through gradient descent, and the formula is: The plurality of fixed-wing aircraft agents are parallelly constructed in the nonlinear simulation environment, and the construction specifically comprises the following steps: An independent observation space and an action space are allocated to each fixed-wing aircraft agent in the nonlinear simulation environment; 3. The method of claim 2, wherein the method is a method of robust control of a fixed-wing aircraft using adversarial reinforcement learning training. An actor-critic structure based on proximal policy optimization is initialized for each fixed-wing aircraft agent.

4. The method of claim 1, wherein, The observation space comprises a current state and task characteristics, the current state comprises state variables of the aircraft, including target distance information, speed, attitude angle and angular velocity; the task characteristics are the deviation of the current state from a target state; the action space comprises control inputs of a throttle, an elevator, an aileron and a rudder, and each control input has physical boundaries and normalization rules.

5. The method of claim 1, wherein, The actor-critic structure comprises an actor network and a critic network; the actor network generates a control action through output of a continuous action distribution; the critic network is used for estimating a value function of the current state; the actor network and the critic network adopt a neural network structure with three fully connected layers, and each layer has 128 nodes; and a gated recurrent unit module is embedded in the actor network and the critic network. In the nonlinear simulation environment, the strategy parameters of the current fixed-wing aircraft agent are adopted The task set is executed, and trajectory data of the fixed-wing aircraft agent is generated through interaction with the counteracting agent module , wherein is the time t , the observation state corresponding to the observation space, is the time t , the control action of the selected action space; is the time t , the reward value of the time is the time , the observation state corresponding to the observation space; is the length of the trajectory, indicating the number of time steps from the initial state to the terminal state; is the current strategy parameter, indicating the strategy parameter of the agent at the current time, including the parameters of the Actor and Critic networks; According to the obtained trajectory data, the return is calculated in the form of a discount sum , the formula being: wherein, is a discount factor, denotes expectation, denotes time instant t environmental perturbation generated by the adversarial agent module; is a reward function, denotes at time instant t state action is performed and suffers a perturbation reward given by the environment after, Utilizing the return and advantage function The parameter of the Actor network is optimized by the Proximal Policy Optimization algorithm, and the optimization objective function is: wherein, is a probability ratio of the current policy and the old policy; is an optimization objective function of the Actor network; is an expected operation on a sample, representing an average over all trajectory samples; is a clipping operation; is a clipping parameter; The Critic network is based on the acquired observation status. and returns Predictive value function According to the predicted value function The parameters of the Critic network are optimized using the loss function of the Critic network, as shown in the formula: wherein, is the loss function for the Critic network, represents an estimate of the return, according to the return determined; is the value prediction of the Critic network for the observed state s. The actor-critic structure based on proximal policy optimization is jointly trained through the task set according to the plurality of fixed-wing aircraft agents and the adversarial agent module, and the training specifically comprises the following steps:

6. The method of claim 5, wherein the method is a method of robust control-oriented adversarial reinforcement learning training for fixed-wing aircraft. In each training step, the network parameters of the actor network and the critic network are updated by optimizing the objective functions and of the actor network and the critic network. The reward function is obtained according to the current state of the fixed-wing aircraft, a target state in the task set, a control action and adversarial disturbance, and comprises the following parts: An attitude tracking error penalty is used for measuring the deviation between the current attitude of the aircraft and a target attitude in a task, and is derived from the observation state and the target state in the task set; An attitude stability penalty is used for restraining the attitude angle change rate of the aircraft, and is derived from the state change in a continuous time step; A control input penalty is used for limiting the control amplitude of the throttle, the elevator, the aileron and the rudder, and is derived from the control action output by the actor network. The control change penalty is used to constrain the change of the continuous time step control action, and is derived from the difference between the control action of the current time step and the control action of the previous time step.

7. The method of claim 1, wherein, The loss function of the adversarial agent module is represented as: wherein, a loss function for the adversarial agent module, parameters of the adversarial agent module; a rollback buffer storing states during the training process , a mixed action and a tuple of corresponding reward data; denotes expectation; a n-step return, denoting the cumulative sum of rewards from the current time step t for the next H time steps; denotes the predicted value of the n-step return by AdvNet. 8.The method of claim 1, wherein, The normal agent control strategy comprises parameters of an Actor network and a Critic network in a trained Actor-Critic structure. 9.The method of claim 1, wherein, During the training process, the disturbance constraint parameter of the adversarial agent module is dynamically adjusted based on a curriculum learning mechanism according to the completion of the flight task, and specifically comprises: Before the training starts, a completion criterion corresponding to each training task in the task set is preset, the completion criterion is a binary judgment rule for judging whether the agent executing the training task meets the minimum completion requirement of the task set, and the completion criterion comprises a trajectory deviation threshold, an attitude error threshold or a time-to-reach limit; In a parallel simulation environment, for each parallel running fixed-wing aircraft agent, the completion of the training task in the task set by the agent in the current round is evaluated according to the preset completion criterion, and if the corresponding completion criterion is met, the task is determined to be completed, otherwise the task is determined to be failed; The disturbance constraint parameter is updated according to the determination result, and the formula is: wherein, , respectively represent the disturbance constraint parameters at time t , and is a preset disturbance constraint adjustment step length. The several agents in the parallel simulation environment are divided into several groups, different disturbance upper limits are set for each group, and the updated amplitude limiting processing is performed according to the disturbance upper limit.

Images