Rotorcraft snake maneuvering path control system and method based on asymmetric deep reinforcement learning

Abstract

The application discloses a rotorcraft serpentine maneuvering track control system and method based on asymmetric deep reinforcement learning, and belongs to the field of intelligent flight control. The application includes building a hierarchical control architecture, fusing current and prospective lateral track error to construct a Markov model, outputting a target roll angle in the outer loop, and solving physical control variables by a bottom PID controller. The application designs an asymmetric network structure, a strategy network receives observation data containing Gaussian noise to make decisions, two evaluation networks evaluate the value of actions based on real states, an asymmetric TD3 algorithm is used for training, and a composite reward function is set to optimize control effect. The application does not need to rely on an accurate dynamic model, effectively resists sensor interference, reduces the gap between virtual and real migration, has high track tracking precision and stable control, can inhibit maneuvering overshoot, and is suitable for rotorcraft low-altitude reconnaissance, special maneuvering and other scenes.

Description

Technical Field

[0001] This invention relates to a control system and method for a rotorcraft serpentine maneuver trajectory based on asymmetric deep reinforcement learning, belonging to the field of robot intelligent control. Background Technology

[0002] As an aircraft with unique flight capabilities such as vertical takeoff and landing, hovering, lateral flight, and inverted flight, the rotorcraft plays an irreplaceable role in fields such as cargo transport, search and rescue, low-altitude reconnaissance, and special operations. In these complex missions, rotorcraft often need to perform slalom maneuvers. This is a highly challenging dynamic flight mission that requires the rotorcraft to maintain a specific altitude and forward speed while continuously and rapidly performing high-G lateral rolls to traverse a series of staggered thresholds. However, due to the inherent characteristics of rotorcraft—including roll, pitch, yaw, and strong coupling of multiple takeoff and landing channels, underactuation, high nonlinearity, and susceptibility to complex aerodynamic interferences such as ground effect and rotor downwash—track tracking control for slalom maneuvers remains an extremely challenging problem in the field of flight control.

[0003] Currently, existing technologies for tracking and controlling complex flight paths of rotorcraft can be mainly divided into two categories. The first category is traditional control methods based on precise mathematical models, such as PID control, linear quadratic regulators (LQR), or sliding mode control. The core limitation of this type of method lies in its over-reliance on precise aerodynamic and dynamic models of the rotorcraft. In high-dynamic flight such as serpentine maneuvers, obtaining high-precision mathematical models is extremely difficult and costly. At the same time, classical controllers can usually only achieve optimal performance near a specific trim point. Faced with continuous maneuvers with large ranges and high overloads, controllers with fixed parameters struggle to adapt to dynamically changing environments, easily leading to increased tracking errors or even system instability.

[0004] The second category is the intelligent control method based on deep reinforcement learning (DRL), which has emerged in recent years. Although this method eliminates the dependence on precise dynamic models, it still faces many obstacles in practical applications. First, conventional end-to-end reinforcement learning directly maps the state to the underlying servo actions (such as the various pitch control variables of the rotor). This not only leads to an exponential explosion in the action exploration space and makes the network extremely difficult to converge, but also the direct output of high-frequency control commands by the black-box network can easily cause control surface chattering, seriously threatening the structure and flight safety of the rotorcraft. Second, existing reinforcement learning algorithms usually assume that the agent is in an ideal, noise-free "God's-eye view" during training. However, in the real physical world, airborne sensors inevitably have Gaussian noise or drift. This difference between ideal and reality leads to a serious "sim-to-real gap" when the strategy trained in simulation is deployed in actual use, resulting in extremely poor anti-interference capabilities. In addition, existing intelligent tracking strategies often only focus on the current lateral tracking error and lack the ability to predict the intersecting curves ahead, which makes it easy for rotorcraft to overshoot the trajectory when cornering at high speed and makes it difficult to establish an ideal roll attitude in advance.

[0005] In view of this, there is an urgent need in the field for an intelligent maneuver control method that can not only get rid of the dependence on the precise dynamic model of the rotorcraft, but also effectively bridge the gap between virtual and real migration, overcome sensor noise interference, and have high-level safety assurance and flight path prediction capabilities, so as to achieve accurate, agile and smooth tracking of complex serpentine flight paths. Summary of the Invention

[0006] To address the shortcomings of the existing technologies, the present invention aims to provide a control system and method for a rotorcraft serpentine maneuver trajectory based on asymmetric deep reinforcement learning. This system achieves an accurate dynamic model independent of the rotorcraft, outputs the target roll angle command through the reinforcement learning outer loop, executes high-frequency servo control through the underlying PID inner loop, and introduces an asymmetric state observation mechanism during the training phase (the policy network inputs noisy states, and the evaluation network inputs real states). Ultimately, this system achieves a rotorcraft serpentine slalom trajectory tracking target with noise resistance and robustness.

[0007] Technical solution: A method for controlling the serpentine maneuver trajectory of a rotorcraft based on asymmetric deep reinforcement learning, comprising the following steps: A Markov model and hierarchical control structure are constructed for tracking the serpentine maneuver path of a rotorcraft. The environmental state is defined to include real state variables and noisy observation variables. The state variables of the Markov model include the current lateral trajectory tracking error and the look-ahead lateral trajectory tracking error of the predicted point ahead of the rotorcraft. The control variable output by the Markov model is the target roll angle of the rotorcraft. The target roll angle is input to the bottom-level controller to calculate the physical control variables of the rotorcraft. An asymmetric deep reinforcement learning-based policy network and dual evaluation network are constructed. The input of the policy network is the noisy observation variable, and the output is the control action, which is the target roll angle. The input of the dual evaluation network is the real state variable and the control action, and the output is the action value assessment. The policy network and the dual evaluation network are trained using an asymmetric reinforcement learning method. During training, the dual evaluation network is used to calculate the target value based on the real state variables of the next time step to update the dual evaluation network, and the gradient of the dual evaluation network is calculated using the real state variables to guide the policy network, which generates actions based on the noisy observation variables, to be updated. After training, the trained strategy network is used as the final rotorcraft serpentine maneuver control strategy network. In actual flight control, the flight state with sensor noise is input into the strategy network, and the target roll angle control underlying system is output to achieve serpentine flight path tracking of the rotorcraft.

[0008] Furthermore, the method for obtaining the forward lateral trajectory tracking error includes: calculating the relative positional relationship between the current position of the rotorcraft and the estimated point based on the estimated point at a preset distance in front of the current position of the rotorcraft, and obtaining the forward lateral trajectory tracking error.

[0009] Furthermore, the target roll angle is input to the underlying controller to calculate the physical control quantities of the rotorcraft, including: inputting the target roll angle together with the target speed and altitude into the underlying speed loop, attitude loop and angular rate loop PID controller, and finally calculating the lateral cyclic pitch, longitudinal cyclic pitch, yaw pedal and collective pitch control quantities of the rotorcraft.

[0010] Furthermore, the construction of the asymmetric deep reinforcement learning-based policy network and dual evaluation network includes: constructing a policy network based on noisy observations, wherein the input of the policy network is the noisy observation variable and the output is the target roll angle; and constructing a dual evaluation network based on the real state, wherein the dual evaluation network comprises two networks with the same structure but different parameters, wherein the input is the real state variable and the control action, and the output is the action value evaluation quantity.

[0011] Furthermore, the step of training the policy network and the dual-evaluation network using an asymmetric reinforcement learning method includes: Using a target policy network, smooth target actions are generated based on noisy observation variables at the next time step; Using a target dual-evaluation network, the target Q value is calculated based on the real state variables at the next time step, and the parameters of the dual-evaluation network are updated by gradient descent based on the mean square error between the Q value evaluated by the dual-evaluation network and the target Q value. When the delayed update frequency is reached, the gradient of the dual evaluation network is calculated using the real state variables to guide the update of the policy network that generates actions based on noisy observation variables. The weights of the target evaluation network and the target policy network are updated according to the target soft update tracking ratio.

[0012] Furthermore, the noisy observation variable is generated by injecting Gaussian sensor noise into the real state variable to obtain the noisy observation variable.

[0013] Furthermore, the one-step reward function of the Markov model includes: a track-keeping reward based on the current lateral trajectory tracking error, a heading alignment reward, a forward propulsion reward, a forward velocity error penalty, and a smoothing penalty for changes in adjacent control actions.

[0014] Furthermore, the real state variables include: the lateral deviation distance, horizontal speed, and forward speed error of the rotorcraft in the inertial coordinate system; the three-axis speed, roll angle, pitch angle, and three-axis angular velocity in the body coordinate system; the heading error between the current heading and the tangent direction of the target route; and the action input of the previous control cycle.

[0015] On the other hand, the present invention also provides a rotorcraft serpentine maneuver trajectory control system based on asymmetric deep reinforcement learning, comprising: The model building module is used to construct a Markov model and hierarchical control structure for tracking the serpentine maneuvering flight path of a rotorcraft. The environmental state is defined by real-state variables and noisy observation variables. The state variables of the Markov model include the current lateral trajectory tracking error and the look-ahead lateral trajectory tracking error at the predicted point ahead of the rotorcraft. The control variable output by the Markov model is the target roll angle of the rotorcraft, which is input to the lower-level controller to calculate the physical control variables of the rotorcraft. A network construction module is used to construct an asymmetric deep reinforcement learning-based policy network and a dual evaluation network. The input of the policy network is the noisy observation variable, and the output is the control action, which is the target roll angle. The input of the dual evaluation network is the real state variable and the control action, and the output is the action value assessment. The training module is used to train the policy network and the dual evaluation network using an asymmetric reinforcement learning method. During the training process, the dual evaluation network is used to calculate the target value based on the real state variables of the next time step to update the dual evaluation network, and the gradient of the dual evaluation network is calculated using the real state variables to guide the policy network, which generates actions based on the noisy observation variables, to be updated. The control execution module is used to take the trained strategy network as the final rotorcraft serpentine maneuver control strategy network after training is completed. In actual flight control, the flight state with sensor noise is input into the strategy network and the target roll angle control underlying system is output to realize the tracking of the rotorcraft's serpentine flight path.

[0016] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0017] Beneficial effects: Compared with the prior art, the solution provided by the present invention has the following three substantial features and significant progress: 1. This invention successfully bridges the "Sim-to-Real" gap between simulation and reality. Traditional reinforcement learning algorithms struggle to handle noise from real-world sensors. This invention innovatively employs asymmetric information flow during the training phase: the Critic network operates from a "God's-eye view," utilizing ideal, noise-free real-world conditions for accurate action value assessment; while the Actor network relies solely on noisy observations injected with Gaussian noise to make decisions. This mechanism forces the Actor network to learn to make the most robust responses under uncertain and perturbed information, enabling the trained model to be directly deployed on physical rotorcraft with real-world measurement noise, exhibiting extremely strong noise resistance.

[0018] 2. The method of this invention adopts a hierarchical control architecture, which significantly reduces the exploration dimension and ensures the flight safety of the underlying layers. This invention abandons the inefficient approach of directly controlling the four underlying channel control surfaces using conventional end-to-end reinforcement learning, and decouples the system into a "reinforcement learning guidance outer loop" and a "traditional PID steady-state inner loop". The outer loop agent only needs to output a single-dimensional macroscopic command (target roll angle), which greatly reduces the action exploration space and accelerates network convergence; the complex nonlinear high-frequency control surface mapping is handled by the PID algorithm, avoiding control surface chattering caused by the direct output of oscillating signals by the black-box network, thus ensuring the structural safety and system stability of the rotorcraft during high-G maneuvering flight at the physical hardware level.

[0019] 3. This invention integrates look-ahead error and motion smoothing constraints to achieve highly agile and smooth complex trajectory tracking. The invention not only includes the rotorcraft's current lateral tracking error in the state space but also creatively introduces a predicted look-ahead error for a specific distance ahead of the flight path (e.g., 20 meters, 50 meters), endowing the agent with "foresight" and enabling it to establish roll bank before entering a curve, much like a human pilot. Simultaneously, a smoothing penalty term is introduced into the composite reward function for changes in adjacent control commands, effectively suppressing abrupt changes in the control signal and ensuring extremely high trajectory tracking accuracy and agility for the rotorcraft during continuous high-G slalom maneuvers. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the six degrees of freedom of the aircraft involved in the method of the present invention; Figure 2 This is a schematic diagram of the strategy network in the method of the present invention; Figure 3 This is a schematic diagram of the evaluation network in the method of the present invention; Figure 4 This is a schematic diagram of the deep reinforcement learning architecture in the method of the present invention; Figure 5 This is a schematic diagram illustrating the change of the cumulative reward function in an embodiment of the method of the present invention; Figure 6 This is a schematic diagram of the aircraft altitude in an embodiment of the present invention; Figure 7 This is a schematic diagram of the vacuum velocity of the aircraft in an embodiment of the present invention; Figure 8 This is a schematic diagram of the roll angle of the aircraft in an embodiment of the present invention; Figure 9 This is a schematic diagram of the yaw angle of the aircraft in an embodiment of the present invention; Figure 10 This is a schematic diagram of the aircraft pitch angle in an embodiment of the present invention; Figure 11 This is a schematic diagram of the sideslip angle of the aircraft in an embodiment of the present invention; Figure 12 This is a schematic diagram of the XY plane trajectory of the aircraft in an embodiment of the present invention. Detailed Implementation

[0021] To illustrate the technical solution provided by this invention in detail, a specific description is given below with reference to the accompanying drawings.

[0022] This invention provides a rotorcraft serpentine maneuver trajectory control method based on asymmetric deep reinforcement learning, combined with... Figure 1-12 As shown, the implementation process is as follows: (1) Construct a hierarchical control architecture and Markov model for tracking the serpentine maneuver path of a rotorcraft, including the following steps: (1-1) Determine the state variables of the Markov model Combination Figure 1 As shown, due to sensor measurement noise in actual flight of the rotorcraft, the environmental state is set to include real state variables. With noisy observed variables ;use This indicates the lateral deviation of the rotorcraft in the inertial coordinate system. This represents the horizontal velocity of the Earth's axis in an inertial frame of reference. Indicates forward velocity error. This represents the three-axis velocities of the rotorcraft in its body coordinate system. Indicates the roll and pitch angles of a rotorcraft. This indicates the relative heading error between the rotorcraft's current heading and the tangential direction of the target flight path. This represents the three-axis angular velocity in the body coordinate system. This indicates the current lateral trajectory tracking error. These represent the forward lateral trajectory tracking error at a preset distance in front of the rotorcraft's current position. This indicates the action input from the previous control cycle; The true state variables of the rotorcraft Markov model for:

[0023] Inject sensor noise that follows a Gaussian distribution into the real state variables Noisy observation variables were obtained. :

[0024] Among them, superscript This indicates the matrix transpose.

[0025] (1-2) Determine the control variables for the Markov model and hierarchical control structure

[0026] Define the control variables for the outer-loop reinforcement learning Markov model. The target roll angle for the rotorcraft ; The underlying physical control variable of the rotorcraft is set to the lateral cyclic pitch control variable. Longitudinal periodic pitch control rudder inputs of the yaw pedal And collective pitch rudder Target roll angle After being solved by the underlying speed loop, attitude loop and angular rate loop PID controller, it is transformed into the above four underlying physical control variables; (1-3) Determine the transition model of the Markov model, expressed as:

[0027] in, At the state sampling moment, the transition model is a time-discrete black-box function that integrates the underlying PID controller and the rotorcraft's flight dynamics, representing the action performed at the current sampling moment. The state transition relationship at the next sampling time; (1-4) Determine the one-step reward function of the Markov model, expressed as:

[0028] in, To maintain the track based on the current lateral error, For the course alignment bonus item, For propulsion rewards related to forward speed, Forward velocity error penalty term, As punishment The control action smoothing penalty term; the one-step reward function is used to comprehensively evaluate the flight tracking accuracy and control stability after executing the control variable; (1-5) The objective function for rotorcraft flight path tracking is determined as follows:

[0029] satisfy:

[0030] in, Represents control sequence , This represents the weight decay factor for future rewards. , Represents the mathematical expectation; (2) Construct an evaluation network and a policy network based on asymmetric deep reinforcement learning, and combine them... Figure 2 As shown, it includes the following steps: (2-1) Construct a policy network Combination Figure 2 As shown, the input to the policy network is noisy observation variables. The output of the policy network is the action of the control variable. The policy network expression is as follows:

[0031] in, These are the weight parameters of the policy network; (2-2) Constructing a dual evaluation network Combination Figure 3 As shown, the evaluation network consists of two networks with the same structure but different parameters. and The input to the evaluation network is the real state variable. and control variables The output of the evaluation network is the cumulative value assessment. The expression for evaluating the network is as follows:

[0032] in, , representing the first evaluation network and the second evaluation network; (3) The asymmetric dual-delay depth deterministic strategy gradient (TD3) method is used to control the rotorcraft trajectory tracking, combined with... Figure 4 As shown, the specific steps are as follows: (3-1) Set the maximum number of training steps to The maximum number of experimental steps in each iteration is Set a training set size of The tracking ratio of the target neural network is set to... Set the policy network latency update frequency to The learning rates of the evaluation network and the policy network are set as follows: and Set a weight decay factor for future losses. ; (3-2) Set the weight parameters of the policy network in step (2-1) as follows: In step (2-2), the weight parameters of the first evaluation network and the second evaluation network are set as follows: Construct a target policy network based on the current network weights. and the first objective evaluation network Second objective evaluation network ; Establish an empty experience set And set its maximum capacity; (3-3) Training begins. Initialize the rotorcraft environment and obtain the initial real state variables. Compared with the initial noisy observed variables ; (3-4) Using the current policy network, generate control actions based on the current noisy observations. And add truncated normal distribution exploration noise, combined with Figure 5 As shown, control actions The input is fed into the hierarchical Markov model of step (1) to obtain the true state for the next step. Noisy observations and one-step reward function ; (3-5) Record the state transition information as an asymmetric empirical sample. And store the sample in the experience set. middle; (3-6) Count the number of samples in the empirical set. If the number of samples reaches the size of the training set... Then from the experience set Randomly selected from The network is updated using a sample of experience. (3-7) Generate smooth target control actions according to the following formula:

[0033] in To smooth out noise for the strategy; (3-8) Update the weights of the first evaluation network and the second evaluation network according to the following formula:

[0034] (3-9) Make the policy network lag time count ,like right If the remainder is zero, then the asymmetric deterministic policy gradient method is used, which updates the weights of the current policy network using the gradient provided by the first evaluation network based on the real state.

[0035] (3-10) Update the weights of the target evaluation network and the target policy network according to the soft update formula:

[0036] (3-11) Repeat steps (3-4) to (3-10) until the maximum number of training steps is reached, and then update the current policy network. As the final deployment strategy network, in actual aircraft control, the flight state with sensor noise is directly input into the final strategy network, and the network outputs the target roll angle to control the underlying system, realizing the serpentine flight path tracking of the rotorcraft based on deep reinforcement learning.

[0037] Example: The control architecture in this example consists of an upper-level "asymmetric TD3 reinforcement learning outer loop" and a lower-level "classical PID control inner loop" connected in series. The outer loop operates at a lower frequency (e.g., 0.1s control cycle) to handle macroscopic nonlinear trajectory planning and attitude decision-making; the inner loop operates at a higher frequency (e.g., 0.01s control cycle) to solve the underlying servo actions.

[0038] Combination Figure 6-12 As shown, the specific implementation steps are as follows: S1. Constructing a snake-like maneuvering mission scenario and a Markov decision process model. In the simulation environment, the takeoff parameters and serpentine maneuver reference path of the rotorcraft are set. To improve the generalization ability of the strategy, the level flight distance for the rotorcraft to enter the path is set at... Randomly distributed within a range, with an initial velocity of to Choose from between, and set the target desired speed to be the current initial speed increased by 1 / 2. .

[0039] The flight path is generated using a cubic spline curve. Based on the rotorcraft's entry point, the paths are positioned forward... Control points are set at [location], and the horizontal coordinates (Y-axis) of the corresponding control points are alternately set to [value]. and Furthermore, derivative boundary conditions are set at the beginning and end to make the inbound and outbound routes smooth.

[0040] State variables (real state and noisy observation): The various physical quantities of the rotorcraft in the inertial coordinate system and the body coordinate system constitute the basic state.

[0041] Real State Set It contains 17 dimensions:

[0042] in, This represents the current lateral deviation; This is the velocity of the Earth's axis; This represents the error between the forward velocity of the aircraft and the velocity of the target. The three-axis velocity of the machine body; For roll and pitch angles; This is the heading error (the angle between the current heading and the tangent of the ideal route). The angular velocity of the three axes of the machine body; This represents the lateral error between the current point and the ideal spline curve. The current X-coordinate of the rotorcraft is in front. and The estimated lateral tracking error at the location; This refers to the actions performed in the previous cycle.

[0043] Due to sensor errors in actual flight, Inject physical noise that follows a normal distribution into each dimension Generate a noisy observation set In this embodiment, the noise standard deviation of a specific dimension... Set as: Position and velocity error term attitude angle angular velocity The action itself is noiseless.

[0044] Determine the control variables: The action space of the Markov model is one-dimensional, that is, the target roll angle of the output rotorcraft. To ensure flight safety, its maneuvering space is strictly limited. Within the range.

[0045] One-step reward function: One-step reward function It is defined as the sum of five parts, used to comprehensively evaluate the performance of a rotorcraft: The first item is the reward for lateral trajectory tracking: The Gaussian kernel function is used to maximize the reward for zero error; The second item is the heading alignment bonus: ; The third item is the forward progress reward: This encourages rotorcraft to maintain kinetic energy above the threshold; The fourth item is the speed error penalty: ; The fifth item is a penalty for smooth movement: It is used to punish high-frequency jitter.

[0046] S2: Establish a hierarchical PID controller at the bottom layer.

[0047] The target roll angle output by the reinforcement learning agent Instead of acting directly on the rotor, it enters the underlying three-stage cascaded PID controller: Velocity loop: Receives the target's forward velocity, vertical altitude, and set lateral velocity from reinforcement learning output, subtracts them from the current feedback, and outputs the desired pitch angle command. yaw rate command and vertical speed command .

[0048] Attitude loop: Receives the target roll angle from reinforcement learning output. The desired pitch angle output by the speed loop After PID calculation, the desired roll and pitch rates of the aircraft are output.

[0049] Angular Rate Loop: Receives the desired three-axis angular rate and vertical velocity from the attitude loop and velocity loop, combines them with the current real feedback, and calculates using PID and derivative filtering to output the final physical control quantity: lateral periodic pitch ( ), longitudinal periodic pitch ( ), yaw pedals ( ) and total distance ( The underlying control cycle is... In each reinforcement learning decision step ( The process is repeated 10 times in a continuous loop to maintain a stable posture.

[0050] Constructing an Asymmetric Dual-Delay Deep Deterministic Policy Gradient Network (Asymmetric TD3)

[0051] By combining the concept of asymmetric training, we construct the Actor network and the Double Critic network.

[0052] Policy Network (Actor): Employs a 3-layer fully connected neural network. The input layer has a dimension of 17 (receiving...). The features are extracted through the hidden layer (e.g., 256 neurons, using the Tanh activation function), and finally the output layer is mapped to the target layer using the Tanh function. And multiply by the maximum range of motion ( Output Action .

[0053] Double Critic Network: To avoid overestimation of Q-values, two identical Critic networks are constructed. and Each network employs a 3-layer fully connected neural network, with its input being a concatenated vector of state and action variables. It is important to note that the Critic network has a 17-dimensional input state and specifically receives noise-free real-world states during the training phase. The hidden layers use the ReLU activation function, and the output layer uses a linear activation function to output the expected Q value.

[0054] S4: Offline Training Process for Asymmetric Reinforcement Learning

[0055] Initialization parameters: Set the maximum number of training steps to... Step, Batch Size is 256, and the decay factor for future loss weights. Target network soft update coefficient Establish a capacity of A collection of experience replays.

[0056] Data Acquisition: In each environmental interaction step, the Actor network is based on the current noisy observations. Generate target rolling motion To increase the exploratory nature of the movement, the standard deviation of the movement amplitude was added to the movement. The noise is explored using a normal distribution, and boundary clipping is performed. After the environment performs an action, data is collected. Store in the playback pool.

[0057] Evaluate network updates: Randomly sample batches of data from the replay pool and calculate the target action:

[0058] in To smooth out the noise in the target strategy.

[0059] Calculate the target Q value (from a God's-eye view):

[0060] Update based on mean squared error loss function (MSE Loss) and Network parameters:

[0061] Policy network delayed update: Every certain number of steps (e.g., Delay Freq = 1 or 2), the Actor network is updated based on the gradient direction provided by the real state, enabling it to learn to make optimal actions in noisy states.

[0062] The target network parameters are then soft-updated.

[0063] S5: Real-world / High-fidelity simulation deployment

[0064] After training, the Critic network is discarded. In actual deployment or testing, the rotorcraft flight control system directly uses the trained Actor network. At this point, the Actor network relies solely on real-time state feedback from onboard sensors, which inevitably contains measurement noise (equivalent to...). This allows for a stable output of the desired roll angle. The final attitude response is achieved by the inner-loop PID controller of the onboard servo system. Field tests have demonstrated that this layered and asymmetric architecture ensures that the height error is controlled within the allowable range and is unaffected by sensor signal drift, successfully completing all slalom driving targets.

Claims

1. A method for controlling the serpentine maneuver trajectory of a rotorcraft based on asymmetric deep reinforcement learning, characterized in that, Includes the following steps: A Markov model and hierarchical control structure are constructed for tracking the serpentine maneuver path of a rotorcraft. The environmental state is defined to include real state variables and noisy observation variables. The state variables of the Markov model include the current lateral trajectory tracking error and the look-ahead lateral trajectory tracking error of the predicted point ahead of the rotorcraft. The control variable output by the Markov model is the target roll angle of the rotorcraft. The target roll angle is input to the bottom-level controller to calculate the physical control variables of the rotorcraft. An asymmetric deep reinforcement learning-based policy network and dual evaluation network are constructed. The input of the policy network is the noisy observation variable, and the output is the control action, which is the target roll angle. The input of the dual evaluation network is the real state variable and the control action, and the output is the action value assessment. The policy network and the dual evaluation network are trained using an asymmetric reinforcement learning method. During training, the dual evaluation network is used to calculate the target value based on the real state variables of the next time step to update the dual evaluation network, and the gradient of the dual evaluation network is calculated using the real state variables to guide the policy network, which generates actions based on the noisy observation variables, to be updated. After training, the trained strategy network is used as the final rotorcraft serpentine maneuver control strategy network. In actual flight control, the flight state with sensor noise is input into the strategy network, and the target roll angle control underlying system is output to achieve serpentine flight path tracking of the rotorcraft.

2. The rotorcraft serpentine maneuver trajectory control method based on asymmetric deep reinforcement learning according to claim 1, characterized in that, The method for obtaining the forward lateral trajectory tracking error includes: calculating the relative positional relationship between the current position of the rotorcraft and the estimated point based on the estimated point at a preset distance in front of the current position of the rotorcraft, and obtaining the forward lateral trajectory tracking error.

3. The rotorcraft serpentine maneuver trajectory control method based on asymmetric deep reinforcement learning according to claim 1, characterized in that, The target roll angle is input to the bottom-level controller to calculate the physical control quantities of the rotorcraft, including: inputting the target roll angle together with the target speed and altitude into the bottom-level speed loop, attitude loop and angular rate loop PID controller, and finally calculating the lateral cyclic pitch, longitudinal cyclic pitch, yaw pedal and collective pitch control quantities of the rotorcraft.

4. The rotorcraft serpentine maneuver trajectory control method based on asymmetric deep reinforcement learning according to claim 1, characterized in that, The construction of the asymmetric deep reinforcement learning-based policy network and dual evaluation network includes: constructing a policy network based on noisy observations, wherein the input of the policy network is the noisy observation variable and the output is the target roll angle; and constructing a dual evaluation network based on the real state, wherein the dual evaluation network comprises two networks with the same structure but different parameters, wherein the input is the real state variable and the control action, and the output is the action value evaluation quantity.

5. The rotorcraft serpentine maneuver trajectory control method based on asymmetric deep reinforcement learning according to claim 4, characterized in that, The step of training the policy network and the dual-evaluation network using an asymmetric reinforcement learning method includes: Using a target policy network, smooth target actions are generated based on noisy observation variables at the next time step; Using a target dual-evaluation network, the target Q value is calculated based on the real state variables at the next time step, and the parameters of the dual-evaluation network are updated by gradient descent based on the mean square error between the Q value evaluated by the dual-evaluation network and the target Q value. When the delayed update frequency is reached, the gradient of the dual evaluation network is calculated using the real state variables to guide the update of the policy network that generates actions based on noisy observation variables. The weights of the target evaluation network and the target policy network are updated according to the target soft update tracking ratio.

6. The rotorcraft serpentine maneuver trajectory control method based on asymmetric deep reinforcement learning according to claim 1, characterized in that, The noisy observation variable is generated by injecting Gaussian sensor noise into the real state variable to obtain the noisy observation variable.

7. The rotorcraft serpentine maneuver trajectory control method based on asymmetric deep reinforcement learning according to claim 1, characterized in that, The one-step reward function of the Markov model includes: a track-keeping reward based on the current lateral trajectory tracking error, a heading alignment reward, a forward propulsion reward, a forward velocity error penalty, and a smoothing penalty for changes in adjacent control actions.

8. The rotorcraft serpentine maneuver trajectory control method based on asymmetric deep reinforcement learning according to claim 1, characterized in that, The real state variables include: the lateral deviation distance, horizontal speed, and forward speed error of the rotorcraft in the inertial coordinate system; the three-axis speed, roll angle, pitch angle, and three-axis angular velocity in the body coordinate system; the heading error between the current heading and the tangent direction of the target route; and the action input of the previous control cycle.

9. A rotorcraft serpentine maneuver trajectory control system based on asymmetric deep reinforcement learning, characterized in that, include: The model building module is used to construct a Markov model and hierarchical control structure for tracking the serpentine maneuvering flight path of a rotorcraft. The environmental state is defined by real-state variables and noisy observation variables. The state variables of the Markov model include the current lateral trajectory tracking error and the look-ahead lateral trajectory tracking error at the predicted point ahead of the rotorcraft. The control variable output by the Markov model is the target roll angle of the rotorcraft, which is input to the lower-level controller to calculate the physical control variables of the rotorcraft. A network construction module is used to construct an asymmetric deep reinforcement learning-based policy network and a dual evaluation network. The input of the policy network is the noisy observation variable, and the output is the control action, which is the target roll angle. The input of the dual evaluation network is the real state variable and the control action, and the output is the action value assessment. The training module is used to train the policy network and the dual evaluation network using an asymmetric reinforcement learning method. During the training process, the dual evaluation network is used to calculate the target value based on the real state variables of the next time step to update the dual evaluation network, and the gradient of the dual evaluation network is calculated using the real state variables to guide the policy network, which generates actions based on the noisy observation variables, to be updated. The control execution module is used to take the trained strategy network as the final rotorcraft serpentine maneuver control strategy network after training is completed. In actual flight control, the flight state with sensor noise is input into the strategy network and the target roll angle control underlying system is output to realize the tracking of the rotorcraft's serpentine flight path.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 8.

Images