A simplified reinforcement learning optimal tracking control method with input constraints

By simplifying reinforcement learning and high-order all-drive system theory, a three-network architecture is constructed and a synchronization error update law is designed. This solves the problems of computational resource overload, continuous excitation and input constraints in reinforcement learning in real physical systems, and achieves efficient and stable optimal tracking control.

CN123113355APending Publication Date: 2026-11-17NANKAI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611375670.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-07
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing reinforcement learning optimal control techniques face problems such as computational resource overload, continuous excitation condition limitations, and inability to handle actuator input constraints in real physical systems, resulting in compromised system stability and accuracy.

Method used

A simplified reinforcement learning method is adopted, and the system is transformed into a standard error dynamic model through high-order all-drive system theory. A three-network architecture is constructed and a synchronous error update law is designed. Combined with non-quadratic performance index and hyperbolic tangent function, optimal control under input constraints is achieved.

Benefits of technology

It achieves optimal tracking control with low computational overhead and high precision, eliminates the curse of dimensionality, eliminates the need for continuous excitation conditions, solves the system instability caused by actuator saturation, and ensures semi-globally consistent eventual boundedness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123113355A_ABST
    Figure CN123113355A_ABST
Patent Text Reader

Abstract

This invention discloses a simplified reinforcement learning optimal tracking control method and system with input constraints, belonging to the field of automatic control technology. Addressing the problems of the "curse of dimensionality," reliance on continuous excitation conditions, and difficulty in handling input constraints in existing reinforcement learning control, this invention utilizes high-order full-drive theory for state reconstruction, constructing a minimalist architecture containing only three neural networks: an identifier, an evaluator, and an executor, completely eliminating the computational bloat of traditional recursive designs. It introduces a non-quadratic performance index function and dynamic equivalent gain technique to directly generate an analytical optimal control law with hyperbolic tangent saturation characteristics, strictly ensuring that the actuator output does not exceed the limits. Furthermore, it designs a structurally decoupled synchronization error update law, eliminating the PE condition and achieving stable weight convergence and high-precision tracking without probe noise. This invention has extremely low computational overhead, strong robustness, and high engineering deployment feasibility.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of automatic control and artificial intelligence, and in particular to an optimal tracking control method and system based on simplified reinforcement learning (RL) and high-order fully actuated (HOFA) theory for unknown high-order nonlinear systems with actuator input saturation constraints. Background Technology

[0002] In modern practical engineering applications (such as space robotic arm control, multi-rotor UAV trajectory tracking, and autonomous surface vessels), the controlled systems typically exhibit strong nonlinearity, high-order dynamics, and highly uncertain internal parameters. Reinforcement learning (RL)-based optimal control methods, due to their powerful self-learning and online approximate dynamic programming capabilities, have become the mainstream solution for solving such complex nonlinear system control problems. However, existing reinforcement learning-based optimal control techniques face the following three severe technical bottlenecks when deployed in real physical systems:

[0003] First, the "curse of dimensionality" and computational resource overload: existing RL control methods for strictly feedback nonlinear systems heavily rely on a backstepping recursive design framework. For In a hierarchical system, each virtual control subsystem requires an independent network of identifiers, critics, and actors. This means the entire control system requires maintenance and online updates. A neural network. In embedded devices with limited computing power, this can lead to severe computational latency, making it impossible to meet real-time control requirements.

[0004] Second, the constraint of continuous incentive (PE): existing algorithms inevitably introduce the product term of the basis function vector and its transpose when deriving the network weight update law for the evaluator and executor (i.e., The matrix has a minimum eigenvalue that is zero over most of the operating range, causing stability analysis based on the Lyapunov equations to fail. To ensure network weight convergence, the system often needs to artificially inject broadband probe noise to meet the stringent continuous excitation (PE) condition, which severely compromises the system's steady-state tracking accuracy.

[0005] Third, the system lacks the ability to handle input constraints (actuator saturation): the output amplitude of actual physical actuators (such as motors and servo valves) is strictly constrained by physical limits. Existing simplified RL control architectures are typically only applicable to ideal linear intervals without input constraints. If an unconstrained optimal algorithm is forcibly applied directly to a real system, it will generate extremely large control commands when faced with large initial errors, leading to severe actuator saturation, drift and divergence of learned parameters, and ultimately causing system instability and collapse. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, such as high computational complexity, reliance on continuous excitation conditions, and inability to handle physical input amplitude constraints, the present invention aims to provide an optimal control method and system for high-order nonlinear systems with input constraints based on simplified reinforcement learning. The technical solution of the present invention is as follows:

[0007] An optimal control method for high-order nonlinear systems with input constraints based on simplified reinforcement learning includes the following steps:

[0008] Step S1: Obtain the current state information of the controlled high-order nonlinear system, as well as the desired reference trajectory signal;

[0009] Step S2: Using the theory of high-order all-drive systems, the dynamic model of the controlled high-order nonlinear system is converted into an equivalent standard error dynamic system model;

[0010] Step S3: Define a non-quadratic performance index function, which includes a quadratic penalty term for tracking error and a non-quadratic integral penalty term for control input saturation, to reflect the physical input amplitude constraint of the actuator;

[0011] Step S4: Construct a simplified reinforcement learning architecture consisting of only three single-hidden-layer neural networks, including: an identifier network for online approximation of the dynamics of an unknown system, an evaluator network for evaluating the gradient of performance metrics, and an executor network for generating the optimal control policy;

[0012] Step S5: Based on the standard error dynamic system model and performance index function, design the synchronization error update law for the three neural networks; the update law iteratively updates the neural network weights online without relying on the continuous excitation (PE) condition;

[0013] Step S6: Based on the updated executor network weights and combined with the dynamic equivalent gain technique, calculate the actual control input containing the hyperbolic tangent saturation function, and apply it to the controlled system.

[0014] The beneficial effects of this invention include:

[0015] Completely eliminate the curse of dimensionality: Deeply integrate reinforcement learning with HOFA theory, completely abandoning the multi-level recursive design of backstepping. For any For a first-order nonlinear system, global optimal control can be achieved with only three neural networks, compared to traditional methods. This network greatly reduces computational overhead.

[0016] Eliminating the Continuous Excitation (PE) Condition: A novel synchronization error update law for structural decoupling is proposed, completely eliminating the tendency of minimal eigenvalues ​​to tend to zero in traditional update laws. Matrix coupling terms. The system can still guarantee stable convergence of the neural network weight estimation error even without any external probe noise.

[0017] Perfectly solves the controllability bottleneck under severe saturation: Introduces a non-quadratic integral cost function to directly generate native hyperbolic tangent at the executor network output. The analytical optimal control law for the function is further demonstrated theoretically. It is shown that, provided specific control gain boundary conditions are met, the closed-loop system signal still exhibits semi-globally uniform eventual boundedness (SGUUB) even under extremely saturated conditions. Attached Figure Description

[0018] Figure 1 A flowchart illustrating an optimal control method for a high-order nonlinear system with input constraints based on simplified reinforcement learning, provided in an embodiment of the present invention;

[0019] Figure 2 A block diagram illustrating the interaction logic between the high-order all-drive (HOFA) transformation and the simplified three-network architecture provided in this embodiment of the invention;

[0020] Figure 3 A structural block diagram of an optimal control system provided in an embodiment of the present invention;

[0021] Figure 4 This is a simulation diagram of the constrained control input curve under the introduction of input constraints in an embodiment of the present invention;

[0022] Figure 5 The figures show the bounded convergence curves of the weights of the three neural networks (identifier, evaluator, and executor) in this embodiment of the invention, as well as the simulation results of the synchronization error convergence between the evaluator and the executor. Detailed Implementation

[0023] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0024] Example 1: Optimal Control Method

[0025] Reference Figure 1 and Figure 2 This embodiment provides an optimal control method for high-order nonlinear systems with input constraints based on simplified reinforcement learning. The detailed execution steps are as follows:

[0026] Steps S1 to S2: System Modeling and High-Order All-Drive (HOFA) Transformation

[0027] Regarding an unknown For a strictly feedback nonlinear system, a high-order total-drive (HOFA) transformation technique is used to eliminate hierarchical coupling. The system output tracking error is defined as... ,in For system output, The set reference trajectory signal.

[0028] By reconstructing the state, the original high-order system is equivalently mapped to a dynamic system model with standard error:

[0029]

[0030] in, Indicating error First-order time derivative, For the constrained control input of the actuator to be designed, This is to centralize the lumped nonlinear functions that contain all the unknown dynamics of the original system. Let the vector be a state vector. The above equation can be written in state-space form:

[0031]

[0032] in , For the standard integral chain matrix, .

[0033] Step S3: Design of non-quadratic cost function with input constraints to ensure actual control quantity Strictly not exceeding the physical limit amplitude of the hardware actuator Instead of the traditional quadratic control penalty, we define the following non-quadratic performance index function with input constraints:

[0034]

[0035] in, It is a positive definite weight matrix. To control the weighting parameters. This non-quadratic penalty term is used in the control quantity. Approaching the physical limit When its value approaches infinity, it inherently limits the optimal control law from going out of bounds.

[0036] Based on optimal control theory, by solving the Hamilton-Jacobi-Bellman (HJB) equations, the analytical optimal control law with physical constraints is derived as follows:

[0037] .

[0038] Step S4: Constructing a Minimalist Reinforcement Learning Three-Network Architecture

[0039] Reference Figure 2 Construct an Actor-Critic-Identifier architecture consisting of exactly 3 single-hidden-layer neural networks:

[0040] 1. Identifier Networks: Used to approximate the dynamics of unknown systems. Let its basis functions be... The weights are estimated as follows: The system dynamic estimate is then... .

[0041] 2. Critic network: used to approximate the gradient of the cost function. Through mathematical reconstruction, its estimated value is expressed as... .in For constant gain, This is a positive definite solution to the Lyapunov equation. For evaluator weight estimation, , which is the error state basis function.

[0042] 3. Actor Network: Used to output final control commands. Substituting the analytical optimal control law, we obtain the executor output as:

[0043]

[0044] Step S5: Execution of the Synchronous Update Law Without PE Conditions To ensure the convergence of online learning of network weights and eliminate the constraint of continuous activation (PE) conditions, the following adaptive online update law is designed:

[0045]

[0046]

[0047]

[0048] The above update law design introduces evaluator weights. To guide the weight of the executor This makes the equivalent condition for the optimal Hamiltonian residual directly point to This decoupling design completely avoids the need for unsatisfactory rank conditions in traditional algorithms. The inversion of a matrix depends on its positive definiteness.

[0049] Step S6: Guarantee and Implementation of Semi-Globally Consistent Eventually Bounded (SGUUB)

[0050] Due to actual control input Include To address saturated nonlinearity, this invention introduces an equivalent gain technique. The equivalent gain is defined as follows: ,in This is the virtual control input inside the hyperbolic tangent function.

[0051] To ensure the stability of the closed-loop control system even when deep actuator saturation occurs, this invention constructs a Lyapunov function. Furthermore, by differentiating and scaling, the following gain boundary conditions must be satisfied:

[0052] ,in The number of network nodes. Let the system order be . This represents the equivalent minimum attenuation gain under extremely saturated conditions. During algorithm initialization, a reasonably large control gain is selected. And in order to keep the initial state within a controllable region ( This ensures that the system can overcome initial large error drift.

[0053] Reference Figure 4 The controlled system can be smoothly truncated at the input. Within the boundary, no integral saturation phenomenon occurs;

[0054] Reference Figure 5 The three neural network weights converged synchronously to the smallest neighborhood without the injection of probe noise, achieving high-precision optimal tracking control.

[0055] Example 2: Optimal Control System

[0056] Reference Figure 3 The present invention also provides an optimal control system for a high-order nonlinear system with input constraints based on simplified reinforcement learning. This system is used to implement the method described in Example 1, and includes:

[0057] Data acquisition module: Connected to sensors, it is used to acquire the current state information of the controlled high-order nonlinear system and receive externally set reference trajectory signals;

[0058] State transition module: Embedded with the HOFA transformation algorithm, used to convert the physical state variables of a nonlinear system into a standard error dynamic model required for control implementation;

[0059] The reinforcement learning control module is the core computing unit of this system. It contains only three single-hidden-layer neural networks and a synchronization error update algorithm without PE conditions. It uses a non-quadratic cost function for online evaluation and calculates the optimal control law in real time that strictly satisfies the actuator amplitude upper limit constraint.

[0060] Control execution module: Includes digital-to-analog conversion and power amplification circuits, used to convert the numerical optimal control law output by the reinforcement learning control module into physical drive signals and apply them to the actuators of the controlled mechanical or electrical system.

[0061] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An optimal control method for high-order nonlinear systems with input constraints based on simplified reinforcement learning, characterized in that, Includes the following steps: Steps S1: Obtain the current state information of the controlled high-order nonlinear system, as well as the desired reference trajectory signal; Step S2: Using the High-Order All-Drive (HOFA) system theory, the dynamic model of the controlled high-order nonlinear system is transformed into an equivalent standard error dynamic system model. Step S3: A non-quadratic performance index function is defined, which includes a quadratic penalty term for the system tracking error and a non-quadratic integral penalty term for control input saturation, reflecting the physical input amplitude constraints of the actuator. Step S4: A simplified reinforcement learning architecture consisting of only three single-hidden-layer neural networks is constructed. This simplified reinforcement learning architecture includes: an Identifier network for online approximation of the unknown system dynamics, a Critic network for evaluating the gradient of the non-quadratic performance index function, and an Actor network for generating the optimal control strategy. Step S5: Based on the standard error dynamic system model and the non-quadratic performance index function, a synchronization error update law for the three neural networks is designed. This synchronization error update law iteratively updates the neural network weights online without relying on continuous excitation (PE) conditions. Step S6: Calculate the actual control input containing the hyperbolic tangent saturation function based on the updated actuator network weights, and apply it to the actuator of the controlled high-order nonlinear system.

2. The method according to claim 1, characterized in that, In step S3, the non-quadratic performance index function is specifically defined as follows: in, To track the error state vector, It is a positive definite weight matrix. To control the weight parameters, For actual control input, The maximum physical input constraint amplitude of the actuator. It serves as an auxiliary variable for integration.

3. The method according to claim 1, characterized in that, In step S6, the actual control input The calculation formula is: in, For the set control gain, Let be the positive definite solution matrix of the Lyapunov equations. The input matrix is ​​a constant. For reference trajectory signal First-order time derivative, and These are the weight estimation vector and basis function vector of the identifier network, respectively. and These are the weight estimation matrix and basis function vector of the executor network, respectively.

4. The method according to claim 1, characterized in that, In step S5, the synchronization error update law of the evaluator network and the executor network is designed as follows: in, This is the weight estimation matrix for the evaluator network. This is the weight estimation matrix for the executor network. and These are learning rate parameters that are greater than zero. It is a row vector with all elements being 1; the synchronization error update law eliminates the influence of the minimum eigenvalue of the covariance matrix of the neural network basis function being zero through decoupling design.

5. The method according to claim 3, characterized in that, To ensure the semi-globally consistent eventual boundedness (SGUUB) of the system under input saturation, the control gain... The following conditions must be met: in, Let the system order be . This represents the number of hidden layer nodes in a single neural network. It is the equivalent minimum decay gain under extremely saturated conditions, and is determined by the inherent nonlinearity of the initial state of the system.

6. An optimal control system for a high-order nonlinear system with input constraints based on simplified reinforcement learning, characterized in that, include: Data acquisition module: used to acquire the current state information and reference trajectory signal of the controlled high-order nonlinear system; State transition module: used to perform high-order full-drive HOFA transformation, converting the state information into an equivalent standard error dynamic model; Reinforcement learning control module: Built-in simplified reinforcement learning architecture and synchronous error update law as described in any one of claims 1 to 5, used for online calculation of the optimal control law that strictly satisfies the physical input amplitude constraints of the actuator; Control execution module: used to convert the optimal control law into actual driving physical signals and apply them to the controlled high-order nonlinear system.