Model predictive control strategy learning method based on zero-order gradient estimation

Abstract

The invention discloses a model prediction control strategy learning method based on zero-order gradient estimation, which comprises a controlled object model and a neural network control strategy architecture, and realizes offline control strategy learning of a black box or a non-microsystem by alternately performing data acquisition and strategy training in a closed-loop simulation environment. According to the method, a model prediction control parameter optimization problem including system state constraint and input constraint is constructed, a mixed gradient estimation method combining finite difference and back propagation is adopted, and the gradient of a Lagrange loss function to neural network parameters is calculated and iteratively updated. According to the scheme, the control stability, robustness and generalization ability are verified in a double-water-tank system, a Lorentz system and a four-rotor system, and the control precision and training efficiency of a complex nonlinear system are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of model predictive control technology, and more specifically to a model predictive control strategy learning method based on zero-order gradient estimation. Background Technology

[0002] Model Predictive Control (MPC) is an advanced control method that predicts future behavior based on a system model and achieves constraint control by solving for the optimal control sequence online. Due to its good dynamic performance and constraint handling capabilities, this method has been widely used in process control, robot control, and UAV flight control. However, traditional MPC methods rely on accurate system models and high-performance online solvers; their real-time optimization process typically requires significant computational resources, making it difficult to meet real-time requirements in high-dimensional nonlinear systems.

[0003] With the development of machine learning technology, researchers have attempted to introduce neural networks into the MPC framework to achieve offline learning of control policies, thereby avoiding the overhead of online optimization. Common learning-based MPC methods include behavior cloning-based control policy learning and differentiable predictive control (DPC) based on differentiable models. However, behavior cloning methods require a large amount of high-quality data generated by traditional MPC, while DPC requires the system model and loss function to be differentiable throughout the entire training domain. This limits its application in real-world systems containing non-differentiable elements such as logical judgments, dead-zone characteristics, or saturation nonlinearities.

[0004] To address these issues, some studies have introduced zero-order optimization methods, such as Simultaneous Perturbation Stochastic Approximation (SPSA) or Deterministic Coordinate Gradient Estimation (CGE), to train control policies even when gradients are unknown. However, existing zero-order optimization methods suffer from high computational complexity and slow convergence in deep neural network scenarios. For example, in neural networks with hundreds of thousands of parameters, relying solely on zero-order perturbation estimation leads to gradient noise accumulation, resulting in time-consuming and unstable training processes.

[0005] In view of this, the present invention proposes a model predictive control strategy learning method based on zero-order gradient estimation. Summary of the Invention

[0006] The purpose of this invention is to provide a model predictive control policy learning method based on zero-order gradient estimation, which aims to solve the problems of existing learning methods being difficult to efficiently train model predictive control policies in non-differentiable or black-box systems, having high computational complexity, and insufficient real-time performance.

[0007] In a first aspect, the present invention provides a model prediction control policy learning method based on zero-order gradient estimation, comprising the following steps:

[0008] A nominal dynamic model of the controlled object and a neural control strategy network are established to form a closed-loop simulation environment. Based on the closed-loop system, system state and control input data are collected, and an instability threshold is set to filter stable trajectories to obtain a closed-loop simulation dataset.

[0009] A model predictive control parameter optimization problem is constructed, which includes system state constraints and input constraints, with neural network parameters as optimization variables. A Lagrangian loss function that integrates control performance indicators, state constraints, and input constraints is constructed. A hybrid gradient estimation method combining finite difference and backpropagation is used to calculate the gradient of the loss function with respect to the neural network parameters.

[0010] The neural network parameters are updated based on the gradient until the loss function converges, thus obtaining a neural control strategy that can be used for nonlinear controlled objects.

[0011] As a preferred technical solution of the first aspect of the present invention, in a given prediction time domain Internally, the neural network controls the strategy parameters. As the optimization variable, the objective function of the parameter optimization problem of the Lorentz predictive control model is to minimize the expected cumulative loss function under different initial states and system parameter distributions.

[0012] ;

[0013] in: Network parameters for neural control strategies, For stage cost function, The terminal cost function must satisfy the system dynamic constraints. and constraints and .

[0014] As a preferred embodiment of the first aspect of the present invention, the Lagrange loss function comprises a control performance term, a state constraint penalty term, an input constraint penalty term, and a terminal state penalty term, wherein:

[0015] The control performance term measures the deviation between the system state and the reference trajectory.

[0016] Constraint penalties introduce penalties when the system state or control inputs exceed the allowable range.

[0017] As a preferred embodiment of the first aspect of the present invention, the gradient estimation process includes: applying a finite-amplitude perturbation to the control sequence output by the neural control policy network, calculating the change in the loss function before and after the perturbation to estimate the zero-order gradient of the loss function relative to the control sequence, and then combining the local gradient of the control sequence relative to the neural network parameters calculated by backpropagation to obtain the mixed estimated gradient of the loss function with respect to the network parameters, which is used for policy parameter updating.

[0018] As a preferred technical solution of the first aspect of the present invention, during the strategy training process, closed-loop data acquisition and parameter updates are carried out alternately. When the change amplitude of the loss function is less than the preset threshold in multiple consecutive training rounds, or when the control error tends to stabilize, the training is determined to be converged and the optimization is terminated; and only the data in the stable operating range is retained for training.

[0019] As a preferred embodiment of the first aspect of the present invention, the neural control strategy network adopts a multilayer perceptron structure, the input layer receives system state information, the output layer generates control sequences in the prediction time domain, and the intermediate layer adopts a Gaussian error linear unit activation function to enhance nonlinear mapping capability.

[0020] As a preferred technical solution of the first aspect of the present invention, the controlled object model is a black box dynamics model, which only provides input and output data interfaces and is used to train neural control strategies under non-differentiable or unknown system structure conditions.

[0021] As a preferred technical solution of the first aspect of the present invention, the method is applicable to nonlinear controlled systems with multiple inputs and multiple outputs. The nonlinear controlled system includes a dual-water tank system, a Lorentz chaotic system, and a quadcopter flight control system. It can still maintain closed-loop stable control and accurate trajectory tracking when the system parameter disturbance range reaches 50% of the rated value.

[0022] As a preferred technical solution of the first aspect of the present invention, the hybrid gradient estimation architecture reduces the computational complexity of training from the dimension of neural network parameters to the dimension of output, and significantly improves training efficiency and convergence speed while maintaining control performance.

[0023] Secondly, the present invention provides a neural network control strategy training system for executing the first aspect, comprising a controlled object model and a neural network control strategy architecture.

[0024] The controlled object model is a black box model, used to access the model's input and output data, but it is impossible to directly obtain the model's internal state or parameter information;

[0025] The neural network control strategy architecture is used to provide corresponding control commands based on the system response data output by the controlled object model, thus serving as a complete controller.

[0026] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0027] This invention introduces a neural control policy network into the model predictive control framework and employs a hybrid gradient solution method combining zero-order gradient estimation and backpropagation. This enables efficient training of the neural control policy without requiring the controlled object to analyze gradient information. Because the loss function integrates control performance, state constraints, and input constraints, and uses the Lagrange multiplier method to uniformly handle constraints, the trained neural control policy can achieve stable control and trajectory feasibility under complex constraints. Furthermore, the use of a closed-loop simulation data acquisition and iterative optimization mechanism allows the control policy to continuously improve control performance and generalization capabilities in real dynamic environments. Thus, while maintaining the performance advantages of traditional model predictive control, it significantly reduces online computational complexity and improves adaptability and robustness in black-box or nonlinear systems. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0029] Figure 1 This is a schematic diagram of the model predictive control strategy training process provided in the embodiments of this application;

[0030] Figure 2 This is a diagram illustrating the closed-loop control effect of a dual-tank system provided in an embodiment of this application.

[0031] Figure 3 A diagram illustrating the closed-loop control effect of the Lorentz system provided in this application embodiment;

[0032] Figure 4 A diagram illustrating the trajectory tracking and control effect of a quadcopter system provided in an embodiment of this application;

[0033] Figure 5 The figure shows the experimental results of the generalization of the control strategy of the quadrotor system provided in the embodiments of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0035] Throughout the accompanying drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. The described embodiments are only a part of the embodiments of this application, not all of them. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0036] Example 1

[0037] This application provides a neural network control strategy training system, including: a controlled object model and a neural network control strategy architecture.

[0038] The controlled object model is a black box model, used to access the model's input and output data, but it is impossible to directly obtain the model's internal state or parameter information;

[0039] The neural network control strategy architecture is used to provide corresponding control commands based on the system response data output by the controlled object model, thus serving as a complete controller.

[0040] The neural network control strategy training system described in this embodiment can execute a model predictive control strategy learning method based on zero-order gradient estimation. By alternately performing data acquisition and strategy optimization in a closed-loop simulation environment, the neural control strategy is trained to achieve stable and efficient control of the controlled object.

[0041] Example 2

[0042] Please see Figure 1 As shown, this embodiment provides a model predictive control strategy learning method based on zero-order gradient estimation, which is applied to the neural network control strategy training system described in Embodiment 1, and includes the following steps;

[0043] S101, Construct a parameter optimization problem based on model predictive control and a Lagrange loss function.

[0044] It should be noted that traditional MPC requires solving a constrained optimization problem in each control cycle to obtain the optimal control sequence. However, this online optimization process is computationally intensive and difficult to meet the requirements of high-frequency real-time control. Therefore, this embodiment, based on the idea of ​​Model Predictive Control, introduces a neural network as a control strategy approximator. Through offline training, it learns the mapping relationship from system state to control sequence. In the inference phase, only one forward propagation is needed to output the control quantity, significantly improving control real-time performance. During the training phase, the parameters of the neural network are considered as optimization variables, and the degree of mismatch between the data response predicted by the black-box model and the reference trajectory is used as the optimization objective function. Since the black-box model is not differentiable, traditional backpropagation cannot directly obtain the gradient. Therefore, a zero-order gradient estimation method is adopted. By performing finite difference estimation on the perturbation of network parameters and the resulting change in the objective function, the gradient direction is approximately obtained, thereby achieving iterative updates of the network parameters. After training, the neural network can generate near-optimal control sequences under unknown perturbations and complex constraints, replacing the online optimization process of traditional MPC and achieving high-precision, high-real-time control strategy generation.

[0045] In the parameter optimization problem based on model predictive control, the optimization variables are the neural network parameters of the neural control strategy (the neural network parameters include parameters such as the weights and biases of the neural network). The Lagrangian function encodes the control performance, state, and control constraints. By solving this optimization problem, the optimized network parameters are obtained to obtain the trained neural control strategy.

[0046] Understandably, the optimization framework structure includes a parameter optimization problem based on model predictive control that contains system state constraints and input constraints, and a Lagrange loss function as the optimization objective (the optimization objective is to minimize the loss function, and the constructed parameter optimization problem also needs to satisfy the constraints while optimizing the controller performance. Therefore, the constraints are designed in the form of penalty functions, and a large penalty is given for violating the constraints. Thus, several penalty functions can be weighted and added with the control performance metric function to obtain a complete loss function. Since the constraints are already included in the loss function, the optimization problem becomes an unconstrained optimization problem, and we only need to minimize the loss function).

[0047] Among them, the optimization framework structure includes the model prediction control parameter optimization problem containing system state constraints and input constraints, and the Lagrange loss function.

[0048] The optimization variables for the parameter optimization problem based on model predictive control are the network parameters of the neural control strategy architecture. A Lagrange loss function is constructed based on the control performance index, state constraints, and input constraints. The Lagrange loss function encodes the control performance and constraint information. The trained neural control strategy can be obtained by solving the model predictive control parameter optimization problem.

[0049] To further explain, the Lagrange loss function includes a control performance term, a state constraint penalty term, an input constraint penalty term, and a terminal state penalty term. It is used to comprehensively measure the control accuracy and the degree of constraint satisfaction, and to provide the objective function basis for subsequent neural control policy training.

[0050] S102, based on the closed-loop system composed of the controlled object model and the neural control strategy architecture, data is collected to obtain the closed-loop simulation dataset;

[0051] Specifically, data acquisition is performed based on the neural network control strategy described in Example 1. The neural control strategy architecture takes the current system state as input and outputs control commands to drive the controlled object model. The controlled object model generates the system state at the next moment according to the control commands, forming a closed-loop operation mechanism of state-control-feedback. In the closed-loop simulation environment, system state, control input, and constraint-related data are continuously collected, and an instability threshold is set to determine the effective range of system operation. When the system state exceeds the instability threshold, the current simulation process is terminated, and the collected effective state trajectory and corresponding control data are stored in the dataset.

[0052] S103 uses the collected closed-loop simulation dataset to train the neural control strategy based on the hybrid gradient estimation method.

[0053] It should be understood that using the collected data as the initial state, a neural control policy can output a control sequence. Based on the controlled object model, the loss function value under this control sequence can be calculated. By perturbing the control sequence and calculating the loss function value, the zeroth-order gradient of the loss relative to the control sequence can be obtained using the zeroth-order gradient method. Then, backpropagation is used to calculate the gradient of the loss function relative to the neural control policy parameters, which is used for training the neural control policy. After a certain number of neural control policy updates, data can be collected again for retraining, thus iteratively obtaining a well-trained neural control policy. In other words, based on the closed-loop simulation dataset collected in step S102, the neural control strategy is trained with optimized parameters. The training process is performed on the parameter optimization problem and Lagrangian function described in Example 1. The value of the Lagrangian function is calculated based on the collected initial state and other data, and the gradient of the Lagrangian function value relative to the network parameters of the neural control strategy is calculated using a hybrid gradient estimation method. The calculated gradient is then combined with the gradient descent method for optimization, thereby achieving training and updating. Through continuous gradient descent optimization, the average value of the Lagrangian function calculated on the collected dataset for the control commands generated by the neural control strategy is reduced.

[0054] In one possible implementation, a multi-input multi-output controlled object is selected. Regarding the content in S103, this application embodiment also provides a possible implementation, which is described in detail below:

[0055] The training logic for neural control strategies based on the hybrid gradient estimation method is as follows:

[0056] S103-1, The parameter optimization problem of constructing the Lorentz predictive control model based on the optimization framework structure established in step S101; wherein, the Lorentz predictive control model is a nonlinear model predictive control mathematical model established based on the Lorentz system dynamics equation, which is used to realize the rolling prediction of system state and optimization of control input within the finite prediction time domain.

[0057] To accommodate complex nonlinear controlled objects with multiple inputs and outputs, this embodiment uses a three-input, three-output nonlinear chaotic system, namely a Lorentz chaotic system. This Lorentz chaotic system is a simplified model of fluid convection in meteorology, used to describe the behavior of convection in the atmosphere. However, due to its nonlinear characteristics, the system exhibits complex chaotic phenomena. Its differential equation model with control variables (Lorentz system dynamics model) is as follows:

[0058] ;

[0059] in, This is the Prandtl number. This is the Rayleigh number. For example: (This refers to a geometric factor.) , and These represent the three state variables of the system. , and These represent the rates of change corresponding to the state variables. , and The system's control input is used to apply independent external control to each state channel to regulate the system's evolution trajectory. By introducing three independent control inputs, external regulation is applied to each state channel to achieve dynamic control of complex systems whose state evolution trajectories are extremely sensitive to initial values ​​and cannot be directly solved by traditional gradient-based optimization methods. This is used to verify the trainability and stability of the neural control strategy based on zero-order gradient estimation proposed in this invention in non-differentiable or black-box systems.

[0060] Based on the aforementioned differential equation model, state constraints and input constraints are defined within the model predictive control optimization framework. A mathematical model is established to predict future state evolution and generate control input sequences, namely the Lorentz predictive control model. This model is used for subsequent Lagrange loss function calculation and neural control policy parameter optimization.

[0061] Specifically, in a given prediction time domain Internally, the neural network controls the strategy parameters. As the optimization variable, minimizing the expected cumulative loss function under different initial states and system parameter distributions, the complete parameter optimization problem of the Lorentz predictive control model (including optimization objective, optimization variables, dynamic constraints, state constraints, and control constraints) is as follows:

[0062] ;

[0063] ;

[0064] ;

[0065] ;

[0066] ;

[0067] ;

[0068] in, These are the network parameters for the neural control strategy, i.e., the decision variables for the optimization problem; This represents the expectation under the initial state and the distribution of constraint parameters, corresponding to the optimization objective of the optimization problem (minimizing this objective). This is the stage cost function, used to measure the deviation between the system state and the reference state at the k-th prediction step; This is the terminal cost function, used to ensure the stability of the prediction at the end of the time domain;

[0069] This is a discrete-time evolution model of a Lorentz system, describing the dynamic changes of the system state under control input, where sub-indexes... Indicates at time Status and input; To find the desired neural control strategy, a strategy derived from the system state and parameters To the optimal control sequence mapping ; Indicates the prediction range. Reference values ​​representing the state. and These represent state constraints and input constraints, respectively. and These are state constraints and control constraints, respectively. and For the MPC cost function term with terminal penalty; These are the state dimension, the input dimension, and the reference state dimension.

[0070] It should be noted that by minimizing the above objective function, the optimal neural control strategy that meets the control performance requirements can be obtained under given constraints. The theoretical construction of the parameter optimization problem is independent of the closed-loop simulation dataset. However, in the subsequent training process, the closed-loop simulation data collected in step S102 can be used to numerically approximate the expectation operator, thereby realizing the calculation of the loss function and the optimization and updating of the control strategy parameters.

[0071] S103-2, based on the parameter optimization problem, jointly models the objective function and constraints, constructs a unified Lagrangian loss function to measure the control performance and constraint satisfaction of the neural control strategy in the prediction time domain, transforms the original constrained optimization problem into an unconstrained optimization problem, which is convenient for optimization using the gradient descent method.

[0072] Optionally, the Lagrangian loss function of the unified constraint and control performance index is used to transform the above-mentioned constrained parameter optimization problem into an unconstrained optimization problem, and the function is the optimization objective of the transformed unconstrained optimization problem:

[0073] ;

[0074] Part One To measure control performance, a custom loss function can be defined based on the specific control task. For example, a quadratic loss function can be used to track a reference signal. Part Two and Part Three These are the state constraint penalty function and the input constraint penalty function, which can be defined according to task requirements; Part Four This is the terminal state penalty term, which is a commonly used part to ensure the stability and recursive feasibility of the MPC closed-loop system. It can be designed with reference to the terminal penalty function in the classic MPC problem. , , and The loss function is divided into weights for each part. In this Lorentz system, due to the simplicity of the task, only the control performance metric is used, i.e. , and All are zero.

[0075] This can be understood as follows: when both the system state and input satisfy the constraints, the penalty term approaches zero; if there is an over-limit or constraint violation, the penalty term increases rapidly with the degree of deviation, thereby guiding the policy update direction to converge towards the feasible region. By minimizing the Lagrange loss function, the optimal neural control policy parameters that balance control accuracy and stability under constraints can be obtained. This loss function provides the objective function basis for the subsequent training process based on hybrid gradient estimation.

[0076] S103-3, based on a closed-loop simulation dataset and the Lagrange loss function, employs a hybrid gradient estimation method to optimize and train the parameters of a neural control strategy.

[0077] It should be understood that the collected closed-loop simulation dataset is used as the initial state sample, the neural control policy outputs a control sequence, and the loss function value under this control sequence is calculated through the controlled object model. Based on this, a small perturbation is applied to the control sequence and the loss change is calculated. The zero-order gradient method is used to obtain the zero-order estimated gradient of the loss function with respect to the control sequence. Subsequently, the backpropagation algorithm is combined to calculate the gradient of the loss function with respect to the neural control policy parameters, thus achieving hybrid gradient estimation.

[0078] Optionally, gradient estimation is used to perform zero-order gradient estimation, calculating the gradient of the loss function with respect to the network parameters—optimization variables—for subsequent gradient descent parameter optimization. The gradient estimation process can be expressed as:

[0079] ;

[0080] in, =3 represents the input dimension of the controlled object. The gradient of the estimated loss with respect to the parameters of the neural control policy. The gradient of the estimated loss with respect to the output of the neural control policy. This is the gradient of the neural control policy output relative to the parameters of the neural control policy.

[0081] By employing the hybrid gradient estimation described above, effective training of non-differentiable or black-box systems can be achieved while maintaining the advantages of model predictive control performance. After several rounds of policy parameter updates, closed-loop simulation data is re-acquired, and the training process is repeated until the loss function converges and the trained neural control policy is obtained.

[0082] S103-4, Data is collected again using the trained neural control strategy to obtain a new dataset, and the neural control strategy is trained based on the new data. This process is repeated to obtain a well-trained neural control strategy.

[0083] Specifically, in this embodiment, the convergence state and termination time of the training process are determined by monitoring the rate of change of the Lagrange loss function and the stability of the control performance index. The training process is considered converged when the decrease in the loss function is less than a threshold in several consecutive training iterations, or when the system control error tends to stabilize.

[0084] When the convergence condition is met or the maximum number of iterations is reached, the training process is terminated, and the final neural control policy parameters are output. These policy parameters can be deployed on the controlled object to achieve stable nonlinear system control, thus completing the entire model predictive control policy learning process based on zero-order gradient estimation.

[0085] Example 3

[0086] The parts not detailed in this implementation are shown in Example 2. By combining finite difference gradient estimation and backpropagation algorithm, this method effectively solves the problem of difficult control strategy training in non-differentiable or black-box systems while maintaining the performance advantages of traditional model predictive control. This method employs a closed-loop simulation data generation mechanism and a composite constraint processing strategy to achieve efficient control strategy learning under different system scenarios. In experiments with a dual-tank system and a Lorentz system, the neural control strategy trained using the proposed hybrid zero-order gradient estimation method (ZPC) showed tracking errors similar to the first-order method (DPC) based on differentiable simulation, but performed better in terms of convergence, robustness, and adaptability.

[0087] like Figure 2 The figure shows the closed-loop simulation test results on a dual-tank system. The left side shows the control effect of the control strategy trained using the ZPC method, and the right side shows the control effect trained using the DPC method. The upper part of the figure shows the changes in the desired liquid level and the actual liquid level after control in the dual-tank system, while the lower part shows the change curves of the two control commands. The results show that the control strategy trained using the ZPC method can achieve rapid and stable tracking of the desired liquid level, with fast dynamic response and small overshoot. In comparison, the DPC method is slightly inferior in terms of steady-state accuracy and disturbance recovery, verifying the effectiveness and superiority of this invention in non-differentiable systems.

[0088] like Figure 3 The figure shows the closed-loop simulation test results on the Lorentz system. The left side shows the effect of the control strategy trained by the ZPC method, and the right side shows the results of the DPC method. The three channels from top to bottom in the figure represent the reference commands and actual state changes after control for the three states of the Lorentz system, respectively. Experiments show that the ZPC method can achieve smooth and accurate tracking of each state variable, and maintain stable closed-loop control under complex nonlinear and chaotic dynamic conditions. Furthermore, when the perturbation range of the quadrotor system parameters reaches 50% of the rated value, the proposed control strategy can still stably track various complex flight trajectories, demonstrating excellent robustness.

[0089] like Figure 4 The image shows the trajectory tracking results under different maximum speeds of the reference trajectory (including circular and nylon tracks). The results show that the control strategy trained using the ZPC method can still smoothly track the reference trajectory in high-speed and highly dynamic scenarios, indicating that the strategy has good adaptability to the trajectory change rate.

[0090] like Figure 5 The figure shows the variation of circular trajectory tracking error under different model parameters (mass, arm length, moment of inertia, etc.) of the quadrotor system, ranging from 0% to 50%, and at different flight speeds (0.5 m / s to 3 m / s). The results demonstrate that the control error is relatively unaffected by changes in system parameters, showcasing the generalization and stability of the proposed method under model uncertainty. Its hybrid gradient estimation architecture reduces computational complexity from the order of magnitude of network parameters to the order of magnitude of output dimensions, significantly improving training efficiency and providing an efficient and feasible solution for offline control strategy learning in complex nonlinear systems.

[0091] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model predictive control strategy learning method based on zero-order gradient estimation, characterized in that, Includes the following steps: A nominal dynamic model of the controlled object and a neural control strategy network are established to form a closed-loop simulation environment. Based on the closed-loop system, system state and control input data are collected, and an instability threshold is set to filter stable trajectories to obtain a closed-loop simulation dataset. A model predictive control parameter optimization problem is constructed, which includes system state constraints and input constraints, with neural network parameters as optimization variables. A Lagrangian loss function that integrates control performance indicators, state constraints, and input constraints is constructed. A hybrid gradient estimation method combining finite difference and backpropagation is used to calculate the gradient of the loss function with respect to the neural network parameters. The neural network parameters are updated based on the gradient until the loss function converges, thus obtaining a neural control strategy that can be used for nonlinear controlled objects.

2. The model predictive control strategy learning method based on zero-order gradient estimation according to claim 1, characterized in that, In a given prediction time domain Internally, the neural network controls the strategy parameters. As the optimization variable, the objective function of the parameter optimization problem of the Lorentz predictive control model is to minimize the expected cumulative loss function under different initial states and system parameter distributions. ； in: Network parameters for neural control strategies, For stage cost function, The terminal cost function must satisfy the system dynamic constraints. and constraints and .

3. The model predictive control strategy learning method based on zero-order gradient estimation according to claim 2, characterized in that, The Lagrange loss function consists of a control performance term, a state constraint penalty term, an input constraint penalty term, and a terminal state penalty term, wherein: The control performance term measures the deviation between the system state and the reference trajectory. Constraint penalties introduce penalties when the system state or control inputs exceed the allowable range.

4. The model predictive control strategy learning method based on zero-order gradient estimation according to claim 3, characterized in that, The gradient estimation process includes: applying a finite-amplitude perturbation to the control sequence output by the neural control policy network, calculating the change in the loss function before and after the perturbation to estimate the zero-order gradient of the loss function relative to the control sequence, and then combining the local gradient of the control sequence relative to the neural network parameters calculated by backpropagation to obtain the mixed estimated gradient of the loss function with respect to the network parameters, which is used for policy parameter updates.

5. The model predictive control strategy learning method based on zero-order gradient estimation according to claim 4, characterized in that, During the policy training process, closed-loop data acquisition and parameter updates are carried out alternately. When the change in the loss function is less than the preset threshold in multiple consecutive training rounds, or when the control error tends to stabilize, the training is determined to be converged and the optimization is terminated; and only the data in the stable operating range is retained for training.

6. The model predictive control strategy learning method based on zero-order gradient estimation according to claim 5, characterized in that, The neural control strategy network adopts a multilayer perceptron structure. The input layer receives system state information, the output layer generates control sequences in the prediction time domain, and the intermediate layer uses Gaussian error linear unit activation function to enhance nonlinear mapping capability.

7. The model predictive control strategy learning method based on zero-order gradient estimation according to claim 6, characterized in that, The controlled object model is a black box dynamics model, which only provides input and output data interfaces and is used to train neural control strategies under non-differentiable or unknown system structure conditions.

8. The model predictive control policy learning method based on zero-order gradient estimation according to claim 7, characterized in that, The method is applicable to nonlinear controlled systems with multiple inputs and multiple outputs, including dual-tank systems, Lorentz chaotic systems, and quadcopter flight control systems. It can still maintain closed-loop stable control and accurate trajectory tracking when the system parameter disturbance range reaches 50% of the rated value.

9. The model predictive control strategy learning method based on zero-order gradient estimation according to claim 8, characterized in that, The hybrid gradient estimation architecture reduces the computational complexity of training from the neural network parameter dimension to the output dimension, significantly improving training efficiency and convergence speed while maintaining control performance.

10. A neural network control strategy training system, characterized in that, The method for learning a model predictive control policy based on zero-order gradient estimation as described in any one of claims 1 to 9 includes a controlled object model and a neural network control policy architecture. The controlled object model is a black box model, used to access the model's input and output data, but it is impossible to directly obtain the model's internal state or parameter information; The neural network control strategy architecture is used to provide corresponding control commands based on the system response data output by the controlled object model, thus serving as a complete controller.

Images