Value function training method based on model prediction extension

Abstract

The invention discloses a value function training method based on model prediction extension, and the method comprises the steps: inputting an environment state through an Actor network, outputting a cost parameter through neural cost mapping, and transmitting the cost parameter to a differentiatable MPC module integrated at a tail end; the differentiable MPC module solves a convex quadratic programming problem in a prediction time domain, generates future prediction states, actions and rewards, and transmits the future prediction states, actions and rewards to the Critic network; the Critic network carries out value estimation and MPVE multi-step return target construction, and learning is carried out from real interaction data and a model prediction trajectory; forward sampling and data interaction are carried out, training and parameter updating of the Critic network and the Actor network are completed, and a trained value function is obtained. According to the method, the sample utilization rate and the value estimation accuracy of reinforcement learning can be improved on the premise that the calculation burden is not increased, so that the strategy can converge more quickly in a complex dynamic environment and has higher robustness and generalization ability.

Description

Technical Field

[0001] This invention relates to the field of robot intelligent control technology, and in particular to a value function training method based on model prediction extension. Background Technology

[0002] In the field of intelligent robot control, the traditional Actor-Critic reinforcement learning algorithm estimates the state-action value function through temporal difference (TD) learning. While it has achieved some success in continuous control tasks, its low sample efficiency is a significant bottleneck, requiring extensive interaction and trial-and-error to converge, which is even more pronounced in high-dimensional and complex environments. Insufficient long-term credit allocation capability further restricts policy optimization efficiency; the single-step TD update mechanism struggles to effectively propagate delayed reward signals, resulting in a lack of long-term guidance for the policy gradient direction. During policy optimization, actions generated by the Actor network may exceed the state-action distribution range of the Critic network's training data, leading to distribution shift. The TD(λ) method is highly sensitive to the λ parameter, making it difficult to achieve a stable optimum in the bias-variance tradeoff. Existing improvements such as the TD(k) trick and experience replay alleviate some problems locally, but still fail to fundamentally resolve the contradiction between long-term credit allocation and distribution shift.

[0003] Model predictive control (MPC) is an online optimization method based on system dynamics models, widely used in high-dynamic tasks such as UAV flight and robotic arm operation. However, its cost function often requires manual design and repeated parameter tuning by experts in the relevant field, resulting in insufficient adaptability and versatility, and difficulty in handling dynamic changes in the task. In most MPC frameworks, high-level tasks need to be converted into reference trajectories (planning layer) before being tracked by the controller (control layer). This hierarchical decoupling approach may lead to abnormal behavior when unmodeled dynamics or environmental disturbances occur, causing the control performance to no longer be optimal. Furthermore, while some sampling-based MPC methods (such as MPPI) can handle complex nonlinear problems, their computational and memory overhead is significant, making it difficult to meet real-time requirements, especially when deployed on embedded robotic systems. For example, when the dimensionality of control variables is high or the prediction time domain is long, the hardware computing unit and storage requirements increase dramatically, thus limiting the application scope of MPC.

[0004] In recent years, researchers have attempted to combine reinforcement learning with MPC to solve the above problems, mainly in three technical paths: (1) Cost function learning method: Automatically learn the cost function hyperparameters of MPC through data-driven methods to improve performance in specific tasks. The learning effect of this type of method is limited by the pre-set cost function basic structure, and the training process relies on a large amount of simulation data, making it difficult to deploy and apply quickly in variable or unknown scenarios. (2) Dynamic model learning method: Improve the prediction accuracy of MPC by optimizing the dynamic model. This type of method is prone to error accumulation in long-term prediction, and model mismatch will cause the controller performance to drop significantly or even sharply in out-of-distribution environments. (3) Joint learning method: Attempt to simultaneously optimize the cost function and dynamic model of MPC to improve control performance. However, due to the conflict between optimization objectives, the training process is often unstable, and the deep coupling problem between the cost function training and the MPC prediction process has not been solved.

[0005] Therefore, existing methods combining reinforcement learning and MPC still suffer from insufficient robustness and poor stability. Summary of the Invention

[0006] The problem to be solved by this invention is to provide a value function training method based on model prediction extension, focusing on the training mechanism of the value function in the Actor-Critic reinforcement learning framework, so as to improve the learning efficiency and decision robustness of the agent in complex, dynamic and uncertain environments.

[0007] This invention adopts the following technical solution: a value function training method based on model prediction extension, comprising the following steps:

[0008] Step 1: Construct the Actor-Critic reinforcement learning framework. Integrate a differentiable MPC module at the end of the Actor network. The state acquisition module inputs the environmental state into the Actor network, outputs the cost parameters through the neural cost mapping network, and passes them to the differentiable MPC module to generate a predicted action sequence and output the first control action to the environment. The environment provides feedback on the next state and reward and writes the data into the storage unit.

[0009] Step 2: In each control iteration, the differentiable MPC module solves the convex quadratic programming problem in the prediction time domain based on the current state, generating the predicted future state, action and reward, which is then passed to the Critic network.

[0010] Step 3: The Critic network performs value estimation and constructs the MPVE multi-step reward objective based on the data and predicted trajectory of the storage unit, while learning from real interaction data and model predicted trajectory.

[0011] Step 4: Construct a reward function, perform forward sampling and data interaction, complete the training and parameter update of the Critic network and Actor network, obtain the trained value function, and generate the internal predicted trajectory and corresponding predicted reward in the current state.

[0012] Preferably, the Actor network described in step 1 takes a normalized observation vector as input. After passing through a neural cost mapping network Output the cost parameters of the quadratic cost function, including: the angular cost matrix. and linear term vectors ; mapped via Sigmoid and affine to the interval ; These represent the minimum and maximum allowable values ​​of the diagonal weights and linear coefficients of the quadratic cost function output by the neural cost mapping network, respectively.

[0013] The observation vector Represented as: , For the observation of the organism's condition, To guide observations based on the trajectory.

[0014] The action space Represented as: , It is the mass-normalized total thrust; It is the body angular velocity command.

[0015] Preferably, in step 2, the differentiable MPC module in the prediction time domain The problem of solving a convex quadratic programming problem internally is expressed as: ; in, For the defined extended vector; This represents the relative time step index within the prediction time domain. ; In order to be in For nonlinear dynamic models The first-order linearization result.

[0016] Preferably, in step 2, the differentiable MPC module is based on the current observation vector. Prediction time domain is Future state-action sequence: This includes the following sub-steps:

[0017] Step 2.1: The neural cost mapping network calculates the current normalized observation vector. Output cost function parameters and Based on the stability of time-domain parameters, the cost parameter It remains unchanged throughout the entire prediction time domain;

[0018] Step 2.2, in the current system state At this point, a first-order linearization is performed using a nonlinear dynamic model to obtain a time-invariant linear approximation model: ;

[0019] Step 2.3, with Solve the convex quadratic programming problem for the initial state to obtain the optimal control sequence. By combining the linear model with forward recursion, the corresponding predicted state sequence is generated. And through the observation mapping function , converted into predicted observation sequence To form a complete predicted trajectory .

[0020] Preferably, in step 3, the MPVE multi-step reward target is constructed as follows:

[0021] Build n-step rewards , used for Loss calculation Indicates from time step Starting, based on the future The n-step reward target value constructed from real-world interaction data.

[0022] Build -Rewards , represented as truncated form, Indicates from time step Rising Weighted mixed return estimate.

[0023] For each point in the predicted trajectory, set the MPVE step-by-step objective: ; in, This indicates that the trajectory is predicted based on the model at time step The predicted reward obtained at that time This represents the Critic network's value estimation function for the input state. Indicates starting from the current state Departure The observation vector corresponding to the terminal state obtained from the step prediction. Indicates time step From the predicted trajectory of time step The initial model predicted the target value for extended returns.

[0024] Constructing the Critic joint loss: ; in, Indicates the state of the Critic network. State value function estimate, This represents the weighting coefficient of the MPVE auxiliary loss. This represents the main value loss based on real-world interaction data. This represents the MPVE auxiliary loss based on the model-predicted trajectory.

[0025] Constructing multi-step reward objectives by predicting trajectories: ; in, Indicates the time from the current moment to the nth The total discount factor for each step This is Critic's estimate of the predicted endpoint state.

[0026] Preferably, in step 4, the training and parameter updating of the Critic network and Actor network are performed as follows:

[0027] Step 4.1, Forward Sampling and Data Interaction: Based on the current strategy Interact with the simulation environment or real physical system to execute several complete trajectories or a rollout of fixed duration; at each time step The intelligent agent performs actions Then, the obtained transfer samples Stored in the experience replay buffer, where, For the current observation value, For immediate rewards for environmental feedback, This is the next state to enter after performing an action.

[0028] Step 4.2, Critic network parameter update:

[0029] The Critic network samples a trajectory segment from the experience replay buffer, synchronously calls the DiffMPC solver, and applies the current state... Generate the corresponding predicted state sequence and predicting reward sequences And through the observation mapping function Transform the predicted state into a predicted observation sequence .

[0030] True trajectory branch: Calculate truncated segments based on sampled fragments -Rewards And construct the time-series difference loss term. .

[0031] Predicted trajectory branch: Extracted from the predicted trajectory calculated by the DiffMPC solver at the corresponding time step. and By observing the mapping function Transform the predicted state into a predicted observation. For each predicted observation Construct a local multi-step reward objective; ;

[0032] And construct the MPVE reward error based on the predicted trajectory: ; The two losses are weighted according to a preset non-negative weight. Weighted summation to form the total value loss function .

[0033] Further use of the Adam optimizer Regarding Critic network parameters Gradient descent updates are performed to allow the value function to simultaneously fit dual supervision signals from real-world interactions and model-predicted trajectories.

[0034] Step 4.3: Actor network parameter update, parameters of the neural cost mapping network. The PPO-clip objective function is used for updates, and the gradient of the policy log probability is backpropagated through the analytical derivative chain of the DiffMPC solver:

[0035] For each sampling action Calculate the log probability under the current strategy. ;

[0036] During backpropagation, by controlling the mean Neural network parameters gradient Analytical calculations are performed, and the actions are sampled according to a Gaussian policy during the training phase, as follows: ; in, Indicates the exploration of the covariance matrix. For differentiable model predictive controller solvers, The parameters represent the probability distribution of a Gaussian policy. This represents a multivariate Gaussian probability distribution. Represented as the mean vector of the action distribution, it is The implicit function represents the learnable parameters of the neural cost mapping network.

[0037] Step 4.4: After each complete training cycle, explore the covariance matrix. according to By employing exponential decay, the randomness of the policy's actions is gradually reduced, prompting the agent to transition from breadth-based exploration to precision-based utilization.

[0038] Preferably, the Critic network employs a joint update mechanism, with a total value loss function. This includes: a temporal difference loss term on the true trajectory and an MPVE auxiliary loss term on the predicted trajectory. The MPVE auxiliary loss term is expressed as: ; in, and Let these represent the t-th step state and the k-th step reward predicted by the model from the current state, respectively. This is a value estimate of the state by the Critic network; the MPVE auxiliary loss term is compared with the true trajectory. Loss terms by weight Joint optimization.

[0039] Preferably, the end-to-end optimization and deployment of the policy network is performed as follows:

[0040] During the policy optimization phase, the Actor network is updated based on the advantage function estimated by the Critic. Based on the differentiability of the differentiable MPC module, the gradient is propagated to the cost mapping network at the Actor end via the MPC to achieve joint optimization of policy and prediction. The advantage function includes, but is not limited to, the generalized advantage estimation (GAE) in PPO.

[0041] During the deployment phase, the first control action output by MPC is adopted. As an execution instruction, rolling time-domain model predictive control is performed: in each control cycle, based on the current system state... The trained diffMPC solver is invoked in real time to generate the optimal action sequence in the finite time domain, and only the first action is applied. The process is repeated on the controlled object in the next cycle, forming a closed-loop feedback control.

[0042] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0043] 1. Significantly improved sample efficiency: This invention, through the Model Prediction Extension (MPVE) mechanism, enables the Critic network to utilize the predicted trajectory information generated by MPC during training, which significantly improves sample efficiency, effectively alleviates the long-term credit allocation difficulty in reinforcement learning, and improves training performance while reducing the need for interactive data.

[0044] 2. Reduced sensitivity to λ parameter and more stable training: This invention effectively alleviates the problem of value estimation bias and training instability when the value of λ is small by providing multi-step rewards through MPC prediction. It can still maintain a stable policy convergence process under low λ value conditions, reduce the sensitivity of the algorithm to the λ parameter, and broaden the adjustable range in practical application scenarios.

[0045] 3. More stable value estimation and mitigation of distribution bias problem: The Critic network of this invention, by introducing D(k) loss, can jointly utilize the multi-step reward information of real interaction trajectories and MPC predicted trajectories, thereby mitigating the negative impact of data distribution mismatch during training.

[0046] 4. Reduced resource consumption and high performance in complex control tasks: The method of this invention can achieve higher final cumulative rewards in high-dynamic UAV flight and complex trajectory tracking tasks, and shows a strong convergence trend in the early stage of training. It achieves higher training rewards and stronger policy robustness under the same sample budget, showing good generalization ability. At the same time, the first control action output by MPC is directly used as the execution instruction in the deployment stage, avoiding the dependence of traditional hierarchical control architecture on additional trajectory tracking modules. It can run on control platforms with high real-time requirements and is suitable for embedded deployment needs.

[0047] 5. Overall technical and social effects: By improving sample efficiency, this invention significantly reduces the need for interaction in simulation and real environments, thereby reducing equipment energy consumption and hardware wear, enhancing training stability, and reducing the cost of repeated manual parameter tuning; it can provide a more reliable solution for autonomous flight of UAVs, robot operation and other intelligent control applications, and helps to promote the application of reinforcement learning and MPC technology in engineering scenarios. Attached Figure Description

[0048] Figure 1 This is a flowchart of the Actor-Critic reinforcement learning control method integrating a differentiable MPC module according to the present invention.

[0049] Figure 2 This is a framework diagram of the Actor-Critic reinforcement learning control system integrating a differentiable MPC module according to the present invention.

[0050] Figure 3 This is a flowchart of the Critic training based on MPC predicted trajectory according to the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0052] In one embodiment of the present invention, an innovative control method combining model predictive control (MPC) and the deep reinforcement learning Actor-Critic framework is proposed. By introducing the model predictive value extension (MPVE) mechanism, the efficient integration of reinforcement learning and MPC is achieved.

[0053] In this embodiment, the Actor-Critic reinforcement learning control system framework integrating a differentiable MPC module is as follows: Figure 2 As shown, it includes: a state acquisition module, an Actor network, a differentiable MPC module, an environment / controlled object, a Critic network, and an experience playback pool and prediction storage unit, intuitively demonstrating the embedding position and role of the differentiable MPC module in the Actor-Critic reinforcement learning control system.

[0054] The specific workflow is as follows: The state acquisition module inputs the environmental state into the Actor network, and the cost parameters output by the Actor are passed to the differentiable MPC module. The differentiable MPC module generates a predicted action sequence and outputs the first control action to the environment. The environment provides feedback on the next state and reward, and writes the data into the storage unit. The Critic network is trained and updated based on the stored data and predicted trajectory.

[0055] In this embodiment, the process of training the Critic network based on the predicted trajectory of the differentiable MPC module is as follows: Figure 3 As shown, the MPVE mechanism plays a role in the training of the Critic network.

[0056] The specific workflow is as follows: The differentiable MPC module generates the predicted state, action, and reward for the next H steps through the predicted trajectory generation unit; the predicted reward is discounted and accumulated through the predicted reward accumulation unit; the predicted endpoint value is estimated through the endpoint state value estimation unit; the results of the discount accumulation and value estimation are combined in the multi-step reward construction module to generate the MPVE reward objective, which is then passed to the Critic update unit. The Critic update unit trains the value function based on this objective to obtain the final trained value function.

[0057] Furthermore, the value function training method based on model prediction extension in this embodiment includes the following steps:

[0058] 1. Construct an Actor-Critic reinforcement learning control system with an integrated differentiable MPC module.

[0059] The overall framework and process of the Actor-Critic reinforcement learning control system in this embodiment are as follows: Figure 1 As shown, differentiable model predictive control (DiffMPC) is integrated as the terminal module of the Actor policy network, fully utilizing the policy optimization capabilities of Actor-Critic reinforcement learning while maintaining the robustness of MPC based on model prediction. Environmental control commands are obtained by solving the MPC optimization problem, a process that effectively utilizes system dynamics to generate the optimal control sequence based on the current state.

[0060] (1) Actor Network: The Collaborative Working Mechanism of Neural Cost Mapping and Differentiable MPC

[0061] In this embodiment, the Actor network is derived from a neural cost mapping. This is achieved by taking normalized observations as input. The output is the quadratic cost function parameters required by DiffMPC, including: the diagonalized cost matrix. With linear term vectors (Diagonalization reduces degrees of freedom and ensures numerical stability), the quadratic cost function takes the form: ; in, , Representing time steps The corresponding system state vector and control input vector, superscript Indicates transpose. Indicates the prediction time domain length of MPC; parameters , , Indicates at time step The corresponding state-action joint quadratic cost matrix at that time. Indicates at time step The corresponding linear cost vector at that time, These represent the dimensions of the system state vector and the control input vector, respectively.

[0062] The constraints are: ; ; in, This represents the initial state of the MPC optimization problem. This represents the current real-time system state obtained from the environment or sensors. This represents the feasible set of control actions.

[0063] In this embodiment, Constrained to be a diagonal matrix, All values ​​are output by a neural cost mapping network. To ensure numerical stability and positive definiteness, the network output is mapped via a sigmoid function and then affined to the interval. , These represent the minimum and maximum allowable values ​​of the diagonal weights and linear coefficients of the quadratic cost function output by the neural cost mapping network, respectively.

[0064] As a preferred embodiment, in this example, .

[0065] Furthermore, at any moment Define the extended vector ( DiffMPC in the prediction time domain The internal solution is the following convex quadratic programming problem: ; in, This represents the relative time step index within the prediction time domain. ; In order to be in nonlinear dynamics The first-order linearization result.

[0066] It is particularly important to note that in the design of time-domain parameter stability, the cost parameter... It remains unchanged throughout the entire prediction time domain (i.e., it depends only on the current observation). Instead of updating with each prediction step, this significantly reduces the policy search space and improves training stability.

[0067] In addition, the DiffMPC solver needs to support differentiability (e.g., through KKT conditions or implicit function theorems) to control the mean. Neural network parameters gradient It can be parsed.

[0068] Finally, the actions are sampled according to the Gaussian policy during the training phase: .

[0069] As a preferred option, explore covariance Initially set as a scalar (e.g.) During training, press Attenuation, set during deployment .

[0070] (2) Critic Network: Value Estimation and MPVE Multi-Step Return Construction

[0071] In this embodiment, the value function is based on the Critic network. Estimating state value, MPVE (Model Predictive Value Expansion) utilizes DiffMP in the current state. The generated internal predicted trajectory and corresponding predicted rewards (Calculated by the task reward function on the predicted state), and transformed into a predicted observation sequence through observation mapping. Construct a multi-step reward objective: ; in, Discount factor (usually taken as) ).

[0072] Furthermore, construct a multi-step reward objective:

[0073] n-step reward (used for TD(λ) loss calculation): ; in, Indicates from time step Starting, based on the future The n-step reward target value constructed from real-world interaction data. Indicates the discount factor. This indicates that the agent is at time step The reward function obtained from interacting with the environment. Indicates the state of the Critic network. State value function estimate, express The observation vector at time t.

[0074] λ-reward (truncated form): ; in, This represents the return smoothing factor. This represents the exponentially decaying weight of the return at step n. This indicates the maximum number of unfolding steps. Indicates from time step Rising Weighted mixed return estimate.

[0075] MPVE step-by-step objectives (for each point in the predicted trajectory): ; in, This indicates that the trajectory is predicted based on the model at time step. The predicted reward obtained at that time This represents the Critic network's value estimation function for the input state. Indicates starting from the current state Departure The observation vector corresponding to the terminal state obtained from the step prediction. Indicates time step From the predicted trajectory of time step The initial model predicted the target value for extended returns.

[0076] Critic joint loss: ; in, This represents the weighting coefficient of the MPVE auxiliary loss. Represents the true trajectory The time-series difference loss term on, This represents the MPVE auxiliary loss term on the predicted trajectory.

[0077] This embodiment enables the Critic network to learn from both real-world interaction data and model-predicted trajectories simultaneously, significantly improving sample efficiency and value estimation accuracy.

[0078] 2. Observation and Action Space.

[0079] (1) Definition and preprocessing of observation vectors

[0080] In this embodiment, for the typical task of a drone passing through a door frame, the observation vector is defined as: ; Among them, the observation of the body's condition is The linear velocity in the body coordinate system is expressed as The pose is represented by a 9-dimensional rotation matrix vectorization. .

[0081] Trajectory-guided observation is as follows: ; in, It represents the relative positions of the four corner points of the current door and the next door in the machine coordinate system (each corner point 3D→4×3=12).

[0082] All observations are normalized online during training to ensure the numerical stability of the network input.

[0083] (2) Definition of action vector and physical constraints

[0084] In this embodiment, the action space is: ;in, It is the mass-normalized total thrust; It is the body angular velocity command.

[0085] These physical constraints are treated as inequality constraints in DiffMPC. Explicit processing ensures that the generated control commands comply with the system's physical limitations.

[0086] 3. Reward function and training and parameter update mechanism

[0087] (1) Reward function

[0088] This embodiment uses a sparse gate progress reward, represented as: ; These correspond to the following situations: collision (position outside the track boundary or contact with the door frame), passing through the current door (entering the door area from the center of the machine), completing the track (passing through all 5 doors), and other situations.

[0089] In the formula, For time step The current target gate center coordinates corresponding to the UAV. For time step The body position at that time It is a volume velocity penalty coefficient used to suppress maneuver instability caused by excessive angular velocity in the control input; For drones in time step The body's angular velocity vector at time t is composed of the body's three angular velocity components, expressed as: ; in, The drone at time step Time-looping body Angular velocity of the axis.

[0090] This design allows reinforcement learning to directly optimize non-differentiable, sparse rewards, overcoming the limitation of traditional MPC which requires manually designing dense, differentiable costs.

[0091] (2) Training process and parameter update mechanism

[0092] 1) Forward sampling and data interaction: In this embodiment, before each round of training begins, the environmental state is reset, including the drone's position, attitude, linear velocity and angular velocity, as well as wind field conditions, external disturbances and noise variables; at the same time, the internal states of the Actor network and the Critic network are cleared to ensure the independence and repeatability of the sampling trajectories in each round.

[0093] Then, based on the current strategy Interact with simulation environments or real physical systems to execute several complete trajectories or rollouts of fixed duration.

[0094] At each time step The intelligent agent performs actions Then, the obtained transfer samples Stored in the experience replay buffer, where, For the current observation value, For immediate rewards for environmental feedback, This is the next state to enter after performing an action.

[0095] The data storage unit is a rolling buffer, which is a fixed-capacity circular queue used to store state transition samples in chronological order. When the buffer is full, the earliest stored data will be overwritten in a first-in-first-out manner to ensure that the sample batches used by the Critic network and Actor network during training always contain the latest interaction data.

[0096] 2) Critic network parameter update:

[0097] Mini-batch trajectory fragments are sampled from the experience replay buffer; Critic network parameters. Updates are performed by minimizing the joint supervision objective, including:

[0098] True trajectory branch: Calculate truncated segments based on sampled fragments Return , build Loss items;

[0099] Predicted trajectory branch: Extracted from the predicted trajectory calculated by the DiffMPC solver at the corresponding time step. and By observing the mapping function Transform the predicted state into a predicted observation. For each predicted observation Constructing a local multi-step reward objective ;

[0100] Furthermore, the two losses are weighted according to a preset formula. (e.g., 0.5) Weighted summation to form the total loss. .

[0101] The Adam optimizer is used to perform multiple rounds (e.g., 10 rounds) of mini-batch gradient descent on the loss to fully fit the dual value signals from the real and model predictions, thereby improving the generalization ability and prediction consistency of the value function.

[0102] 3) Actor network parameter update

[0103] Parameters of the Actor network (i.e., neural cost mapping) The PPO-clip objective function is used for updates. The gradient of the policy log probability is backpropagated through the analytical derivative chain of the DiffMPC solver:

[0104] For each sampling action First, calculate its log probability under the current policy. This probability depends on the mean of the DiffMPC output. .because yes The implicit function is obtained by automatically calling the gradient interface built into the DiffMPC module (e.g., Jacobian calculation based on the implicit function theorem) during backpropagation. And thus calculate accurately .

[0105] In this embodiment, the mechanism ensures that the policy gradient directly responds to the feedback signal of the value function, and even in scenarios where task rewards are sparse and discontinuous, stable and efficient policy improvement can be achieved through value guidance.

[0106] During the update process, explore the noise covariance. Press after each complete training cycle The process of attenuation gradually shifts from exploration to utilization.

[0107] (3) Multi-scale MPVE fitting mechanism

[0108] To enhance the adaptability of the Critic network to dynamics at different time scales, this embodiment introduces multi-scale MPVE fitting, as follows:

[0109] Multiple MPVE targets are constructed in parallel, corresponding to different prediction step sizes. } Calculate the MPVE loss for each step size separately, and incorporate it into the total loss by weighted summation.

[0110] This method can combine different n-step TD returns (e.g., n=3, 5, 10) with MPVE objectives to form a hybrid multi-step value estimate.

[0111] The predicted trajectories reuse the same DiffMPC solution, requiring no additional model calls and keeping computational overhead manageable. This design enables the value function to simultaneously capture both short-term precise dynamics and long-term task structure, effectively improving the policy's generalization ability and task completion rate in complex environments.

[0112] In summary, this invention significantly improves the performance of reinforcement learning and model predictive control (MPC) in several aspects by introducing a value function training mechanism based on model prediction extension (MPVE).

[0113] First, this invention utilizes the Model Prediction Extension (MPVE) mechanism to enable the Critic network to leverage predicted trajectory information generated by MPC during training, thereby significantly improving sample efficiency. Comparative experiments in highly dynamic flight tasks (such as Split-S trajectory tracking) demonstrate that, with the same interaction sample budget, the MPVE method achieves higher cumulative rewards in fewer training steps, and its learning convergence speed is significantly faster than the baseline AC-MLP algorithm. This indicates that the MPVE mechanism of this invention effectively alleviates the long-term credit allocation difficulty in reinforcement learning, improving training performance while reducing the need for interaction data.

[0114] Furthermore, in experiments with different discount parameters λ, the method of this invention exhibits stronger stability than the traditional TD(λ) method, enabling a stable policy convergence process even under low λ values. As λ increases (e.g., λ≈0.6), although the difference between the two methods decreases, the method of this invention still outperforms the baseline method in terms of convergence speed and training stability, reducing the algorithm's sensitivity to the λ parameter and broadening the adjustable range in practical application scenarios.

[0115] Furthermore, by introducing the TD(k) technique, the Critic network of this invention can jointly utilize multi-step reward information from real interaction trajectories and MPC predicted trajectories, thereby mitigating the negative impact of data distribution mismatch during training. Comparative experimental results show that this method can effectively reduce the instability of value estimation, making the training curve smoother and the overall convergence process more stable. Especially in high-dimensional continuous control tasks, the Critic network using the method of this invention exhibits significantly less fluctuation in the estimation of the value function than the baseline method.

[0116] Furthermore, in highly dynamic UAV flight and complex trajectory tracking tasks, this invention demonstrates superior performance compared to baseline methods. For example, in the Split-S flight task, the policy using this invention achieves a higher final cumulative reward and exhibits a strong convergence trend early in training. In addition, in other robot control tasks (such as Ant and Walker2d in simulated environments), this invention also achieves higher training rewards and stronger policy robustness with the same sample budget, demonstrating good generalization ability.

[0117] Furthermore, during the deployment phase, this invention directly uses the first control action output by the MPC as the execution instruction, avoiding the dependence of traditional hierarchical control architectures on additional trajectory tracking modules. Experimental analysis shows that the computational overhead of this method during the inference phase is only slightly increased compared to the baseline method without MPC, but far lower than that of traditional sampling-based MPC control methods. Simultaneously, since the MPC module is differentiable and has a finite prediction time domain, this method can run on control platforms with high real-time requirements, adapting to embedded deployment needs.

[0118] In terms of overall technical and social impact, this invention significantly reduces the need for interaction in simulation and real environments by improving sample efficiency, thereby reducing equipment energy consumption and hardware wear; it reduces the sensitivity of the λ parameter and alleviates the distribution offset problem; it reduces the cost of repeated manual parameter tuning by enhancing training stability; and it provides a more reliable solution for autonomous flight of UAVs, robot operation and other intelligent control applications by improving control performance in complex tasks, which helps to promote the application of reinforcement learning and MPC technology in engineering scenarios.

[0119] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A value function training method based on model prediction extension, characterized in that, Includes the following steps: Step 1: Construct the Actor-Critic reinforcement learning framework. Integrate a differentiable MPC module at the end of the Actor network. The state acquisition module inputs the environmental state into the Actor network, outputs the cost parameters through the neural cost mapping network, and passes them to the differentiable MPC module to generate a predicted action sequence and output the first control action to the environment. The environment provides feedback on the next state and reward and writes the data into the storage unit. Step 2: In each control iteration, the differentiable MPC module solves the convex quadratic programming problem in the prediction time domain based on the current state, generating the predicted future state, action and reward, which is then passed to the Critic network. Step 3: The Critic network performs value estimation and constructs the MPVE multi-step reward objective based on the data and predicted trajectory of the storage unit, while learning from real interaction data and model predicted trajectory. Step 4: Construct a reward function, perform forward sampling and data interaction, complete the training and parameter update of the Critic network and Actor network, obtain the trained value function, and generate the internal predicted trajectory and corresponding predicted reward in the current state.

2. The value function training method based on model prediction extension according to claim 1, characterized in that, The Actor network described in step 1 takes the normalized observation vector as input. After passing through a neural cost mapping network Output the cost parameters of the quadratic cost function, including: the angular cost matrix. and linear term vectors ; The quadratic cost function is in the form of: ； The constraints are: ； ； in, , Representing time steps The corresponding system state vector and control input vector, superscript Indicates transpose. Indicates the total length of the prediction time domain; This represents the initial state of the predicted sequence. This indicates the actual state of the system at the current moment. This represents the physically feasible set of control inputs.

3. The value function training method based on model prediction extension according to claim 2, characterized in that, The observation vector Represented as: , For the observation of the organism's condition, To guide observations based on the trajectory; The action space Represented as: , For mass-normalized total thrust; This is the angular velocity command for the machine.

4. The value function training method based on model prediction extension according to claim 2, characterized in that, In step 2, at time Define the extended vector The differentiable MPC module in the prediction time domain The problem of solving a convex quadratic programming problem internally is expressed as: ； in, This represents the relative time step index within the prediction time domain. ; In order to be in For nonlinear dynamic models The first-order linearization result, the extended vector .

5. The value function training method based on model prediction extension according to claim 2, characterized in that, In step 2, the differentiable MPC module, based on the current observation vector... Prediction time domain is Future state-action sequence: This includes the following sub-steps: Step 2.1: The neural cost mapping network calculates the current normalized observation vector. Output cost function parameters and ; Based on the stability of time-domain parameters, the cost parameter It remains unchanged throughout the entire prediction time domain; Step 2.2, in the current system state At this point, a first-order linearization is performed using a nonlinear dynamic model to obtain a time-invariant linear approximation model: ； Step 2.3, with Solve the convex quadratic programming problem for the initial state to obtain the optimal control sequence. By combining the linear model with forward recursion, the corresponding predicted state sequence is generated. And through the observation mapping function , converted into predicted observation sequence Form a complete predicted trajectory .

6. The value function training method based on model prediction extension according to claim 2, characterized in that, In step 3, the MPVE multi-step reward target is constructed as follows: Construct an n-step reward for the real trajectory Loss calculation: ； in, Indicates from time step Starting, based on the future The n-step reward target value constructed from real-world interaction data. Indicates the discount factor. This indicates that the agent is at time step Instant rewards gained from interacting with the environment Indicates the state of the Critic network. State value function estimate, express The observation vector at time; Build -Returns, expressed in truncated form: ； in, This represents the return smoothing factor. This represents the exponentially decaying weight of the return at step n. This indicates the maximum number of steps that can be expanded. Indicates from time step Rising Weighted mixed return estimate; For each point in the predicted trajectory, set the MPVE step-by-step objective: ； in, This indicates that the trajectory is predicted based on the model at time step The predicted reward obtained at that time This represents the Critic network's value estimation function for the input state. Indicates starting from the current state Departure The observation vector corresponding to the terminal state obtained from the step prediction. Indicates time step From the predicted trajectory of time step The initial model predicted the target value for expanded returns; Construct a joint loss for the Critic network and build a multi-step reward objective by predicting trajectories: ； in, Indicates the time from the current moment to the nth The total discount factor for each step This is Critic's estimate of the predicted endpoint state.

7. The value function training method based on model prediction extension according to claim 1, characterized in that, In step 4, the reward function, using a sparse gate progress reward, is expressed as: ； in, For time step The current target gate center coordinates corresponding to the UAV. For time step The body position at that time For volume velocity penalty coefficient, For drones in time step The body's angular velocity vector at that time is composed of the body's three angular velocity components.

8. The value function training method based on model prediction extension according to claim 1, characterized in that, In step 4, the training and parameter updates of the Critic network and Actor network are performed as follows: Step 4.1, Forward Sampling and Data Interaction: Based on the current strategy Interact with the simulation environment or the real physical system at each time step The intelligent agent performs actions Then, the obtained transfer samples Stored in the experience replay buffer, where, For immediate rewards for environmental feedback, The next state to enter after performing an action; Step 4.2, Critic network parameter update: The Critic network samples a trajectory segment from the experience replay buffer, synchronously calls the DiffMPC solver, and applies the current state... Generate the corresponding predicted state sequence and predicting reward sequences ; True trajectory branch: Calculate truncated segments based on sampled fragments -Rewards Construct the temporal difference loss term: ; Predicted trajectory branch: Extracted from the predicted trajectory calculated by the DiffMPC solver at the corresponding time step. and For each predicted state Construct a local multi-step reward objective and build an MPVE reward error based on the predicted trajectory, then assign the two losses to preset non-negative weights. Weighted summation to form the total value loss function ; Use the Adam optimizer Regarding Critic network parameters Gradient descent is performed to update the value function so that it fits the dual supervision signals from real-world interactions and model-predicted trajectories. Step 4.3, Actor network parameter update: Neural cost mapping network parameters The objective function is updated using PPO-clip, and the gradient of the policy log probability is backpropagated through the analytical derivative chain of the DiffMPC solver for each sampling action. Calculate the log probability under the current strategy. ; During backpropagation, by controlling the mean Neural network parameters gradient The analysis is performed, and the actions are sampled according to the Gaussian strategy during the training phase; Step 4.4: After each complete training cycle, the covariance matrix will be explored and exponentially decayed to reduce the randomness of the strategy's actions.

9. The value function training method based on model prediction extension according to claim 8, characterized in that, The local multi-step reward objective is expressed as: ； The MPVE reward error based on the predicted trajectory is expressed as: ； The Critic network employs a joint update mechanism, with a total value loss function. Includes: actual trajectory The temporal difference loss term and the MPVE auxiliary loss term on the predicted trajectory are expressed as follows: ； in, This represents the weighting coefficient of the MPVE auxiliary loss. Represents the true trajectory The time-series difference loss term on, This represents the MPVE auxiliary loss term on the predicted trajectory.

10. The value function training method based on model prediction extension according to claim 1, characterized in that, The end-to-end optimization and deployment of the policy network is performed as follows: During the policy optimization phase, the Actor network is updated based on the advantage function estimated by the Critic. Based on the differentiability of the differentiable MPC module, the gradient is propagated to the cost mapping network at the Actor end via the MPC to achieve joint optimization of policy and prediction. The advantage function includes, but is not limited to, the generalized advantage estimation (GAE) in PPO. During the deployment phase, the first control action output by MPC is adopted. As an execution instruction, rolling time-domain model predictive control is performed: in each control cycle, based on the current system state... The trained diffMPC solver is invoked in real time to generate the optimal action sequence in the finite time domain, and only the first action is applied. The process is repeated on the controlled object in the next cycle, forming a closed-loop feedback control.

Images