A physically enhanced automatic driving control method and system with full-link differentiability

Abstract

The present disclosure relates to a physically enhanced automatic driving control method and system. The method comprises: constructing a full-link differentiable PERL dynamics model to reconstruct the real vehicle dynamics with high fidelity; establishing a differentiable preview PID execution layer, calculating the lateral error and longitudinal speed error based on the dynamic preview distance, and using the PID control law supporting gradient backpropagation to generate the steering wheel angle command and the vehicle longitudinal acceleration command; using the SAC network to adaptively adjust the PID gain parameters and the preview distance in real time according to the current vehicle state. The full-link differentiable closed-loop training architecture designed by the present application directly guides the end-to-end optimization of control parameters using the gradient information of the dynamics residual, effectively solving the Sim2Real model distortion problem, while ensuring the physical interpretability and smoothness of the control strategy, the trained strategy network significantly improves the path tracking accuracy and robustness of the vehicle in complex working conditions.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving and intelligent control technology, specifically to a fully differentiable physical augmented control method and system, which executes the decision-making process in real time through the vehicle control system and actuates the required vehicle subsystems to jointly perform dynamic driving tasks. Background Technology

[0002] With the development of autonomous driving technology, simulation-based algorithm verification has become a crucial part of system development. However, in the process of transferring control strategies trained in simulation environments to real vehicles, challenges such as model distortion and insufficient control robustness remain constant.

[0003] The existing technologies mainly have the following two types of solutions, but both have the following obvious limitations.

[0004] The first category is control methods based on traditional physical models. Existing solutions typically employ idealized single-track or dual-track dynamic models combined with PID or MPC controllers. However, idealized models struggle to accurately describe unmodeled dynamics (such as mechanical transmission backlash and actuator response delay) and environmental uncertainties present in real vehicles (such as the nonlinear friction characteristics between tires and the ground, and microscopic road surface irregularities). This "model-reality gap" leads to control parameters that perform well in simulations often exhibiting large tracking errors or even system instability in real vehicles. Furthermore, the parameters of traditional PID controllers ( The aiming distance is usually a fixed value or a static value based on a lookup table, which cannot meet the dynamic adjustment needs of the vehicle under extreme conditions.

[0005] The second category is data-driven end-to-end intelligent control methods. Existing solutions mainly utilize deep reinforcement learning (such as SAC, DDPG) or imitation learning algorithms to directly construct a mapping from sensor data to control commands (such as steering wheel angle, acceleration). Although such methods possess strong nonlinear fitting and adaptive capabilities, they have significant drawbacks:

[0006] Lack of physical constraints and interpretability: The control commands output by pure neural networks lack dynamic constraints, which can easily generate high-frequency jitter. This not only causes wear on the actuators but also fails to meet the requirements for ride comfort. In the event of an unfamiliar working condition, it is very easy to output abnormal commands, which poses a safety hazard.

[0007] Low training efficiency and difficulty in convergence: Traditional model-free reinforcement learning usually updates the policy only through sparse reward signals, treating the environment as a "black box" and failing to utilize the differential information of the dynamic model to guide the optimization direction, resulting in low sampling efficiency and a long and unstable training process.

[0008] Sim2Real has poor generalization ability: because pure data-driven models tend to overfit the characteristics of the simulation environment, the performance of the strategy will drop sharply when the physical parameters of the real world (such as the road friction coefficient) are inconsistent with the simulation. Summary of the Invention

[0009] The purpose of this invention is to propose a fully differentiable physical-enhanced autonomous driving control method and system, which aims to solve the problems that often result in large tracking errors or even system instability in real vehicles due to the distortion of simulation models and the lack of physical constraints in pure end-to-end reinforcement learning in autonomous driving control.

[0010] Specifically, a fully differentiable physical augmentation (PAR) autonomous driving control method includes: training a fully differentiable PERL dynamics model, configured as consisting of a physical prior layer and a residual regression layer in parallel, trained by minimizing the predicted vehicle state and the actual vehicle state of the physical layer and the residual regression layer; freezing the parameters of the fully differentiable PERL dynamics model after training; pre-training a SAC (Sensitive Acceleration Control) policy network after freezing the parameters of the fully differentiable PERL dynamics model, configured to adaptively adjust the PID gain parameter value and the aiming distance in real time based on the vehicle state; and jointly fine-tuning the fully differentiable PERL dynamics model and the SAC policy network after the SAC policy network pre-training is completed, based on the current... The vehicle state and control inputs are determined by a fully differentiable PERL dynamics model, which predicts the vehicle state at the next moment. This next-moment vehicle state is input into the SAC (Self-Controlled Acceleration) policy network, which outputs the corresponding PID gain parameter value and aiming distance. The actuator calculates the lateral and longitudinal velocity errors based on the dynamic aiming distance. Based on these errors and gain parameter values, a PID control law supporting gradient backpropagation is used to generate control inputs. The autonomous vehicle updates its state under the influence of these control inputs. Fine-tuning the predicted error gradient during training utilizes the fully differentiable characteristics of the PERL dynamics model and the actuators, introducing direct physical gradients through a chain rule, and directly backpropagating them to the SAC policy network. During inference, the trained SAC network uses the PID gain parameter value and aiming distance generated based on the actual vehicle state to achieve autonomous driving control. In one embodiment, the control inputs consist of a steering wheel angle command and a longitudinal acceleration command. The steering wheel angle command is generated by the lateral control channel based on the lateral error and is used to actuate the vehicle's steering subsystem. The longitudinal acceleration command is generated by the longitudinal control channel based on the longitudinal velocity error and is used to actuate the vehicle's drive and braking subsystems. These two commands work together to achieve joint control of different functional vehicle subsystems. A computer-readable storage medium storing a computer program that can be loaded by a processor and execute any of the methods described in this disclosure.

[0011] A fully differentiable physical augmentation autonomous driving control system includes: a PERL dynamics model training module configured to train a fully differentiable PERL dynamics model, which consists of a physical prior layer and a residual regression layer in parallel, predicting the vehicle state at the next moment based on the current vehicle state and control variables; a SAC policy network training module configured to pre-train the SAC policy network after freezing the parameters of the fully differentiable PERL dynamics model, wherein the SAC policy network is configured to adaptively adjust the PID gain parameter values ​​and the aiming distance in real time based on the input vehicle state; and a joint fine-tuning module configured to jointly fine-tune the fully differentiable PERL dynamics model and the SAC policy network after the SAC policy network pre-training is completed, fine-tuning the SAC policy network in a micro-process. During training, the current vehicle state is used as the input to the fully differentiable PERL dynamics model, and the vehicle state at the next moment is input to the SAC policy network. The SAC policy network outputs the corresponding PID gain parameter value and the aiming distance. The actuator calculates the lateral error and longitudinal velocity error based on the dynamic aiming distance. Based on the above errors and gain parameter values, the PID control law supporting gradient backpropagation is used to generate control quantities. The autonomous vehicle updates its state under the action of the control quantities. The prediction error gradient in the fine-tuning training utilizes the fully differentiable characteristics of the PERL dynamics model and the actuator to introduce the direct physical gradient through the chain rule, which is directly backpropagated to the SAC policy network. During inference, the PID gain parameter value and aiming distance generated by the trained SAC network based on the actual vehicle state are used to achieve autonomous driving control.

[0012] Beneficial technical effects: This invention achieves adaptive and smooth optimization of high-fidelity Sim2Real and control strategies by constructing a full-link differentiable gradient transmission architecture. Furthermore, the system coordinates the output of steering wheel angle and longitudinal acceleration commands, realizing joint control of vehicle steering, drive, and braking subsystems, significantly improving path tracking accuracy, control robustness, and model convergence speed under complex working conditions. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the overall architecture of a physically enhanced autonomous driving control system that is differentiable across the entire chain. Detailed Implementation

[0014] As the background information suggests, there is an urgent need for a control architecture that can both faithfully reproduce the dynamic characteristics of the real physical world and combine the physical interpretability of PID control with the adaptive capabilities of reinforcement learning. In particular, how to construct a fully differentiable closed-loop system and use the gradient of the dynamic residual to directly guide the dynamic optimization of control parameters to solve the problems of Sim2Real transfer difficulties, unsmooth control, and low training efficiency is a technical challenge that urgently needs to be addressed in this field.

[0015] Based on this, the present invention proposes a full-link differentiable intelligent driving control architecture, which consists of a "PERL dynamics reconstruction module", a "differentiable PID control module" and a "SAC algorithm and strategy optimization module". Through the collaborative work of each part, the simulation environment is accurately mapped to the real physical world, and an intelligent control strategy with physical interpretability and robustness is trained based on this environment.

[0016] The following will be combined with the appendix Figure 1 This application provides a clear and complete description of how the technical solution in this case is implemented. Obviously, the described implementation methods are only a part of the implementation methods in this case, not all of them. Based on the implementation methods in this case, all other implementation methods obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0017] (I) PERL Dynamics Reconstruction Module

[0018] This invention first constructs a simulation environment model that can simulate real vehicle characteristics with high fidelity and supports gradient backpropagation—a fully differentiable PERL dynamics model. Its main function is to guide the subsequent differentiable PID control module and SAC algorithm and strategy optimization module to learn the correct real-world dynamics and thus output ideal actions during the model training phase by reconstructing the simulation environment.

[0019] The PERL dynamics model consists of a physical prior layer and a residual regression layer, which are constructed in parallel. Their outputs are superimposed and fused at the end of each time step. By using the physical prior layer to capture the nominal motion laws and the residual regression layer to fit the unmodeled dynamics and environmental disturbances, a high-fidelity reconstruction of the real vehicle dynamics is achieved through linear superposition.

[0020] (1.1) Physical Prior Layer

[0021] The physical prior layer, based on classical Newtonian mechanics, calculates the vehicle's motion trend under ideal conditions, capturing the main low-frequency and large-amplitude laws of vehicle dynamics. Specifically, a three-degree-of-freedom (3-DOF) single-track dynamics model is selected. This model ignores the vehicle's roll and pitch motions, focusing on the longitudinal, lateral, and yaw motions, which are most critical for tracking planar paths.

[0022] Input vector: Current state of the vehicle With control actions .in, and The coordinates of the vehicle's center of gravity. For the vehicle's heading angle, For the longitudinal speed of the vehicle, For the vehicle's lateral speed, Let yaw rate be the vehicle's angular velocity.

[0023] Control quantity = ,in, This is a steering angle command. This is a longitudinal acceleration command.

[0024] Physical parameter P: includes vehicle mass m and moment of inertia Front and rear wheelbase Nominal lateral stiffness of front and rear wheels .

[0025] The physical model in the physical prior layer is based on the Newton-Euler equations to calculate vehicle state variables. micro-components The specific calculation process is as follows.

[0026] Tire slip angle calculation: .

[0027] Nominal tire force calculation: , .

[0028] The dynamic differential equation is shown below.

[0029]

[0030] The kinematic differential equations are shown below.

[0031]

[0032] Physical layer output: The nominal state at the next time step is obtained through Euler integration. Figure 1 (Next state in the process) , The interval between adjacent time points.

[0033] (1.2) Residual Regression Layer

[0034] The Residual Regression Layer (RRL) is a parallel deep neural network designed to fit unmodeled dynamics and time-varying parameter disturbances that the physical layer cannot describe. Specifically, these include: the nonlinear saturation characteristics of tires under extreme conditions (the failure region of the linear assumption of the physical layer), random disturbances caused by microscopic unevenness of the road surface, and mechanical backlash and response delay of actuators (steering gear, brakes).

[0035] The network architecture is designed using a fully connected multilayer perceptron (MLP) structure as the residual network, with the following specific configuration.

[0036] Input layer: Receives a concatenated vector of vehicle state and control input from the physical prior layer. And perform layer normalization to accelerate convergence.

[0037] Hidden layers: Three hidden layers are set up for example, with 128, 64, and 32 neurons respectively. The activation function is ReLU, which introduces non-linear features while ensuring effective gradient propagation in the deep network and avoiding gradient vanishing.

[0038] Output layer: Linear output layer, output dimension is the same as the vehicle state vector. The dimensions are consistent (i.e., 6-dimensional), directly corresponding to the correction values ​​of the state variables.

[0039] Forward computation process: Let the network weights be... The input is The calculation process of RRL can then be represented as nested matrix operations:

[0040]

[0041] in, This is the activation function. ， ， These are the output vectors of the first, second, and third hidden layers. 、 、 Here are the weight matrices for the first, second, and third hidden layers. , and These are the bias vectors for the first, second, and third hidden layers. The weight matrix of the output layer. This is the bias vector for the output layer. The vehicle state correction amount is the output of the residual regression layer prediction, and is related to the vehicle state. Same dimensions.

[0042] (1.3) Hybrid output reconstruction

[0043] The nominal predictions from the physical prior layer are superimposed with the residual corrections from the RRL layer to generate the final vehicle state prediction. This process is linear in computation and fully preserves the differentiability of the model.

[0044] Calculation formula:

[0045] It provides the correct direction and approximate range of vehicle movement, ensuring that the model will not produce outrageous predictions that violate common sense in areas with scarce data (areas not covered by the neural network). It dynamically compensates for the complexities of the real world. For example, when a vehicle skids while cornering at high speed, the physical layer calculates... The output may be smaller (based on the linear tire assumption), while the RRL layer will output a larger one. Compensation shall be provided to make It can accurately reflect the actual degree of sideslip.

[0046] The overall structure constitutes a "gray box" model, possessing both the interpretability of a white-box physical model and the fitting accuracy of a black-box neural network. This approach proposes a residual learning paradigm based on physical confidence constraints. By introducing a residual energy regularization term into the loss function, the neural network is forced to fit only unmodeled dynamic features, thus preserving the interpretability and boundary safety of the nominal physical model while maintaining the Sim2Real transfer accuracy.

[0047] During training, the following loss function is minimized. To update network weights .

[0048]

[0049] in, This represents the real-world label data at time t+1. This represents the nominal state prediction value of the physical prior layer at time t+1. This represents the correction value for the state variables in the predicted output of the residual regression layer. This represents the regularization coefficient of the physical confidence constraint term.

[0050] The loss function with physical constraint regularization ensures that the RRL layer only fits the "residuals" not covered by the physical model, rather than completely replacing the physical model. The prediction error term measures the output of the hybrid model (physical + residuals) relative to real-world data. The Euclidean distance between them drives the network to approximate the real dynamic characteristics. The physical confidence constraint term is then implemented by introducing a regularization coefficient. This penalty applies to the magnitude of the residual output. This forceful the network to output close to zero in regions where the physics model can accurately predict (e.g., low-speed straight-line driving); only in regions where the physics model fails (e.g., high-speed sideslip) is the network allowed to output a larger correction. This ensures the model's robustness in Sim2Real transfer learning.

[0051] Unlike traditional black-box models, the RRL layer in this invention is fully differentiable. In the subsequent SAC policy optimization step, this layer not only predicts the state but also performs gradient propagation. It allows the calculation of the partial derivatives of the next-time state with respect to the control input. This enables reinforcement learning algorithms (SAC) to directly compute policy gradients using the chain rule, achieving model-based end-to-end policy optimization.

[0052] (ii) Differentiable PID control module

[0053] This invention constructs a fully differentiable actuator module, aiming to transform the abstract parameters generated by the upper-level SAC algorithm and strategy optimization module into specific vehicle control commands. Unlike traditional PID, this module softens and smooths all operators in the preview search, error calculation, and control law generation processes, constructing a control output... To input parameters The complete gradient path makes It can be calculated precisely.

[0054] (2.1) Input Definition

[0055] Current vehicle status = With control actions .in, and The coordinates of the vehicle's center of gravity. For the vehicle's heading angle, For the longitudinal speed of the vehicle, For the vehicle's lateral speed, The vehicle's yaw rate; This is a steering angle command. This is a longitudinal acceleration command.

[0056] Reference trajectory : Consists of a series of discrete path points and its corresponding tangential angle curvature Composition and reference speed composition.

[0057] Dynamic control parameters The SAC algorithm and strategy optimization module optimizes based on the vehicle's current state. The output PID parameters specifically include seven scalar parameters. These correspond to the proportional, integral, and derivative gain parameters for lateral control, and the proportional, integral, and derivative gain parameters and dynamic aiming distance for longitudinal control, respectively.

[0058] (2.2) Adaptive Pre-aiming

[0059] Traditional methods typically find the nearest matching point based on a fixed pre-target distance (using the argmin operation), which is non-differentiable and can lead to gradient breakage.

[0060] This invention proposes an adaptive pre-aiming method, which is achieved through dynamic pre-aiming and error calculation. The error calculation includes matching point localization based on soft attention and lateral error. The calculation is performed, and the non-differentiable problem is solved by matching point localization based on soft attention.

[0061] A. Dynamic pre-aiming

[0062] Based on the pre-aiming distance output by the SAC algorithm and the strategy optimization module Calculate the virtual preview point in the vehicle's forward field of view. : , Let be the coordinates of the vehicle's center of mass at the current moment. This is the vehicle's heading angle at the current moment.

[0063] Virtual aiming point right It is continuously differentiable.

[0064] Gradient path: This ensures that changes in the aiming distance can be directly transmitted to the position coordinates.

[0065] B. Matching point localization based on soft attention (solving the non-differentiability problem)

[0066] To replace the non-differentiable "nearest point search", a Gaussian kernel function based on Euclidean distance is used to calculate the attention weights of each point on the reference trajectory. .

[0067]

[0068] in This is the temperature coefficient. For the first reference trajectory A discrete path point, For the first reference trajectory There are discrete path points, and the value of j ranges from 0 to N (corresponding to the reference trajectory point set). N represents the total number of reference trajectory points.

[0069] The matching point is obtained by weighted summation. and its normal vector The expression is as follows.

[0070]

[0071]

[0072] In the above formula, For the reference trajectory, the first Attention weights corresponding to each path point For the reference trajectory, the first The normal vectors corresponding to each path point.

[0073] Physical meaning: When During movement, weight It will change smoothly, making the matching points The continuous sliding along the trajectory ensures the differentiability of the matching process.

[0074] C. Error Calculation

[0075] Error calculation includes calculating lateral error. and longitudinal velocity error The details are as follows.

[0076] The lateral error of the virtual gap point relative to the matching point is calculated using the following formula.

[0077]

[0078] because and All are The error is a differentiable function. Pre-aiming distance gradient It exists and its value is valid. This allows neural networks to automatically learn to "reduce speed before a curve" through gradient descent. To obtain more sensitive error feedback.

[0079] The following formula is used to calculate the deviation between the vehicle's current longitudinal speed and the expected speed at the reference trajectory matching point.

[0080]

[0081] in, The desired reference velocity at the matching point, obtained based on soft attention weights, can be calculated using the formula... Seeking, For the first Reference speed for each path point This represents the vehicle's current longitudinal speed.

[0082] (2.3) Differentiable PID Logic

[0083] Traditional PID controllers often include discrete state machine logic and hard truncation. To simulate the mechanical limits of a physical actuator while avoiding the "vanishing gradient" problem (i.e., the gradient is 0 in the cutoff region) caused by hard truncation, this module constructs a PID controller with Tanh, using pure algebraic calculations to generate continuously differentiable control commands.

[0084] First, based on the control error at the current moment By combining the integral and differential of historical errors, the mechanical limit value is calculated. Normalized PID internal control variables As shown below.

[0085]

[0086] in, , and To control the proportional, integral, and derivative gain parameters, This is the limit value.

[0087] Subsequently, the tanh function is used to smooth and limit the normalized variable and restore it to the actual physical dimensions to generate the final control command, as shown below.

[0088]

[0089] In practical autonomous driving control, the aforementioned general differentiable PID control law is instantiated as lateral control channels and longitudinal control channels, respectively. Specifically, when used for lateral control, the control command... Corresponding steering wheel angle commands (Right now ), corresponding to the steering system that controls the vehicle, limit value Mechanical limit value corresponding to steering wheel angle Controlling error Corresponding lateral error The corresponding gain parameter set is , , When used for longitudinal control, the control commands... Corresponding longitudinal acceleration command (Right now ), corresponding to the control of the vehicle's drive and braking subsystems, limit values Mechanical limit value corresponding to longitudinal acceleration Controlling error Corresponding longitudinal velocity error The corresponding gain parameter set is , , .

[0090] because The function retains a weak gradient even in the saturation region. This ensures that even when the control quantity reaches its limit, the SAC algorithm and policy optimization module can still receive gradient feedback to "reduce gain", avoiding getting trapped in local extrema, thereby achieving rapid convergence and oscillation suppression in parameter tuning.

[0091] (III) SAC Algorithm and Strategy Optimization

[0092] This solution utilizes the Flexible Actor-Critic (SAC) algorithm to construct the top-level decision-making module. Unlike traditional end-to-end control that directly outputs steering wheel angle and acceleration, this module acts as the driver, dynamically adjusting key parameters of the aforementioned differentiable PID controller in real time based on the current environmental conditions. These parameters include lateral and longitudinal PID gain and aiming distance. To achieve optimal path tracking performance.

[0093] (3.1) Network structure and action space

[0094] The policy network employs a Gaussian policy network constructed using a multilayer perceptron (MLP). The input is the current vehicle state. Output the mean of the action distribution and logarithmic standard deviation The strategy outputs a 7-dimensional normalized adjustment coefficient vector. The actual physical parameters input to the PID layer The following linear mapping calculations ensure that the parameters are within a physically reasonable range.

[0095]

[0096] For actual physical parameters The basic reference value vector, This is a vector representing the allowable adjustment range of the actual physical parameters. This represents the Hadamard product (i.e., the element-wise multiplication of two vectors).

[0097] Training is performed by randomly sampling from the experience replay pool. A reparameterization trick is used for sampling to preserve the differentiability of the gradients, providing a basis for subsequently receiving backpropagated gradients from the physical model.

[0098] The value network adopts a dual-Q network structure to evaluate (state, action) pairs. Its input is a concatenated vector of state and action. The evaluation result is the cumulative value of the current PID parameter configuration over a long time span, effectively eliminating the overestimation bias of Q value.

[0099] Based on this, the overall optimization process of the SAC algorithm consists of the total loss function. Driver. Among them, Specifically, it consists of two parts: first, the evaluation loss calculated by the value network based on the temporal difference (TD) error; and second, the policy loss calculated by the policy network based on the maximum entropy objective.

[0100]

[0101] Among them, the loss function of the value network The calculation based on minimizing the time difference (TD) error is shown below.

[0102]

[0103] Loss function of policy network The calculation is based on the maximum entropy objective, as shown below.

[0104]

[0105] In the above formula, and These are the weight parameters for the value network and the policy network, respectively. For experience replay pool; The single-step reward calculated by the system based on the current state and action; Discount factor; This is a temperature coefficient used to adjust the weight of the entropy term to control the agent's exploration level. This is the target value network used to stabilize the training process.

[0106] (3.2) Policy optimization based on model gradient

[0107] Although this invention uses the SAC framework, it introduces model-based gradient enhancement during the policy update phase, which is the core difference from traditional model-free RL:

[0108] The first point is the maximum entropy objective. This invention adopts the maximum entropy reinforcement learning objective, which simultaneously maximizes the cumulative reward and policy entropy, thereby enhancing the agent's exploration ability and robustness under complex conditions.

[0109] The second point is the construction of an end-to-end gradient flow that runs throughout the entire system, which is also the core innovation of this invention. This is involved in updating the policy network parameters. In addition to utilizing the Q-value gradient provided by the value network, this invention utilizes the full-link differentiability of the environment (PERL) and the actuator (PID) to form an end-to-end gradient flow that propagates back from the loss function to the policy network and the residual regression layer, and introduces direct physical gradients through the chain rule, as shown below.

[0110]

[0111] This mechanism allows the policy network to directly "sense" the differential properties of the dynamic model, thereby quickly converging to the optimal combination of PID parameters with a small number of samples.

[0112] (iv) Three-stage progressive training and joint optimization

[0113] This step describes how to solve the Sim2Real problem and achieve end-to-end optimization through phased training.

[0114] The first phase is PERL model pre-training and identification (Environment Reconstruction). The goal of this phase is to obtain a high-fidelity simulation environment.

[0115] Its loss function is shown below.

[0116]

[0117] This represents the nominal state prediction output of the physical prior layer based on physical parameter P, where P represents the set of physical parameters of the vehicle. It is the fundamental core parameter for constructing the physical prior layer (i.e., the 3-DOF monorail dynamics model). Represents the regularization coefficient of the physical confidence constraint term. express Real-world vehicle status data at any given moment.

[0118] First, identify the physical parameters based on the vehicle model. ,fixed Optimize network parameters (Residual learning).

[0119] Phase two is SAC policy warm-up. In a frozen PERL environment, the standard model-free update rules of SAC are used to train the policy and value network, allowing them to initially grasp the control logic. The loss during SAC policy warm-up is... .

[0120] Phase three is end-to-end fine-tuning. This phase, based on an experience replay pool and an end-to-end update mechanism, samples data from the experience replay pool during the joint fine-tuning phase. The total loss is used for end-to-end updates. The innovation of this stage is to leverage the differentiability of the links to directly backfeed the prediction error gradient to the policy network.

[0121] By using this gradient update policy network, the convergence speed and control accuracy can be significantly improved.

[0122] The PERL model accurately compensates for real-world features such as mechanical wear, delay, and ground friction, enabling the SAC strategy trained in simulation to be seamlessly transferred to real vehicles to bridge the Sim2Real gap. Simultaneously, a differentiable PID layer is introduced as the actuator, avoiding the "high-frequency chattering" phenomenon that may result from the direct output of control quantities by pure neural networks. Furthermore, the PID parameters have clear physical meanings, facilitating engineering monitoring and improving control smoothness and interpretability. Since the entire link (PID + PERL) is differentiable, the SAC algorithm can utilize model gradients for assisted updates (similar to model-based reinforcement learning), resulting in faster convergence and higher training efficiency compared to traditional model-free algorithms.

[0123] In inference applications, the trained SAC network is used to generate PID gain parameter values ​​and aiming distance based on the actual vehicle state to achieve autonomous driving control, which improves the path tracking accuracy and robustness of the vehicle under complex conditions.

[0124] Based on the above architectural description, the technical solution of the present invention can be implemented as a system, method, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present invention.

[0125] For example, a fully differentiable physical augmentation (PAR) autonomous driving control method includes: training a fully differentiable PERL dynamics model, configured to consist of a physical prior layer and a residual regression layer in parallel; training by minimizing the predicted vehicle state and the actual vehicle state of the physical and residual regression layers; freezing the parameters of the fully differentiable PERL dynamics model after training; pre-training a SAC (Sports Acceleration and Control) policy network after freezing the parameters; the SAC policy network is configured to adaptively adjust the PID gain parameter value and the aiming distance in real time based on the vehicle state; after the SAC policy network pre-training is completed, joint fine-tuning of the fully differentiable PERL dynamics model and the SAC policy network is performed; during fine-tuning training, the current vehicle state is considered in conjunction with the vehicle's dynamics model and the SAC policy network. The system uses a fully differentiable PERL dynamics model to predict the vehicle state at the next moment, inputting this state into the SAC (Self-Controlled Acquisition) policy network. The SAC policy network outputs the corresponding PID gain parameter value and the aiming distance. The actuator calculates the lateral and longitudinal velocity errors based on the dynamic aiming distance. Based on these errors and gain parameter values, a PID control law supporting gradient backpropagation is used to generate control quantities. The autonomous vehicle updates its state under the action of these control quantities. The predicted error gradient during training is fine-tuned by using the fully differentiable characteristics of the PERL dynamics model and the actuator to introduce direct physical gradients through a chain rule, which are then directly backpropagated to the SAC policy network. During inference, the trained SAC network uses the PID gain parameter value and aiming distance generated based on the actual vehicle state to achieve autonomous driving control.

[0126] For example, a fully differentiable physical augmentation autonomous driving control system includes: a PERL dynamics model training module configured to train a fully differentiable PERL dynamics model, which consists of a physical prior layer and a residual regression layer in parallel, predicting the vehicle state at the next moment based on the current vehicle state and control variables; a SAC policy network training module configured to pre-train the SAC policy network after freezing the parameters of the fully differentiable PERL dynamics model, whereby the SAC policy network is configured to adaptively adjust the PID gain parameter values ​​and the aiming distance in real time based on the input vehicle state; and a joint fine-tuning module configured to jointly fine-tune the fully differentiable PERL dynamics model and the SAC policy network after the SAC policy network pre-training is completed. In fine-tuning training, the current vehicle state is used as the input to the fully differentiable PERL dynamics model, and the vehicle state at the next moment is input to the SAC policy network. The SAC policy network outputs the corresponding PID gain parameter value and the aiming distance. The actuator calculates the lateral error and longitudinal velocity error based on the dynamic aiming distance. Based on the above errors and gain parameter values, the PID control law supporting gradient backpropagation is used to generate control quantities. The autonomous vehicle updates its vehicle state under the action of the control quantities. The prediction error gradient in fine-tuning training utilizes the fully differentiable characteristics of the PERL dynamics model and the actuator to introduce the direct physical gradient through the chain rule, which is directly backpropagated to the SAC policy network. During inference, the PID gain parameter value and aiming distance generated by the trained SAC network based on the actual vehicle state are used to achieve autonomous driving control.

[0127] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0128] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0129] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0130] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A physically-enhanced autonomous driving control method with end-to-end differentiability, characterized in that, include: The fully differentiable PERL dynamics model was trained. The fully differentiable PERL dynamics model was configured to consist of a physical prior layer and a residual regression layer in parallel. The model was trained by minimizing the predicted vehicle state and the actual vehicle state of the physical layer and the residual regression layer. After training, the parameters of the fully differentiable PERL dynamics model were frozen. After freezing the parameters of the fully differentiable PERL dynamics model, the SAC policy network is pre-trained. The SAC policy network is configured to adaptively adjust the PID gain parameter value and the aiming distance in real time based on the vehicle state. After the SAC policy network is pre-trained, the fully differentiable PERL dynamics model and the SAC policy network are jointly fine-tuned. In the fine-tuning training, based on the current vehicle state and control input, the fully differentiable PERL dynamics model predicts the vehicle state at the next moment and inputs the vehicle state at the next moment into the SAC policy network. The SAC policy network outputs the corresponding PID gain parameter value and the aiming distance. The actuator calculates the lateral error and longitudinal velocity error based on the dynamic aiming distance. Based on the above error and gain parameter values, the PID control law that supports gradient backpropagation is used to generate control inputs respectively. The autonomous vehicle updates the vehicle state under the action of the control input. The prediction error gradient in the fine-tuning training utilizes the fully differentiable characteristics of the PERL dynamics model and the actuator to introduce the direct physical gradient through the chain rule and directly backpropagates it to the SAC policy network. During inference, the trained SAC network is used to generate PID gain parameter values ​​and aiming distance based on the actual vehicle state to achieve autonomous driving control.

2. The method according to claim 1, characterized in that, The control input consists of steering wheel angle commands and longitudinal acceleration commands.

3. The method according to claim 1, characterized in that, During training, the residual regression layer employs a loss function with physical constraint regularization. in, This represents the real-world label data at time t+1. This represents the nominal state prediction value of the physical prior layer at time t+1. This represents the correction value for the state variables in the predicted output of the residual regression layer. This represents the regularization coefficient of the physical confidence constraint term.

4. The method according to claim 1, characterized in that: The residual regression layer adopts a fully connected multilayer perceptron structure, and its input is a concatenated vector of vehicle state and control quantity. And perform layer normalization to accelerate convergence, vehicle state = Control quantity = Output and vehicle status Correction amount in the same dimension ;in, and The coordinates of the vehicle's center of gravity. For the vehicle's heading angle, For the longitudinal speed of the vehicle, For the vehicle's lateral speed, The vehicle's yaw rate; This is a steering angle command. This is a longitudinal acceleration command.

5. The method according to claim 1, characterized in that, The physical prior layer is based on the classical Newtonian laws of mechanics and uses a three-degree-of-freedom single-track dynamics model to calculate the motion trend of the vehicle under ideal conditions.

6. The method according to claim 1, characterized in that, The actuator calculates lateral and longitudinal velocity errors based on dynamic pre-aiming distance, employing a differentiable error calculation method based on dynamic pre-aiming and soft attention mechanisms. The steps include: Based on the pre-aiming distance output by SAC Calculate the virtual preview point in the vehicle's forward field of view. , , Let be the coordinates of the vehicle's center of mass at the current moment. This represents the vehicle's heading angle at the current moment. Attention weights at each point on the reference trajectory are calculated using a Gaussian kernel function based on Euclidean distance. : in For temperature coefficient, For the first reference trajectory A discrete path point, For the first reference trajectory There are discrete path points, where the value of j ranges from 0 to N, and N is the total number of reference trajectory points; The matching point is obtained by weighted summation. and its normal vector : , ; Calculate the lateral error ; Reference speed based on each waypoint Calculate longitudinal velocity error , This represents the vehicle's current longitudinal speed.

7. The method according to claim 1, characterized in that, The PID control law supporting gradient feedback uses pure algebraic calculations to simulate the mechanical limits of the physical actuator, including: Based on the current lateral distance error and longitudinal velocity error, and combining their respective historical error integrals and derivatives, the unlimited lateral control quantity is calculated. With longitudinal control quantity ; use The functions perform smoothing and limiting respectively, and generate steering wheel angle commands. With longitudinal acceleration command : ， in: This is the mechanical limit value for the steering wheel angle. This is the mechanical limit value for longitudinal acceleration.

8. A computer-readable storage medium, characterized in that: The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 7.

9. A fully differentiable, physically-enhanced automated driving control system, characterized in that, include: The PERL dynamics model training module is configured to train a fully differentiable PERL dynamics model. The fully differentiable PERL dynamics model consists of a physical prior layer and a residual regression layer in parallel, which predicts the vehicle state at the next moment based on the current vehicle state and control variables. The SAC policy network training module is configured to pre-train the SAC policy network after freezing the parameters of the fully differentiable PERL dynamics model. The SAC policy network is configured to adaptively adjust the PID gain parameter value and the aiming distance in real time based on the input vehicle state. The joint fine-tuning module is configured to perform joint fine-tuning of the fully differentiable PERL dynamics model and the SAC policy network after the SAC policy network is pre-trained. In the fine-tuning training, the current vehicle state is used as the input of the fully differentiable PERL dynamics model, and the vehicle state at the next moment is input to the SAC policy network. The SAC policy network outputs the corresponding PID gain parameter value and the aiming distance. The actuator calculates the lateral error and longitudinal velocity error based on the dynamic aiming distance. Based on the above errors and gain parameter values, the PID control law that supports gradient backpropagation is used to generate control quantities respectively. The autonomous vehicle updates the vehicle state under the action of the control quantities. The prediction error gradient in the fine-tuning training utilizes the fully differentiable characteristics of the PERL dynamics model and the actuator to introduce the direct physical gradient through the chain rule, and directly backpropagates it to the SAC policy network. During inference, the trained SAC network is used to generate PID gain parameter values ​​and aiming distance based on the actual vehicle state to achieve autonomous driving control.

Images