An intelligent vehicle stability control method based on deep reinforcement learning

Abstract

The application discloses an intelligent automobile stability control method based on deep reinforcement learning, which comprises the following steps: constructing a multi-dimensional meta training data set; using a meta multi-objective soft Meta-MOSAC algorithm to pre-train the meta training data set to obtain a cross-driving universal base model; based on the current driving system characteristic parameters, the cross-driving universal base model is fine-tuned online to obtain a personalized control model suitable for the target vehicle; a dynamic reward function based on the driving system characteristics is used to output a multi-objective optimal control instruction; the multi-objective optimal control instruction is sent to the vehicle actuator module to realize integrated control of vehicle stability and energy management. The same model can be adapted to all new energy driving systems such as PHEV, BEV and FCEV, the cross-driving migration accuracy is improved, and the comprehensive energy consumption is reduced under the premise of ensuring the same stability index.

Description

Technical Field

[0001] This invention relates to the field of intelligent vehicle control technology, and specifically to an intelligent vehicle stability control method based on deep reinforcement learning. Background Technology

[0002] With the rapid development of new energy vehicle technology, various power configurations such as plug-in hybrid, pure electric, and fuel cell have emerged in the market. Current vehicle stability control technology based on deep reinforcement learning (DRL) faces the following problems: 1. Most existing DRL controllers are designed only for pure electric drive and do not take into account the unique dynamic characteristics of PHEVs such as engine-motor coupling torque fluctuations and FCEVs with power response lag. When migrating across drive systems, retraining is required, resulting in high R&D costs and long development cycles.

[0003] 2. A layered architecture of "upper-level energy management + lower-level stability control" is generally adopted. When stability control is prioritized, torque will be forcibly allocated, resulting in increased energy consumption; when energy management is prioritized, the stability margin will be reduced, which poses a safety hazard in emergency conditions.

[0004] 3. Significant impact during mode switching: When switching between pure electric and hybrid modes, PHEVs experience noticeable torque fluctuations and yaw rate overshoot because the control model cannot quickly adapt to the changes in dynamics, affecting driving comfort and safety.

[0005] To address the aforementioned issues, existing technologies have not yet proposed effective solutions. This invention achieves a single model that adapts to all new energy drive systems through the deep integration of meta-learning and multi-objective reinforcement learning, while simultaneously achieving globally optimal control of stability and energy management. Summary of the Invention

[0006] The present invention proposes a method for intelligent vehicle stability control based on deep reinforcement learning, which can at least solve one of the technical problems in the background art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A method for intelligent vehicle stability control based on deep reinforcement learning includes the following steps: S1. Construct a multi-dimensional training dataset, which contains associated data on the dynamic characteristics, energy characteristics, and operating condition characteristics of different new energy drive systems; S2. The meta-training dataset is pre-trained using the meta-multi-objective soft Meta-MOSAC algorithm to obtain a cross-driver general basic model. S3. Real-time acquisition of the target vehicle's operating status parameters and current drive system characteristic parameters; S4. Based on the current driving system characteristic parameters, the cross-drive general basic model is fine-tuned online to obtain a personalized control model adapted to the target vehicle; S5. Input the operating status parameters into the personalized control model, and output multi-objective optimal control instructions based on the dynamic reward function guided by the characteristics of the drive system; S6. Send the multi-objective optimal control command to the vehicle actuator module to realize integrated control of vehicle stability and energy management.

[0008] As a preferred embodiment of the intelligent vehicle stability control method based on deep reinforcement learning described in this invention, the method for constructing the multi-dimensional meta-training dataset in step S1 specifically includes: S11. Collect the core features of three types of drive systems: plug-in hybrid electric vehicle (PHEV), pure electric vehicle (BEV), and fuel cell electric vehicle (FCEV). The features of PHEV include the engine intervention point and the motor-engine torque coupling coefficient; the features of BEV include the motor torque response time constant; and the features of FCEV include the fuel cell stack power slope limit. S12. Collect energy characteristic data for each type of drive system under different operating conditions, including battery SOC change rate, engine fuel consumption rate, and fuel cell hydrogen consumption rate. S13. The core features and energy characteristic data are associated and labeled with the corresponding vehicle stability state data, including yaw rate, center of gravity sideslip angle and longitudinal acceleration, to form a single drive system sub-dataset. S14. Merge at least 12 different dynamic configurations of single-drive system sub-datasets to construct the multidimensional meta-training dataset.

[0009] As a preferred embodiment of the intelligent vehicle stability control method based on deep reinforcement learning described in this invention, the pre-training process of the Meta-MOSAC algorithm in step S2 specifically includes: S21. Initialize model parameters The multidimensional training dataset is divided into multiple task subsets, each task subset corresponding to a control task of a driving system. S22. In the inner loop, the multi-objective soft Actor-Critic algorithm is used to train each task subset, and the loss function is optimized:

[0010] in To control losses for stability, For energy management losses; S23. In the outer loop, update the initial parameters of the model based on the training results of the inner loop for all task subsets. This allows the model to converge with only a small number of gradient updates on new tasks; S24. Repeat steps S22-S23 until the model converges to obtain the cross-drive general basic model.

[0011] As a preferred embodiment of the intelligent vehicle stability control method based on deep reinforcement learning described in this invention, the method for acquiring the current driving system feature parameters in step S3 is specifically as follows: S31. Read the drive system operating mode signal via the vehicle CAN bus; S32. Based on a preset drive system feature encoding library, map the operating mode signal into a corresponding feature vector. ; S33. Calculate in real time the deviation between the actual output torque and the expected output torque of the current drive system, and apply this to the feature vector. Perform online corrections.

[0012] As a preferred embodiment of the intelligent vehicle stability control method based on deep reinforcement learning described in this invention, wherein: the dynamic reward function guided by the driving system features in step S5 is:

[0013] in, This is a stability bonus item; For energy management rewards; These are dynamic weighting coefficients, and , Based on the basic stability weights, To adjust the coefficient, Let be the energy efficiency sensitivity coefficient of the current drive system, and satisfy the condition that the energy efficiency sensitivity coefficient of PHEV is greater than that of BEV, and the energy efficiency sensitivity coefficient of BEV is greater than that of FCEV.

[0014] As a preferred embodiment of the intelligent vehicle stability control method based on deep reinforcement learning described in this invention, the specific steps of online fine-tuning in step S4 are as follows: S41, the current driving system feature vector Input the dynamic feature alignment module to generate the model input weight adjustment matrix. and output torque limiting matrix ; S42. Using the weight adjustment matrix The input layer parameters of the cross-drive general basic model are adjusted; S43. Based on the target vehicle's recent 5-10 minute running data, perform gradient updates on the parameters of the last two fully connected layers of the model. S44, Using the aforementioned output torque limiting matrix The range of control commands output by the model is constrained to obtain the personalized control model.

[0015] As a preferred embodiment of the intelligent vehicle stability control method based on deep reinforcement learning described in this invention, when a change in the driving mode of the target vehicle is detected, the following seamless transition control step is further included: S71. 100ms before the drive mode switch, preload the feature vector and dynamic weight coefficients corresponding to the target drive mode. ; S72. During the switching process, an exponentially smooth weighting method is used to transition the model input weights and output torque constraints. S73. Synchronously adjust the weight coefficients in the dynamic reward function. This allows for a smooth transition between the goals of stability and energy management optimization; S74. After the switch is completed, a new round of online fine-tuning is started, and the model is fully adapted within 5 control cycles.

[0016] As a preferred embodiment of the intelligent vehicle stability control method based on deep reinforcement learning described in this invention, the multi-objective optimal control command in step S5 includes the torque command of the four-wheel independent drive motor, the braking force command of the brake-by-wire device, and the steering angle command of the active steering system, and all control commands satisfy the maximum output power limit and temperature limit of the corresponding actuator.

[0017] As a preferred embodiment of the intelligent vehicle stability control method based on deep reinforcement learning described in this invention, the real-time collected operating state parameters in step S3 include vehicle yaw rate, center of gravity sideslip angle, longitudinal acceleration, lateral acceleration, wheel speed, steering wheel angle, battery SOC, fuel cell output power, and engine speed, and all parameters have undergone time synchronization processing with a time synchronization error of less than 10ms.

[0018] An intelligent vehicle stability control system includes: The sensor module is used to collect vehicle operating status parameters and drive system characteristic parameters in real time; The memory is used to store the multi-dimensional meta-training dataset and the cross-driver general base model; The processor, which is communicatively connected to the sensor module and the memory, is used to execute the steps of the control method and output multi-objective optimal control instructions. The actuator module, which is communicatively connected to the processor, is used to receive and execute the multi-objective optimal control instructions.

[0019] The beneficial effects of this invention are: The same model of this invention can be adapted to all new energy drive systems such as PHEV, BEV, and FCEV, improving cross-drive migration accuracy and reducing overall energy consumption while ensuring the same stability indicators. When switching PHEV drive modes, the yaw rate overshoot is reduced. Online fine-tuning can be completed in only 5-8 steps of gradient update, with a control cycle of less than 100ms, meeting the real-time control requirements of the vehicle. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the steps of the intelligent vehicle stability control method based on deep reinforcement learning according to the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0022] like Figure 1 As shown, a method for intelligent vehicle stability control based on deep reinforcement learning includes the following steps: S1. Construct a multi-dimensional training dataset, which contains associated data on the dynamic characteristics, energy characteristics, and operating conditions of different new energy drive systems; S2. The meta-multi-objective soft Meta-MOSAC algorithm is used to pre-train the meta-training dataset to obtain a cross-driver general basic model; S3. Real-time acquisition of the target vehicle's operating status parameters and current drive system characteristic parameters; S4. Based on the characteristic parameters of the current drive system, the cross-drive general basic model is fine-tuned online to obtain a personalized control model adapted to the target vehicle. S5. Input the operating status parameters into the personalized control model, and output multi-objective optimal control instructions based on the dynamic reward function guided by the characteristics of the drive system. S6. Send multi-objective optimal control commands to the vehicle actuator module to achieve integrated control of vehicle stability and energy management.

[0023] Specifically, the method for constructing the multi-dimensional meta-training dataset in step S1 includes: S11. Collect the core features of three types of drive systems: plug-in hybrid electric vehicle (PHEV), pure electric vehicle (BEV), and fuel cell electric vehicle (FCEV). The features of PHEV include the engine intervention point and the motor-engine torque coupling coefficient; the features of BEV include the motor torque response time constant; and the features of FCEV include the fuel cell stack power slope limit. S12. Collect energy characteristic data for each type of drive system under different operating conditions, including battery SOC change rate, engine fuel consumption rate, and fuel cell hydrogen consumption rate. S13. Associate and label the core features and energy characteristic data with the corresponding vehicle stability state data, including yaw rate, center of gravity sideslip angle and longitudinal acceleration, to form a single drive system subset dataset. S14. Merge at least 12 different dynamic configurations of single-drive system subsets to construct a multidimensional meta-training dataset.

[0024] Furthermore, the pre-training process of the Meta-MOSAC algorithm in step S2 is as follows: S21. Initialize model parameters The multidimensional training dataset is divided into multiple task subsets, each task subset corresponding to a control task of a driving system. S22. In the inner loop, the multi-objective soft Actor-Critic algorithm is used to train each task subset, and the loss function is optimized:

[0025] in To control losses for stability, For energy management losses; S23. In the outer loop, update the initial parameters of the model based on the training results of the inner loop for all task subsets. This allows the model to converge with only a small number of gradient updates on new tasks; S24. Repeat steps S22-S23 until the model converges to obtain a cross-driver general basic model.

[0026] Specifically, the method for collecting the characteristic parameters of the current driving system in step S3 is as follows: S31. Read the drive system operating mode signal via the vehicle CAN bus; S32. Based on the preset drive system feature encoding library, map the operating mode signal into the corresponding feature vector. ; S33. Calculate in real time the deviation between the actual output torque and the expected output torque of the current drive system, and apply this to the feature vector. Perform online corrections.

[0027] In step S3, the real-time collected operating status parameters include vehicle yaw rate, center of gravity sideslip angle, longitudinal acceleration, lateral acceleration, wheel speed, steering wheel angle, battery SOC, fuel cell output power, and engine speed. All parameters have undergone time synchronization processing, with a time synchronization error of less than 10ms.

[0028] Furthermore, the dynamic reward function driven by the system features in step S5 is as follows:

[0029] in, This is a stability bonus item; For energy management rewards; These are dynamic weighting coefficients, and , Based on the basic stability weights, To adjust the coefficient, Let be the energy efficiency sensitivity coefficient of the current drive system, and satisfy the condition that the energy efficiency sensitivity coefficient of PHEV is greater than that of BEV, and the energy efficiency sensitivity coefficient of BEV is greater than that of FCEV.

[0030] The specific steps for online fine-tuning in step S4 are as follows: S41. Transfer the current driving system feature vector Input the dynamic feature alignment module to generate the model input weight adjustment matrix. and output torque limiting matrix ; S42. Use a weight adjustment matrix. Adjust the input layer parameters of the cross-drive general basic model; S43. Based on the target vehicle's recent 5-10 minute running data, perform gradient updates on the parameters of the last two fully connected layers of the model. S44, Employs an output torque limiting matrix By constraining the range of control commands output by the model, a personalized control model is obtained.

[0031] Furthermore, when a change in the target vehicle's drive mode is detected, the following seamless transition control steps are also included: S71. 100ms before the drive mode switch, preload the feature vector and dynamic weight coefficients corresponding to the target drive mode. ; S72. During the switching process, an exponentially smooth weighting method is used to transition the model input weights and output torque constraints. S73. Synchronously adjust the weight coefficients in the dynamic reward function. This allows for a smooth transition between the goals of stability and energy management optimization; S74. After the switch is completed, a new round of online fine-tuning is started, and the model is fully adapted within 5 control cycles.

[0032] In step S5, the multi-objective optimal control commands include the torque command of the four-wheel independent drive motor, the braking force command of the brake-by-wire brake, and the steering angle command of the active steering system. All control commands satisfy the maximum output power limit and temperature limit of the corresponding actuator.

[0033] The same model can be adapted to all new energy drive systems such as PHEV, BEV, and FCEV, improving cross-drive migration accuracy and reducing overall energy consumption while ensuring the same stability indicators. When switching PHEV drive modes, the yaw rate overshoot is reduced. Online fine-tuning can be completed in only 5-8 steps of gradient update, with a control cycle of less than 100ms, meeting the real-time control requirements of the vehicle.

[0034] In another aspect, the present invention also discloses an intelligent vehicle stability control system, comprising: The sensor module is used to collect vehicle operating status parameters and drive system characteristic parameters in real time; Memory is used to store multi-dimensional meta-training datasets and cross-driver general base models; The processor, which communicates with the sensor module and memory, is used to execute the steps of the control method and output multi-objective optimal control instructions. The actuator module, which communicates with the processor, is used to receive and execute multi-objective optimal control instructions.

[0035] Specifically, the sensor module: Inertial Measurement Unit (IMU): Acquires yaw rate, center of mass sideslip angle, and longitudinal / lateral acceleration; Wheel speed sensor: Collects the rotational speed of the four wheels; Steering wheel angle sensor: Collects the driver's steering wheel angle; Battery Management System (BMS): Collects battery SOC, voltage, current, and temperature; Engine Control Unit (ECU): Collects engine speed, torque, and fuel consumption rate; Fuel cell control unit (FCU): Collects fuel cell output power and hydrogen consumption rate; processor: An on-board power domain controller with a computing power of no less than 100 TOPS and an operating frequency of 10Hz is used to execute the control method steps of this invention.

[0036] Memory: It includes flash memory and RAM. Flash memory is used to store multi-dimensional meta-training datasets and cross-driver general base models, while RAM is used to store real-time running data and experience replay pools.

[0037] Actuator module: Four-wheel independent drive motor controller; Brake-by-wire system controller; Active steering system controller; Engine / fuel cell power controller.

[0038] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0039] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0040] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent vehicle stability control based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Construct a multi-dimensional training dataset, which contains associated data on the dynamic characteristics, energy characteristics, and operating condition characteristics of different new energy drive systems; S2. The meta-training dataset is pre-trained using the meta-multi-objective soft Meta-MOSAC algorithm to obtain a cross-driver general basic model. S3. Real-time acquisition of the target vehicle's operating status parameters and current drive system characteristic parameters; S4. Based on the current driving system characteristic parameters, the cross-drive general basic model is fine-tuned online to obtain a personalized control model adapted to the target vehicle; S5. Input the operating status parameters into the personalized control model, and output multi-objective optimal control instructions based on the dynamic reward function guided by the characteristics of the drive system; S6. Send the multi-objective optimal control command to the vehicle actuator module to realize integrated control of vehicle stability and energy management.

2. The intelligent vehicle stability control method based on deep reinforcement learning according to claim 1, characterized in that: The method for constructing the multidimensional meta-training dataset in step S1 specifically includes: S11. Collect the core features of three types of drive systems: plug-in hybrid electric vehicle (PHEV), pure electric vehicle (BEV), and fuel cell electric vehicle (FCEV). The features of PHEV include the engine intervention point and the motor-engine torque coupling coefficient; the features of BEV include the motor torque response time constant; and the features of FCEV include the fuel cell stack power slope limit. S12. Collect energy characteristic data for each type of drive system under different operating conditions, including battery SOC change rate, engine fuel consumption rate, and fuel cell hydrogen consumption rate. S13. The core features and energy characteristic data are associated and labeled with the corresponding vehicle stability state data, including yaw rate, center of gravity sideslip angle and longitudinal acceleration, to form a single drive system sub-dataset. S14. Merge at least 12 different dynamic configurations of single-drive system sub-datasets to construct the multidimensional meta-training dataset.

3. The intelligent vehicle stability control method based on deep reinforcement learning according to claim 1, characterized in that: The pre-training process of the Meta-MOSAC algorithm in step S2 is as follows: S21. Initialize model parameters The multidimensional training dataset is divided into multiple task subsets, each task subset corresponding to a control task of a driving system. S22. In the inner loop, the multi-objective soft Actor-Critic algorithm is used to train each task subset, and the loss function is optimized: ； in To control losses for stability, For energy management losses; S23. In the outer loop, update the initial parameters of the model based on the training results of the inner loop for all task subsets. This allows the model to converge with only a small number of gradient updates on new tasks; S24. Repeat steps S22-S23 until the model converges to obtain the cross-drive general basic model.

4. The intelligent vehicle stability control method based on deep reinforcement learning according to claim 1, characterized in that: The specific method for collecting the characteristic parameters of the current driving system in step S3 is as follows: S31. Read the drive system operating mode signal via the vehicle CAN bus; S32. Based on a preset drive system feature encoding library, map the operating mode signal into a corresponding feature vector. ; S33. Calculate in real time the deviation between the actual output torque and the expected output torque of the current drive system, and apply this to the feature vector. Perform online corrections.

5. The intelligent vehicle stability control method based on deep reinforcement learning according to claim 4, characterized in that: The dynamic reward function guided by the characteristics of the driving system in step S5 is: ； in, This is a stability bonus item; For energy management rewards; These are dynamic weighting coefficients, and , Based on the basic stability weights, To adjust the coefficient, Let be the energy efficiency sensitivity coefficient of the current drive system, and satisfy the condition that the energy efficiency sensitivity coefficient of PHEV is greater than that of BEV, and the energy efficiency sensitivity coefficient of BEV is greater than that of FCEV.

6. The intelligent vehicle stability control method based on deep reinforcement learning according to claim 5, characterized in that: The specific steps for online fine-tuning described in step S4 are as follows: S41, the current driving system feature vector Input the dynamic feature alignment module to generate the model input weight adjustment matrix. and output torque limiting matrix ; S42. Using the weight adjustment matrix The input layer parameters of the cross-drive general basic model are adjusted; S43. Based on the target vehicle's recent 5-10 minute running data, perform gradient updates on the parameters of the last two fully connected layers of the model. S44, Using the aforementioned output torque limiting matrix The range of control commands output by the model is constrained to obtain the personalized control model.

7. The intelligent vehicle stability control method based on deep reinforcement learning according to claim 6, characterized in that: When a change in the target vehicle's drive mode is detected, the following seamless transition control steps are also included: S71. 100ms before the drive mode switch, preload the feature vector and dynamic weight coefficients corresponding to the target drive mode. ; S72. During the switching process, an exponentially smooth weighting method is used to transition the model input weights and output torque constraints. S73. Synchronously adjust the weight coefficients in the dynamic reward function. This allows for a smooth transition between the goals of stability and energy management optimization; S74. After the switch is completed, a new round of online fine-tuning is started, and the model is fully adapted within 5 control cycles.

8. The intelligent vehicle stability control method based on deep reinforcement learning according to claim 1, characterized in that: The multi-objective optimal control commands mentioned in step S5 include torque commands from the four-wheel independent drive motors, braking force commands from the brake-by-wire brakes, and steering angle commands from the active steering system. All control commands satisfy the maximum output power limit and temperature limit of the corresponding actuator.

9. The intelligent vehicle stability control method based on deep reinforcement learning according to claim 1, characterized in that: The real-time collected operating status parameters in step S3 include vehicle yaw rate, center of gravity sideslip angle, longitudinal acceleration, lateral acceleration, wheel speed, steering wheel angle, battery SOC, fuel cell output power, and engine speed. All of these parameters have undergone time synchronization processing, with a time synchronization error of less than 10ms.

10. An intelligent vehicle stability control system implementing the control method according to any one of claims 1-9, characterized in that, include: The sensor module is used to collect vehicle operating status parameters and drive system characteristic parameters in real time; The memory is used to store the multi-dimensional meta-training dataset and the cross-driver general base model; The processor, which is communicatively connected to the sensor module and the memory, is used to execute the steps of the control method and output multi-objective optimal control instructions. The actuator module, which is communicatively connected to the processor, is used to receive and execute the multi-objective optimal control instructions.

Images