A hybrid electric vehicle mode switching control method based on reinforcement learning

A hybrid electric vehicle mode switching control method was constructed by using a deep Q-learning algorithm. The agent was trained using historical driving data and control rules were extracted, which solved the problems of insufficient adaptability and real-time performance of energy management strategies under operating conditions, and realized adaptive and real-time mode switching control.

CN115257691BActive Publication Date: 2024-11-26TONGJI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210896238.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-07-28
Publication Date
2024-11-26
Estimated Expiration
2042-07-28

AI Technical Summary

Technical Problem

Existing energy management strategies for hybrid electric vehicles are inadequate in terms of adaptability and real-time performance, resulting in unreliable mode switching control.

Method used

A hybrid electric vehicle mode switching control method is constructed using a deep Q-learning algorithm. By acquiring and preprocessing historical driving data, a deep Q-learning network is built, an agent is trained, control rules are extracted and applied to the vehicle controller, and the optimal energy management strategy is output by combining actual driving data.

Benefits of technology

It improves the adaptability of energy management strategies and the reliability of mode switching control, simplifies the action space of reinforcement learning, increases training speed, and ensures real-time performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115257691B_ABST
    Figure CN115257691B_ABST
Patent Text Reader

Abstract

The present invention relates to a hybrid vehicle mode switching control method based on reinforcement learning, the method comprising: using a reinforcement learning algorithm of a deep Q network to control the mode switching of a hybrid vehicle, using the vehicle speed, required power, battery SOC, etc. as the state of reinforcement learning, using the mode switching of the vehicle as the action variable, and using the collected historical driving data to train the deep Q network; extracting the control rules of the mode switching from the trained network to apply to the vehicle controller of the actual vehicle. Compared with the prior art, the present invention uses a deep Q learning algorithm to control the mode switching of the vehicle, which can improve the adaptability of the energy management strategy and ensure the reliability and real-time performance of the mode switching control.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of hybrid electric vehicles, and in particular to a hybrid electric vehicle mode switching control method based on reinforcement learning. Background Art

[0002] The power source of hybrid vehicles consists of electric motors and internal combustion engines. When providing power, they can have different energy distribution methods, thus forming different modes of hybrid vehicles, providing design freedom for the vehicle controller. As one of the core technologies in the development of hybrid vehicles, energy management strategy mainly coordinates the energy distribution between the engine and the motor of the hybrid vehicle to achieve the energy distribution of the vehicle under different driving conditions. An energy management strategy with excellent performance is of great significance for giving full play to the structural advantages of hybrid vehicles, improving fuel economy, and achieving sustainable environmental development.

[0003] At present, energy management strategies include rule-based energy management strategies and optimization-based energy management strategies. Among them, rule-based energy management strategies can be divided into deterministic rules and fuzzy rules, and optimization-based energy management strategies can be divided into global optimization and instantaneous optimization. Rule-based energy management strategies are widely used in real vehicle energy management due to their simplicity, reliability, and strong interpretability. However, the formulation of this energy management strategy relies on experience and has poor adaptability to working conditions. Although the energy management strategy based on optimization can theoretically obtain the optimal energy management under known working conditions, considering that the actual working environment of the car is complex and changeable, this limits its wide application to a certain extent. If the optimal energy management strategy cannot be achieved, the vehicle mode switching control will not meet the reliability and real-time requirements. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a hybrid vehicle mode switching control method based on reinforcement learning. The deep Q learning algorithm is used to control the mode switching of the vehicle, which can improve the adaptability of the energy management strategy and ensure the reliability and real-time performance of the mode switching control.

[0005] The object of the present invention can be achieved by the following technical solution: A hybrid electric vehicle mode switching control method based on reinforcement learning comprises the following steps:

[0006] S1. Acquire historical driving data of the hybrid vehicle and pre-process the historical driving data;

[0007] S2. Build a deep Q learning network and train the deep Q learning network based on the preprocessed historical driving data to obtain a trained intelligent agent;

[0008] S3, extracting control rules from the agent;

[0009] S4, applying the extracted control rules to the vehicle controller, collecting the actual driving data of the hybrid vehicle, combining the control rules, and outputting the optimal energy management strategy;

[0010] S5. According to the optimal energy management strategy, the mode switching of the hybrid vehicle is controlled accordingly.

[0011] Furthermore, the driving data includes vehicle speed and SOC information, and the driving data is specifically collected by sensors.

[0012] Furthermore, the specific process of preprocessing the historical driving data in step S1 is as follows:

[0013] Model vehicle energy requirements;

[0014] The historical vehicle speed data is input into the vehicle energy demand model, and the corresponding power demand data is output.

[0015] Furthermore, the vehicle energy demand model is specifically:

[0016]

[0017] Among them, F req is the driving force required by the car, Cd is the drag coefficient, A is the frontal area, v is the vehicle speed, m is the vehicle mass, g is the acceleration of gravity, θ is the road slope, f is the rolling resistance coefficient, and t is the time.

[0018] Furthermore, the state of the deep Q learning network in step S2 includes vehicle power demand, battery system SOC and vehicle speed, the action of the deep Q learning network is specifically the mode switching of the vehicle, and the reward function of the deep Q learning network reinforcement learning is specifically the instantaneous fuel consumption, instantaneous power consumption, and the penalty when the actual SOC exceeds the set operating range.

[0019] Furthermore, the state variables of the deep Q learning network are specifically:

[0020] S t ∈{(SOC t ,v′ t ,p′ t )|0≤SOC t ≤1,0≤v′ t ≤1,-1≤p′ t ≤1}

[0021] Among them, S is the state, v' is the standardized speed variable, p' is the standardized power demand, and t is the time.

[0022] Furthermore, the normalized speed variable is obtained by dividing the real-time speed by the maximum speed;

[0023] The standardized power demand is specifically obtained by dividing the real-time power demand by the absolute value of the maximum power demand.

[0024] Furthermore, the reward function of the deep Q learning network is specifically:

[0025]

[0026] in, is the instantaneous fuel consumption, is the instantaneous power consumption, λ 1 , 2 , α is the penalty coefficient, high and low are the upper and lower limit marks of the SOC range respectively.

[0027] Furthermore, the step S2 specifically uses the vehicle driving data and the vehicle energy demand model as the external environment of the reinforcement learning agent, and trains the deep Q learning network through the interaction between the agent and the environment.

[0028] Furthermore, the control rule is specifically a table with state variables as indexes.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] 1. The present invention adopts a deep Q learning algorithm to control the mode switching of the vehicle. By constructing a deep Q learning network, the vehicle speed, SOC and power demand are discretized. Reinforcement learning learns experience through the interaction between the intelligent agent and the environment, and uses this to improve the energy management strategy. It has the characteristics of self-learning and self-adaptation, which can not only simplify the action space of reinforcement learning and increase the training speed, but also solve the defect of poor adaptability of the current energy management strategy, so that the energy management strategy has a certain self-learning ability, and ensures the reliability of mode switching control.

[0031] 2. The present invention extracts control rules indexed by state variables from the trained intelligent agent and applies them to the vehicle controller, that is, converts the trained reinforcement learning intelligent agent into a rule-based energy management strategy, which can be directly used for the control of actual vehicles. It combines the advantages of self-learning of reinforcement learning algorithms and the real-time performance of rule-based energy management strategies, and further ensures the adaptability and real-time performance of control. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0033] Figure 2 Schematic diagram of the training process of the deep Q learning network in the present invention;

[0034] Figure 3 This is a schematic diagram of the control principle when the vehicle is actually running in the embodiment;

[0035] Figure 4 Schematic diagram of the application framework in the embodiment. DETAILED DESCRIPTION

[0036] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] Example

[0038] like Figure 1 As shown, a hybrid vehicle mode switching control method based on reinforcement learning includes the following steps:

[0039] S1. Acquire historical driving data of the hybrid vehicle and pre-process the historical driving data;

[0040] S2. Build a deep Q learning network and train the deep Q learning network based on the preprocessed historical driving data to obtain a trained intelligent agent;

[0041] S3, extracting control rules from the agent;

[0042] S4, applying the extracted control rules to the vehicle controller, collecting the actual driving data of the hybrid vehicle, combining the control rules, and outputting the optimal energy management strategy;

[0043] S5. According to the optimal energy management strategy, the mode switching of the hybrid vehicle is controlled accordingly.

[0044] This embodiment applies the above technical solution, such as Figure 2 to Figure 4 As shown, it mainly includes:

[0045] Use sensors to collect data from real hybrid vehicles in real-world driving, including information about vehicle speed over time and SOC information, and use it as input to the vehicle energy demand model;

[0046] The energy demand model of the vehicle is established in advance, taking into account the longitudinal dynamics of the vehicle, including rolling resistance, wind resistance, acceleration resistance, and inertial resistance. The mathematical expression of the vehicle energy demand model is as follows:

[0047]

[0048] In the formula, F req is the driving force of automobile demand, Cd is the drag coefficient, A is the frontal area, v is the vehicle speed, m is the vehicle mass, g is the acceleration of gravity, θ is the road slope, f is the rolling resistance coefficient, and t is the time;

[0049] The established vehicle energy demand model is used in a simulator in the cloud and serves as an external environment for reinforcement learning.

[0050] Construct a deep Q network reinforcement learning (DQN) network, using the car's power demand, battery system SOC, and vehicle speed as the state of the Q network, the vehicle's mode switching as the reinforcement learning action, and the instantaneous fuel consumption, instantaneous power consumption, and the penalty when the actual SOC exceeds the operating range as the reinforcement learning reward function;

[0051] Specifically, the state variables of reinforcement learning are:

[0052] S t ∈{(SOC t ,v′ t ,p′ t )|0≤SOC t ≤1,0≤v′ t ≤1,-1≤p′ t ≤1}

[0053] Where S represents the state, v' represents the normalized speed variable (real-time speed divided by the maximum speed), p' represents the normalized power demand (real-time power demand divided by the absolute value of the maximum power demand), and t represents time;

[0054] The action variable of reinforcement learning is the mode switching signal;

[0055] The reward function of reinforcement learning is specifically:

[0056]

[0057] In the formula, is the instantaneous fuel consumption, is the instantaneous power consumption, λ 1 , 2 , α is the penalty coefficient, high and low are the upper and lower limits of the SOC range;

[0058] The collected car driving data and vehicle energy demand model are then used as the external environment of the reinforcement learning agent, and the Q network is trained through the interaction between the agent and the environment;

[0059] Finally, control rules in a table indexed by reinforcement learning state variables are extracted from the trained agent and directly used in the vehicle controller of an actual hybrid electric vehicle.

[0060] In this embodiment, taking a series hybrid electric vehicle as an example, the specific application process includes:

[0061] 1. Use GPS modules, wheel speed sensors, pedal position signal sensors, etc. to collect data on actual hybrid vehicles during real-world driving, including information on vehicle speed changes over time, SOC information, etc., and use it as the actual working condition of the vehicle energy demand model.

[0062] Second, measure the actual vehicle parameters and establish the vehicle energy demand model, taking into account the longitudinal dynamics of the vehicle, including rolling resistance, wind resistance, acceleration resistance, and inertial resistance, to simulate the actual power demand during vehicle operation. The mathematical expression of the model is as follows:

[0063]

[0064] In the formula, F req is the driving force of automobile demand, Cd is the drag coefficient, A is the frontal area, v is the vehicle speed, m is the vehicle mass, g is the acceleration of gravity, θ is the road slope, f is the rolling resistance coefficient, and t is time.

[0065] 3. The established vehicle energy demand model is used in the cloud simulator as the external environment for reinforcement learning. The intelligent agent trains the mode switching strategy through interaction with the environment.

[0066] Construct a Q network and set several elements of reinforcement learning: use the car's power demand, battery system SOC, and vehicle speed as the input states of the Q network, and divide the driving mode of the series hybrid vehicle into pure electric mode and series mode; use the vehicle's mode switching as the action of reinforcement learning; use the instantaneous fuel consumption, instantaneous power consumption, and the penalty when the actual SOC exceeds the operating range as the reward function of reinforcement learning.

[0067] State variables for reinforcement learning:

[0068] S t ∈{(SOC t ,v′ t ,p′ t )|0≤SOC t ≤1,0≤v′ t ≤1,-1≤p′ t ≤1}

[0069] Where S represents the state, v' represents the standardized speed variable (real-time speed divided by the maximum speed), p' represents the standardized power demand (real-time power demand divided by the absolute value of the maximum power demand), and t represents time.

[0070] The action variable of reinforcement learning is the mode switching signal.

[0071] Reward function for reinforcement learning:

[0072]

[0073] In the formula, is the instantaneous fuel consumption, is the instantaneous power consumption, λ 1 , 2 , α is the penalty coefficient, high and low are the upper and lower limits of the SOC range.

[0074] 4. Discrete reinforcement learning state variables (vehicle speed, battery SOC, vehicle power demand), calculate the optimal mode at each state point based on the trained intelligent agent, and establish a strategy for controlling the switching of series hybrid modes for use in the vehicle controller of actual hybrid vehicles.

[0075] In summary, the deep Q network reinforcement learning algorithm used in this technical solution is used to control the mode switching of the vehicle, which conforms to the discrete action characteristics of the algorithm, can simplify the action space of the reinforcement algorithm, and is conducive to improving the training speed; this technical solution converts the trained reinforcement learning agent into a rule-based energy management strategy, which can be directly used for the control of actual vehicles, thereby combining the advantages of self-learning of the reinforcement learning algorithm and the real-time performance of the rule-based energy management strategy. In addition, this technical solution can also be combined with vehicle network technology: the actual running vehicle uploads data to the cloud, the DQN algorithm is used to train the energy management strategy in the cloud, the trained strategy is extracted into rules, and then downloaded to the vehicle to complete the iterative update of the strategy.

Claims

1. A hybrid vehicle mode switching control method based on reinforcement learning, characterized in that: The following steps are involved: S1. Acquire historical driving data of the hybrid vehicle and pre-process the historical driving data; S2. Build a deep Q learning network and train the deep Q learning network based on the preprocessed historical driving data to obtain a trained intelligent agent; S3, extracting control rules from the agent; S4, applying the extracted control rules to the vehicle controller, collecting the actual driving data of the hybrid vehicle, combining the control rules, and outputting the optimal energy management strategy; S5, according to the optimal energy management strategy, correspondingly controlling the mode switching of the hybrid vehicle; The driving data includes vehicle speed and SOC information, and the driving data is specifically collected by a sensor; The specific process of preprocessing the historical driving data in step S1 is as follows: Model vehicle energy requirements; Input the historical vehicle speed data into the vehicle energy demand model, and output the corresponding power demand data; The vehicle energy demand model is specifically: Among them, F req is the driving force required by the car, Cd is the drag coefficient, A is the frontal area, v is the vehicle speed, m is the vehicle mass, g is the acceleration of gravity, θ is the road slope, f is the rolling resistance coefficient, and t is the time; The state of the deep Q learning network in step S2 includes the vehicle power demand, the battery system SOC and the vehicle speed, the action of the deep Q learning network is specifically the mode switching of the vehicle, and the reward function of the deep Q learning network reinforcement learning is specifically the instantaneous fuel consumption, instantaneous power consumption, and the penalty when the actual SOC exceeds the set operating range; The state variables of the deep Q learning network are specifically: S t ∈{(SOC t ,v’ t ,p’ t )|0≤SOC t ≤1,0≤v’ t ≤1,-1≤p’ t ≤1} Among them, S is the state, v' is the standardized speed variable, p' is the standardized power demand, and t is the time; The standardized speed variable is specifically obtained by dividing the real-time speed by the maximum speed; The standardized power demand is specifically obtained by dividing the real-time power demand by the absolute value of the maximum power demand; The reward function of the deep Q learning network is specifically: in, is the instantaneous fuel consumption, is the instantaneous power consumption, λ1, λ2, α are penalty coefficients, high and low are the upper and lower limit marks of the SOC range respectively.

2. A hybrid vehicle mode switching control method based on reinforcement learning according to claim 1, characterized in that: The step S2 specifically uses the vehicle driving data and the vehicle energy demand model as the external environment of the reinforcement learning agent, and trains the deep Q learning network through the interaction between the agent and the environment.

3. The hybrid vehicle mode switching control method based on reinforcement learning according to claim 1, characterized in that: The control rule is specifically a table with state variables as indexes.