Hybrid electric vehicle layered reinforcement learning energy management method giving consideration to energy source life and energy consumption optimization

Abstract

The invention relates to the technical field of hybrid electric vehicle energy management, in particular to a hybrid electric vehicle layered reinforcement learning energy management method considering energy source life and energy consumption optimization. The method comprises the following steps: establishing an energy management system model and an energy source degradation model; energy source physical characteristics are analyzed, demanded power is processed in a layered mode, and an energy management learning strategy based on historical data is constructed; and energy management is implemented by combining the demand power change, the lithium battery, the super capacitor SOC and the degeneration degree of the energy source. Fuzzy filtering is adopted to shunt required power, a high-frequency peak value is borne by a super capacitor, and low-and-medium-frequency power is proportionally distributed by a lithium battery and a fuel battery according to the degradation degree; energy source degradation and deep reinforcement learning are fused, related parameter weighting is introduced into the ECMS, and lithium battery charging and discharging current is restrained. According to the method, comprehensive energy consumption can be reduced, energy source degradation is slowed down, the working condition application range is widened, overcharge and overdischarge of the lithium battery are avoided, the service life of the lithium battery is prolonged, and the SOC of the energy storage element is stabilized in a reasonable interval.

Description

Technical Field

[0001] This invention relates to the field of hybrid electric vehicle energy management technology, specifically a hierarchical reinforcement learning energy management method for hybrid electric vehicles that balances energy source lifespan and energy consumption optimization. Background Technology

[0002] To address the air pollution and petroleum resource depletion caused by traditional gasoline and diesel vehicles, vehicle electrification and intelligentization have gradually become the most effective solutions. Fuel cell hybrid vehicles, with their advantages of being pollution-free, zero-emission, and having long driving range, have become the subject of much research. With the widespread adoption of new energy vehicles, the degradation of their energy source performance has also become a concern for researchers. To fundamentally improve the energy efficiency of hybrid vehicles while mitigating energy source performance degradation, it is necessary to increase the hybridization of the vehicle's powertrain. However, this often increases the complexity of hybrid vehicle energy management. In existing technologies, the energy sources of hybrid vehicles generally include fuel cells, lithium batteries, and supercapacitors. Due to the relatively simple energy management system, comprehensive management of fuel cell and lithium battery performance degradation and motor load power demand is rarely implemented, thus limiting the overall performance of hybrid vehicles. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a hierarchical reinforcement learning energy management method for hybrid electric vehicles that balances energy source lifespan and energy consumption optimization.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a hierarchical reinforcement learning energy management method for hybrid electric vehicles that balances energy source lifetime and energy consumption optimization, the method comprising the following steps: S1. Establish a model for the degradation of energy sources in hybrid electric vehicles; S2. Constructing a hierarchical energy management method for hybrid electric vehicles based on energy source degradation models and historical driving data; S3. Distribute the energy source output power of hybrid electric vehicles using a tiered energy management method.

[0005] As a further optimization of the hierarchical reinforcement learning energy management method for hybrid electric vehicles that balances energy source lifespan and energy consumption optimization, the specific steps of S1 are as follows: S101. Establish a fuel cell model: ; In the formula, For fuel cell voltage, It is the number of fuel cell plates in the fuel cell stack. It is the voltage of a single fuel cell. It is the activation loss voltage of a single fuel cell. It is the internal resistance loss voltage of a single fuel cell; S102. Establish a fuel cell degradation model: ; In the formula, The percentage represents the health status of the fuel cell. This represents the range of fuel cell performance from start to finish. It is the acceleration coefficient. , , and These represent the performance degradation rates for different driving modes. and These are the number of load changes per hour, the number of start-stop cycles, the idling time, and the high-power load time; S103. Establish a lithium battery model: ; ; In the formula, It is the initial SoC for lithium batteries. It is the current of the lithium battery. It is the state-of-the-art (BOD) selectivity factor for lithium batteries; β is negative during charging and positive during discharging. This refers to the rated capacity of the lithium battery. and These are when the lithium battery SoC is The open-circuit voltage and internal resistance of a lithium battery. It refers to the electrical power of the lithium battery; The nominal capacity of a battery gradually decreases as lithium batteries degrade, therefore: ; In the formula, This indicates the health status of the lithium battery.

[0006] S104. Establish a lithium battery degradation model: ; In the formula, It represents the percentage of lithium battery capacity loss. It refers to the temperature (K) of the lithium battery. It is a power-law factor. It is the total ampere-hour throughput. It is activation energy. It is the universal gas constant (8.314 J / (mol·K)). It is the current rate. and All are model parameters; The state of health (SoH) of a lithium battery is represented as follows: ; S105. Establish a supercapacitor model: ; ; In the formula, and These are the maximum and minimum output voltages of the supercapacitor. It is the equivalent internal resistance of the supercapacitor. It is the electrical power of the supercapacitor; S106. Establish a power distribution model for hybrid electric vehicles: ; In the formula, It is the power demand of the load. It refers to the power of the fuel cell. It refers to the power of the lithium battery. It is the power of the supercapacitor. and The conversion efficiencies of unidirectional DC-DC and bidirectional DC-DC converters, respectively, when the lithium battery and supercapacitor are discharging. and It is positive; when the lithium battery and supercapacitor are charging, and It is negative.

[0007] As a further optimization of the hierarchical reinforcement learning energy management method for hybrid vehicles that balances energy source lifespan and energy consumption optimization, the specific steps of S2 are as follows: S201. Based on historical driving data, the demand power transition probability matrix is ​​obtained by analysis and calculation using the nearest neighbor method. S202. The optimal energy management method for hybrid electric vehicles is obtained by using an adaptive deep reinforcement learning program to learn the SoC maintenance of the energy storage system of the hybrid electric vehicle, the total equivalent hydrogen consumption of each energy source, and the reduction of the degradation degree of the energy source.

[0008] As a further optimization of the hierarchical reinforcement learning energy management method for hybrid vehicles that balances energy source lifespan and energy consumption optimization, the specific steps of S3 are as follows: S301. Obtain the load power requirement of the drive motor based on the change amplitude of the vehicle's accelerator pedal. ; S302, Power requirements based on load A saturation device is used to achieve current shunting between positive and negative power, where the negative power... Powered by supercapacitors, positive power It is powered by three energy sources. S303, based on S106, obtains the weighted SoC of the energy storage system through the integral method; S304, positive power By using a weighted SoC as an input variable to a fuzzy logic system, an adjustable frequency can be obtained. ; S305, Adjustable frequency and positive power The input is fed into an adaptive fuzzy low-pass filter for positive power. Frequency division decoupling outputs high-frequency positive power. and mid-low frequency positive power Among them, high-frequency positive power Provided by supercapacitors, mid-to-low frequency positive power It is supplied by both fuel cells and lithium batteries; S306, Based on the optimal hybrid electric vehicle energy management method and low-frequency positive power This yields the output power ratio of the fuel cell and lithium battery corresponding to the required power. S307, based on the output power ratio and actual power of S306, obtains the corresponding actual power of fuel cells and lithium batteries.

[0009] As a further optimization of the hierarchical reinforcement learning energy management method for hybrid vehicles that balances energy source lifespan and energy consumption optimization, the specific steps of S305 are as follows: S305.1, Adjustable frequency and positive power The input is fed into an adaptive fuzzy low-pass filter to obtain high-frequency positive power. and mid-low frequency positive power ; S305.2, Based on mid-low frequency positive power The study uses the maintenance of the energy storage system's SoC, the degree of energy source degradation, and the reduction of the vehicle's overall total hydrogen consumption as learning objectives to design a multi-objective optimization problem: ; In the formula, t represents time. Total hydrogen consumption, This represents the penalty coefficient for fuel cells. For the hydrogen consumption of fuel cells, As an equivalent factor for battery degradation, This is the penalty coefficient for lithium batteries. This is the equivalent hydrogen consumption of a lithium battery. This is the penalty factor for supercapacitors. This is the equivalent hydrogen consumption of the supercapacitor. The equivalent hydrogen consumption for lithium battery degradation. For the deviation of the current SoC, For reference SoC values; S305.3, Using multi-objective optimization problems as the reward function for deep reinforcement learning algorithms: ; In the formula, η is the penalty coefficient; S305.4. The Reward function obtained in S305.3 is added to the deep learning algorithm to obtain an improved deep learning algorithm. Real-time energy management of hybrid vehicles is performed based on the improved deep learning algorithm.

[0010] The beneficial effects are: 1) This invention comprehensively considers and applies the energy source performance degradation, drive motor load power demand, and energy storage system weighted SOC to the energy management of hybrid vehicles, resulting in better overall performance of hybrid vehicles. On the other hand, due to the adoption of a deep reinforcement learning-based method, this energy management method has strong universality and real-time performance for different operating conditions.

[0011] 2) Based on the physical characteristics of each energy source, this invention first divides the required power before allocating power. The supercapacitor is responsible for the high-frequency power, while the remaining medium and low-frequency power is jointly handled by the fuel cell and lithium battery. Furthermore, during the power allocation process, the maximum charging and discharging current of the lithium battery is constrained to prevent overcharging and over-discharging, thereby maximizing the service life of the power source.

[0012] 3) This invention adopts the idea of ​​minimizing equivalent consumption, and uses the SOC of the energy storage system, the overall equivalent hydrogen consumption of the vehicle, and the degree of performance degradation of the energy source as learning objectives to design a multi-objective optimization function model. On the basis of slowing down the aging of the energy source, it realizes the coordinated regulation of the energy management system of fuel cell hybrid electric vehicles, which improves the economy of the vehicle and extends the service life of the energy source. Attached Figure Description

[0013] Figure 1 This is a structural diagram of the hybrid electric vehicle powertrain system of the present invention;

[0014] Figure 2 This is a schematic diagram of the energy management method of the present invention;

[0015] Figure 3 This is a structural diagram of the present invention combining energy performance degradation and deep reinforcement learning. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figures 1 to 3 During operation, hybrid electric vehicles are powered by hydrogen fuel cells, lithium batteries, and supercapacitors. The hydrogen fuel cell system is the primary power source. The supercapacitor provides or absorbs instantaneous peak power that the hydrogen fuel cell and lithium batteries cannot provide or absorb, while the lithium batteries provide or absorb excess power. The hydrogen fuel cell is connected to a hydrogen storage tank and a DC / DC fuel cell converter. The lithium battery and supercapacitor are respectively connected to a bidirectional DC / DC lithium battery converter and a bidirectional DC / DC supercapacitor converter. The DC / DC fuel cell converter, bidirectional DC / DC lithium battery converter, and bidirectional DC / DC supercapacitor converter are connected in parallel to a DC bus. The DC bus is connected to the traction motor via a DC / AC converter.

[0018] A hierarchical reinforcement learning-based energy management method for hybrid electric vehicles that balances energy source lifetime and energy consumption optimization includes the following steps:

[0019] S1. Establish a model for the degradation of energy sources in hybrid electric vehicles;

[0020] S2. Constructing a hierarchical energy management method for hybrid electric vehicles based on energy source degradation models and historical driving data;

[0021] S3. Distribute the energy source output power of hybrid electric vehicles using a tiered energy management method.

[0022] The specific steps of S1 are as follows.

[0023] S101. Establish a fuel cell model: ; In the formula, For fuel cell voltage, It is the number of fuel cell plates in the fuel cell stack. It is the voltage of a single fuel cell. It is the activation loss voltage of a single fuel cell. It is the internal resistance loss voltage of a single fuel cell;

[0024] S102. Establish a fuel cell degradation model: ; In the formula, The percentage represents the health status of the fuel cell. This represents the range of fuel cell performance from start to finish. It is the acceleration coefficient. , , and These represent the performance degradation rates for different driving modes. and These are the number of load changes per hour, the number of start-stop cycles, the idling time, and the high-power load time;

[0025] S103. Establish a lithium battery model: ; ; In the formula, It is the initial SoC for lithium batteries. It is the current of the lithium battery. It is the state-of-the-art (BOD) selectivity factor for lithium batteries; β is negative during charging and positive during discharging. This refers to the rated capacity of the lithium battery. and These are when the lithium battery SoC is The open-circuit voltage and internal resistance of a lithium battery. It refers to the electrical power of the lithium battery; The nominal capacity of a battery gradually decreases as lithium batteries degrade, therefore: ; In the formula, This indicates the health status of the lithium battery.

[0026] S104. Establish a lithium battery degradation model: ; In the formula, It represents the percentage of lithium battery capacity loss. It refers to the temperature (K) of the lithium battery. It is a power-law factor. It is the total ampere-hour throughput. It is activation energy. It is the universal gas constant (8.314 J / (mol·K)). It is the current rate. and All are model parameters; The state of health (SoH) of a lithium battery is represented as follows: ;

[0027] S105. Establish a supercapacitor model: ; ; In the formula, and These are the maximum and minimum output voltages of the supercapacitor. It is the equivalent internal resistance of the supercapacitor. It is the electrical power of the supercapacitor;

[0028] S106. Establish a power distribution model for hybrid electric vehicles: ; In the formula, It is the power demand of the load. It refers to the power of the fuel cell. It refers to the power of the lithium battery. It is the power of the supercapacitor. and The conversion efficiencies of unidirectional DC-DC and bidirectional DC-DC converters, respectively, when the lithium battery and supercapacitor are discharging. and It is positive; when the lithium battery and supercapacitor are charging, and It is negative.

[0029] The specific steps of S2 are as follows.

[0030] S201. Based on historical driving data, the demand power transition probability matrix is ​​obtained through analysis and calculation using the nearest neighbor method.

[0031] S202. Using the hybrid electric vehicle's energy storage system SoC, the total equivalent hydrogen consumption of each energy source, and the degree of degradation of the energy sources as learning objectives, the optimal energy management method for hybrid electric vehicles is obtained through an adaptive deep reinforcement learning program.

[0032] Energy storage systems include supercapacitors and lithium batteries.

[0033] The established energy management model for hybrid electric vehicles needs to consider factors such as changes in demand power, the state of charge (SOC) of lithium batteries and supercapacitors, and the degree of energy source degradation in order to better manage the energy of hybrid electric vehicles.

[0034] The specific steps for S3 are as follows.

[0035] S301. Obtain the load power requirement of the drive motor based on the change amplitude of the vehicle's accelerator pedal. .

[0036] S302, Power requirements based on load A saturation device is used to achieve current shunting between positive and negative power, where the negative power... Powered by supercapacitors, positive power It is powered by three energy sources.

[0037] S303, based on S107, obtains the weighted SoC of the energy storage system through the integral method.

[0038] S304, positive power By using a weighted SoC as an input variable to a fuzzy logic system, an adjustable frequency can be obtained. .

[0039] Fuzzy logic systems for positive power After fuzzification, fuzzy logic processing, and defuzzification with the weighted SoC, an adjustable frequency can be obtained. Input power demand The fuzzy sets representing the power ranges are {negative large region (NB), negative medium region (NM), negative small region (NS), zero region (ZE), positive small region (PS), positive medium region (PM), positive large region (PB)}, and the fuzzy sets representing the weighted SoC of the hybrid vehicle energy storage system are {small region (S), relatively small region (RS), medium region (M), relatively large region (RB), large region (B)}; output adjustment frequency The fuzzy sets represent the ranges of {small region (S), smaller region (RS), medium region (M), larger region (RB), and large region (B)}.

[0040] S305, Adjustable frequency and positive power The input is fed into an adaptive fuzzy low-pass filter for positive power. Frequency division decoupling outputs high-frequency positive power. and mid-low frequency positive power Among them, high-frequency positive power Provided by supercapacitors, mid-to-low frequency positive power It is supplied by both fuel cells and lithium batteries.

[0041] S301 to S305 are actually for the required power Perform power stratification, separating high-frequency positive power Power is supplied by supercapacitors, which not only makes good use of the physical characteristics of supercapacitors but also greatly improves the lifespan of fuel cells and lithium batteries. As for the allocation of low- and medium-frequency power, a multi-objective optimization problem needs to be designed by taking the degree of energy source degradation, the energy storage system SoC, and the vehicle's overall total hydrogen consumption as learning objectives.

[0042] The specific steps for S305 are as follows.

[0043] S305.1, Adjustable frequency and positive power The input is fed into an adaptive fuzzy low-pass filter to obtain high-frequency positive power. and mid-low frequency positive power .

[0044] S305.2, Based on mid-low frequency positive power The study uses the maintenance of the energy storage system's SoC, the degree of energy source degradation, and the reduction of the vehicle's overall total hydrogen consumption as learning objectives to design a multi-objective optimization problem: ; In the formula, t represents time. Total hydrogen consumption, This represents the penalty coefficient for fuel cells. For the hydrogen consumption of fuel cells, As an equivalent factor for battery degradation, This is the penalty coefficient for lithium batteries. This is the equivalent hydrogen consumption of a lithium battery. This is the penalty factor for supercapacitors. This is the equivalent hydrogen consumption of the supercapacitor. The equivalent hydrogen consumption for lithium battery degradation. For the deviation of the current SoC, For reference SoC values.

[0045] S305.3, Using multi-objective optimization problems as the reward function for deep reinforcement learning algorithms: ; In the formula, η is the penalty coefficient;

[0046] Since the goal of reinforcement learning is to maximize expected reward, the optimization problem designed is to minimize the overall equivalent hydrogen consumption of the vehicle.

[0047] S305.4. The Reward function obtained in S305.3 is added to the deep learning algorithm to obtain an improved deep learning algorithm. Real-time energy management of hybrid vehicles is performed based on the improved deep learning algorithm.

[0048] S306, Based on the optimal hybrid electric vehicle energy management method and low-frequency positive power This yields the output power ratio of the fuel cell and lithium battery corresponding to the required power.

[0049] S307, based on the output power ratio and actual power of S306, obtains the corresponding actual power of fuel cells and lithium batteries.

[0050] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A hybrid electric vehicle hierarchical reinforcement learning energy management method that balances energy source lifetime and energy consumption optimization, characterized in that, The method comprises the following steps: S1, establishing a hybrid electric vehicle energy source degradation model; S2, constructing a hierarchical energy management method of the hybrid electric vehicle based on the energy source degradation model and historical driving data of the hybrid electric vehicle; S3, distributing the output power of the energy source of the hybrid electric vehicle by using the hierarchical energy management method.

2. The hybrid electric vehicle hierarchical reinforcement learning energy management method with consideration of energy source lifetime and energy consumption optimization according to claim 1, wherein, The method for establishing the energy management system model of the hybrid electric vehicle comprises: S101, establishing a fuel cell model: ； wherein is the fuel cell voltage, is the number of fuel cell sheets in the fuel cell stack, is the single sheet fuel cell voltage, is the activation loss voltage of the single sheet fuel cell, is the internal resistance loss voltage of the single sheet fuel cell; S102, establishing a fuel cell degradation model: ； wherein, is the fuel cell health state (percentage), represents the range of fuel cell performance from start to end, is the acceleration factor, , , and are the performance degradation rates for different driving modes, respectively, and are the number of load changes per hour, the number of start-stop, the idling time and the high-power load time, respectively. S103, establishing a lithium battery model: ； ； wherein, is the initial SoC of the lithium battery, is the current of the lithium battery, is the charge-discharge state selection coefficient of the lithium battery, negative when charging and positive when discharging, is the rated capacity of the lithium battery, and are the open-circuit voltage and internal resistance of the lithium battery, respectively, when the SoC of the lithium battery is , is the electric power of the lithium battery; The nominal capacity of the battery gradually decreases with the degradation of the lithium battery, so: ； In the formula, is the state of health of the lithium battery. S104, establishing a lithium battery degradation model: ； wherein, is the percentage of lithium battery capacity loss, is the lithium battery temperature (K), is the power law factor, is the total ampere-hour throughput, is the activation energy, is the universal gas constant (8.314 J / (mol·K)), is the current rate, and are model parameters; The state of health SoH of the lithium battery is expressed as: ； S105, establishing a super capacitor model: ； ； wherein, and Vmax and Vmin are respectively the maximum and minimum output voltage of the supercapacitor, R is the equivalent internal resistance of the supercapacitor, P is the electric power of the supercapacitor; S106, establishing a hybrid electric vehicle power distribution model: ； wherein, is the load demand power, is the fuel cell power, is the lithium battery power, is the supercapacitor power, and are the conversion efficiencies of the unidirectional DC-DC and bidirectional DC-DC, respectively, when the lithium battery and supercapacitor are discharging, and are positive; when the lithium battery and supercapacitor are charging, and are negative.

3. The hybrid electric vehicle hierarchical reinforcement learning energy management method with consideration of energy source lifetime and energy consumption optimization according to claim 1, characterized in that, In S2, a method for constructing a hybrid electric vehicle energy management system based on the minimum equivalent hydrogen consumption is as follows according to the physical characteristics of the fuel cell, super capacitor and lithium battery in the fuel cell hybrid electric vehicle: S201, based on historical driving data, the demand power transfer probability matrix is obtained by analyzing and converting by using the nearest neighbor method; S202, the SoC of the energy storage system of the hybrid electric vehicle is maintained, and the total equivalent hydrogen consumption of each energy source and the degradation degree of the energy source are reduced as the learning target, and the optimal hybrid electric vehicle energy management method is obtained through the adaptive deep reinforcement learning program.

4. The hybrid electric vehicle hierarchical reinforcement learning energy management method with consideration of energy source lifetime and energy consumption optimization according to claim 1, wherein, In S3, the deep reinforcement learning algorithm used in the application is DDPG (Deep Deterministic Policy Gradient). As a further optimization of the above-mentioned hybrid electric vehicle hierarchical reinforcement learning energy management method considering the life and energy consumption optimization of the energy source: the specific steps of S3 are S301, obtaining a load demand power of the driving motor according to a change amplitude of an accelerator pedal of the vehicle ; S302、According to the load demand power The saturation device is used to realize the shunt between positive and negative power, wherein the negative power is borne by the super capacitor, and the positive power is borne by the three energy sources together; S303, based on S106, the weighted SoC of the energy storage system is obtained by integral method; S304, positive power By using a weighted SoC as an input variable to a fuzzy logic system, an adjustable frequency can be obtained. ; S305, the adjustable frequency and positive power input into the adaptive fuzzy low-pass filter, the positive power decoupling, output high-frequency positive power and low-frequency positive power , wherein the high-frequency positive power provided by the super capacitor, low-frequency positive power provided by the fuel cell and lithium battery together; S306、based on the optimal hybrid electric vehicle energy management method and the low-frequency positive power , get the output power ratio of the fuel cell and the lithium battery corresponding to the demand power; S307, based on the output power ratio and the actual power of S306, the actual power of the fuel cell and the lithium battery is obtained.

5. The hybrid electric vehicle hierarchical reinforcement learning energy management method with consideration of energy source lifetime and energy consumption optimization according to claim 1, wherein, As a further optimization of the above-mentioned hybrid electric vehicle hierarchical reinforcement learning energy management method considering the life and energy consumption optimization of the energy source: the specific steps of S305 are: S305.1, the adjustable frequency and positive power input to an adaptive fuzzy low-pass filter, obtaining high-frequency positive power and mid-low frequency positive power ; S305.2, positive power based on mid-low frequency To maintain the SoC of the energy storage system, the degradation level of the energy source, and the overall total hydrogen consumption of the vehicle as learning goals, a multi-objective optimization problem is designed: ； where t is time, is the total hydrogen consumption, is the penalty factor for the fuel cell, is the hydrogen consumption for the fuel cell, is the equivalent factor for the battery degradation, is the penalty factor for the lithium battery, is the equivalent hydrogen consumption for the lithium battery, is the penalty factor for the supercapacitor, is the equivalent hydrogen consumption for the supercapacitor, is the equivalent hydrogen consumption for the lithium battery degradation, is the deviation for the current SoC, is the reference SoC value; S305.3, the multi-objective optimization problem is taken as the Reward function of the deep reinforcement learning algorithm: ； In the formula, η is the penalty coefficient; S305.4, the Reward function obtained in S305.3 is added to the deep learning algorithm to obtain an improved deep learning algorithm, and the hybrid electric vehicle is managed in real time based on the improved deep learning algorithm.

Images