A plug-in hybrid vehicle energy management method based on improved multi-objective DDPG

Abstract

The present invention discloses a plug-in hybrid vehicle energy management method based on improved multi-objective DDPG, comprising the following steps: establishing an energy management system model of a plug-in hybrid vehicle, including a vehicle longitudinal dynamics model, an engine fuel consumption model, a battery equivalent circuit model, a battery life model and a drive / generator model; obtaining the state information of the plug-in hybrid vehicle in actual driving and inputting it into the energy management system model; using the energy management system model as an intelligent body of IMDDPG, configuring the reward function of IMDDPG according to the cumulative fuel consumption of the engine, the offset degree of the battery SOC and the change of the battery health status, using IMDDPG to perform multi-objective optimization on the energy management system model to obtain a trained reinforcement learning model; inputting the initial state information and driving conditions into the reinforcement learning model to obtain an energy management strategy during driving. The present invention comprehensively considers fuel economy and battery life, and improves the optimality and universality of the strategy while ensuring real-time performance.

Description

Technical Field

[0001] The present invention relates to the field of hybrid vehicle energy management, and in particular to a plug-in hybrid vehicle energy management method based on an improved multi-objective DDPG. Background Art

[0002] Plug-in hybrid electric vehicles (PHEVs) have the advantages of both pure electric vehicles and hybrid electric vehicles. They not only solve the shortcomings of pure electric vehicles such as short driving range and imperfect infrastructure, but also further tap the potential of the engine as a driving device. They have become the mainstream research and development direction of major traditional automobile manufacturers in the process of electrification transformation at this stage. PHEVs have a complex structure and multiple driving modes. Therefore, how to reasonably allocate the power requirements of the engine and motor, reasonably switch modes and gears, and achieve optimal energy management is a key technology for PHEVs. At present, the energy management strategies of PHEVs can be divided into rule-based and optimization-based. However, the existing energy management strategies generally have shortcomings such as weak real-time performance, complex calculations, weak adaptability, and unsatisfactory optimization performance. In addition, the battery life is also a key technology that restricts the development of PHEVs. Therefore, considering the economy of PHEVs and the life of batteries, it is of great research value and application value to carry out research on PHEV energy management strategies that are both real-time, adaptable and optimal.

[0003] With the popularization and development of artificial intelligence, energy management strategies based on deep reinforcement learning algorithms have attracted extensive attention from researchers due to their real-time and adaptability characteristics. However, the existing deep reinforcement learning energy management strategies for PHEVs all use multiple objectives (such as fuel consumption and SOC deviation) as a single objective through weight factors as a reward function, and most of them do not take the battery life into consideration.

[0004] Patent application CN114801897A discloses a fuel cell hybrid system energy management method based on the DDPG algorithm. For a hybrid system of dual-stack fuel cells and lithium batteries, a DDPG algorithm model is established, and the algorithm model and the dynamic model are matched for parameters; the state, action and reward of the algorithm model are set; and an optimal objective function based on driving cost is constructed for the hybrid system to reduce the loss of fuel cells and lithium batteries and extend their life. This method has the following shortcomings:

[0005] (1) The various costs of freight trucks are coupled into the objective function, which reduces the diversity of the objective function.

[0006] (2) The four objectives of hydrogen consumption, fuel cell loss, lithium battery degradation and SoC change are coupled into one objective through weight factors, which increases the computational cost of determining the weight factors. At the same time, the weight factors need to be re-formulated according to different models and different objective functions, which is inefficient and reduces the adaptability of the algorithm.

[0007] (3) The optimal solution of multi-objective optimization is often not unique, but a solution set composed of multiple mutually dominant optimal solutions. Therefore, coupling multiple objectives into one objective cannot achieve global multi-objective optimization and reduces the optimality of the solution. Summary of the invention

[0008] The technical problem to be solved by the present invention is to propose a PHEV energy management strategy that integrates real-time, adaptability and optimality to a certain extent, based on continuous action, while achieving optimal PHEV fuel economy and no large deviation in SOC, taking the battery life into consideration, thereby achieving multi-objective PHEV energy management.

[0009] In view of the technical problems existing in the prior art, the present invention provides a plug-in hybrid vehicle energy management method based on an improved multi-objective DDPG, which comprehensively considers fuel economy and battery life, improves the optimality and universality of the strategy while ensuring real-time performance.

[0010] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0011] A plug-in hybrid vehicle energy management method based on an improved multi-objective DDPG comprises the following steps:

[0012] Establish an energy management system model for plug-in hybrid vehicles, including the vehicle longitudinal dynamics model, engine fuel consumption model, battery equivalent circuit model, battery life model, and drive / generator model;

[0013] Acquiring state information of the plug-in hybrid vehicle during actual driving, and inputting the state information into the energy management system model;

[0014] The energy management system model is used as an intelligent agent of IMDDPG, and the reward function of IMDDPG is configured according to the cumulative fuel consumption of the engine, the deviation degree of the battery SOC, and the change of the battery health status. The speed, acceleration, and battery SOC of the plug-in hybrid vehicle are used as the state variables of IMDDPG, and the required power of the plug-in hybrid vehicle is used as the action variable of IMDDPG. IMDDPG is used to perform multi-objective optimization on the energy management system model to obtain a trained reinforcement learning model;

[0015] The initial state information and driving conditions are input into the reinforcement learning model to obtain an energy management strategy during driving.

[0016] Furthermore, using IMDDPG to perform multi-objective optimization on the energy management system model includes the following steps:

[0017] Obtaining the state information s of the energy management system model t Input into the Actor evaluation network to get the corresponding action a t And input the energy management system model, and obtain the corresponding reward r(s) under the influence of the environment t ,a t ) and the next state information s t+1 , the current sample data e t =(s t ,a t ,r(s t ,a t ),s t+1 ) is stored in the experience pool, and this step is repeated until the number of sample data in the experience pool meets the requirement;

[0018] Randomly select samples from the experience pool and write the state information s of the selected samples t Input into the Actor evaluation network to get the corresponding action a t , the status information s t and the corresponding action a t Input into the Critic evaluation network, solve the Pareto optimal frontier to obtain the cumulative reward Q(s) corresponding to the selected sample t ,a t |θ), calculate the cumulative reward Q(s) t ,a t |θ) to obtain the loss function and back-propagate it, and use the gradient ascent method to update the parameters in the Actor evaluation network;

[0019] The state information s of the sample to be selected t Input into the updated Actor evaluation network to obtain the updated corresponding action a t , the status information s t and the corresponding action a after update t Input into the Critic evaluation network, solve the Pareto optimal frontier to obtain the updated cumulative reward Q(s) corresponding to the selected sample t ,a t |θ);

[0020] The next state information s of the sample to be selected t+1 Input into the Actor target network to get the corresponding action a t+1 , the status information s t+1 and the corresponding action a t+1Input the target network of Critic and solve the Pareto optimal frontier to obtain the selected sample in the state information s t+1 and action a t+1 The cumulative reward Q(s) under t+1 ,a t+1 |θ'), calculate the cumulative reward Q(s t+1 ,a t+1 |θ') to obtain the loss function and back-propagate it, and use the gradient descent method to update the parameters in the Critic evaluation network;

[0021] Update the parameters of the Actor evaluation network and the Critic evaluation network to the Actor target network and the Critic target network, and return the state information s of the energy management system model t The steps input into the Actor evaluation network until the number of cycles meets the requirement.

[0022] Furthermore, before updating the parameters of the Actor evaluation network and the Critic evaluation network to the Actor target network and the Critic target network, it is also determined whether the difference between the number of cycles of this cycle and the number of cycles of the last update reaches a preset step length. If so, the parameters of the Actor evaluation network and the Critic evaluation network are updated to the Actor target network and the Critic target network. Otherwise, the execution returns to obtain the state information s of the energy management system model. t Input to the Actor Criteria Network.

[0023] Further, the steps of solving the Pareto optimal frontier include: according to the state information s t and the corresponding action a t Solve for the optimal action value function Q * (s t ,a t ) and randomly select an optimal action-value function on the Pareto optimal frontier as the maximum cumulative reward, or select the optimal action-value function with the smallest target as the maximum cumulative reward.

[0024] Furthermore, the optimal action value function Q * (s t ,a t ) is as follows:

[0025]

[0026] Among them, R t is the cumulative reward with discount, γ is the discount factor, γ∈[0,1], r iis the reward function at time i, i∈[t,T], and T is the termination time.

[0027] Furthermore, the longitudinal dynamics model of the whole vehicle is expressed as follows:

[0028]

[0029] Among them, F D is the driving force, P D is the driving power, T D is the driving torque, v is the vehicle speed, F R 、F A 、F G 、F A′ They are rolling resistance, air resistance, slope resistance and acceleration resistance during vehicle driving, A is the frontal area of ​​the vehicle, C D is the air resistance coefficient, ρ is the air density, c R is the rolling resistance coefficient, m is the total mass of the vehicle, g is the acceleration of gravity, θ is the road slope, δ is the vehicle rotation mass conversion coefficient, is the driving acceleration, r is the wheel radius;

[0030]

[0031] Among them, T EN , T EM1 and T EM2 are the torques of the plug-in hybrid vehicle’s engine, EM1 motor, and EM2 motor, ω EN ,ω EM1 and ω EM2 are the speeds of the plug-in hybrid vehicle engine, EM1 motor and EM2 motor, T Out is the output torque of the transmission, K1 and K2 are the gear ratios of the ring gear and sun gear of PG1 and PG2 of the plug-in hybrid vehicle, and i is the main reducer gear ratio.

[0032] Furthermore, the engine fuel consumption model expression is as follows:

[0033]

[0034] in, is the instantaneous fuel consumption of the engine, m f is the cumulative fuel consumption of the engine, T EN ,ω EN They are the torque and speed of the engine of the plug-in hybrid vehicle, respectively.

[0035] Furthermore, the battery equivalent circuit model expression is as follows:

[0036]

[0037] Among them, U Bat is the battery terminal voltage, I Bat is the battery current, U OC is the open circuit current, R Bat is the internal resistance of the battery, P Bat is the battery power, SOC(0) is the initial SOC value, Q Bat is the battery capacity.

[0038] Furthermore, the battery life model expression is as follows:

[0039]

[0040] Among them, Q Loss is the battery capacity attenuation, α and β are constant terms, E A is the activation energy, R is the molar gas constant, T K is the thermodynamic temperature of the environment, Ah is the ampere-hour flux, z is the power exponential factor, Q Bat is the battery capacity, I Bat is the battery current, EOL is the end of battery life, N is the total number of cycles, SOH is the health of the battery, and SOC is the state of charge of the battery.

[0041] Furthermore, the drive / generator model expression is as follows:

[0042] η EM =f(T EM ,ω EM )

[0043] Among them, T EM is the motor torque, ω EM is the motor speed, η EM is the corresponding motor efficiency;

[0044]

[0045] Among them, P EM is the motor mechanical power, P Bat,EM The power delivered from the battery to the motor.

[0046] Compared with the prior art, the advantages of the present invention are:

[0047] (1) The present invention comprehensively considers the economy of plug-in hybrid electric vehicles and the life of batteries, and realizes multi-objective optimization of the energy management strategy of plug-in hybrid electric vehicles.

[0048] (2) The present invention uses IMDDPG to optimize the energy management strategy of the plug-in hybrid vehicle based on continuous actions in actual driving, which overcomes the shortcoming that the energy management strategy based on discrete actions is difficult to achieve the optimal one and is more in line with the characteristics of actual driving.

[0049] (3) The energy management strategy based on IMDDPG improves the optimality and universality of the strategy through continuous learning while ensuring real-time performance. It gets rid of the dependence of previous energy management strategies on driving conditions, ensures the optimality under standard test conditions and actual driving conditions, and improves the adaptability of the strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of an embodiment of the present invention.

[0051] Figure 2 Schematic diagram of the power system structure of a plug-in hybrid vehicle.

[0052] Figure 3 This is the IMDDPG flow chart. DETAILED DESCRIPTION

[0053] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0054] DDPG is a single-objective reinforcement learning algorithm that can only equate multiple objectives to a single objective through weight factors. First of all, the determination of the weight factor requires prior knowledge or a lot of work, and in theory it is impossible to achieve global optimality. Therefore, the DDPG algorithm has many problems in solving practical problems of multi-objective optimization. Therefore, we propose an improved multi-objective DDPG algorithm - IMDDPG.

[0055] Assume that the continuous state set of the agent is S and the continuous action sequence is A. When the current state of the agent is s t ∈S, take action a t ∈A, the state of the agent is transferred to a new state s under the influence of the environment. t+1 ∈S, the immediate reward generated is r(s t ,a t ).

[0056] Deep reinforcement learning selects the agent's actions by maximizing the cumulative reward, that is, taking into account both immediate rewards and future rewards, and continuously improving the strategy π to maximize the cumulative reward. The strategy corresponding to the maximum cumulative reward is the optimal strategy π. * (a|s). The strategy π is a series of actions taken by the agent from the beginning to the end.

[0057] The agent state is st , take action a t The optimal action value function Q * (s t ,a t )for:

[0058]

[0059] Among them, R t is the cumulative reward with discount, γ is the discount factor, γ∈[0,1], r i is the reward at time i, i∈[t,T], T is the termination time. The optimal action value function Q * (s t ,a t ) follows the Bellman equation:

[0060] Q * (s t ,a t )=E[r(s t ,a t )+γQ * (s t+1 ,a t+1 )|s t ,a t ] (2)

[0062] like Figure 1 As shown, the main process of the IMDDPG algorithm is as follows:

[0063] (1) Figure 1 As shown in the dashed box ①, the current state information s of the agent t Input into the Actor Evaluation Network, the output is the corresponding action (i.e., utilization), or a randomly generated action (i.e., exploration), denoted as a t , and the action a t Input into the environment, and get reward r(s) through interaction with the environment t ,a t ) and the next state information s t+1 . Send the current status information t , selected action a t , the reward r(s t ,a t ) and the next state s t+1 Store it in the experience pool U. Then the state information s t+1 Input into the Actor evaluation network and repeat step (1) until a certain number of e are stored. t =(s t ,a t,r(s t ,a t ),s t+1 ), in this process, the parameters in the Actor evaluation network are not updated. Among them, the ε-greedy algorithm is used to achieve a balance between exploration and utilization when selecting actions, ensuring sufficient exploration and reasonable utilization:

[0064]

[0065] Among them, ε∈[0,1] is the exploration rate. The ε-greedy algorithm chooses exploration with a probability of ε and selects the action of the Actor evaluation network output with a probability of 1-ε. Therefore, in order to ensure the performance of the deep reinforcement learning algorithm and prevent it from falling into the local optimum, ε is generally set to a larger initial value to ensure sufficient exploration ability. As the iteration proceeds, the value of ε gradually decreases to ensure full utilization and accelerate the convergence of the algorithm.

[0066] (2) Figure 1 As shown in the dashed box ②, a sample e is randomly selected from the experience pool U. First, the state information s in e is t Input into the Actor evaluation network to get the corresponding action a t , and then the status information s t and the corresponding action a t Input into the Critic evaluation network together to obtain the state information s t and action a t The cumulative reward Q(s) under t ,a t |θ), θ represents the parameters of the Critic evaluation network. By finding Q(s t ,a t |θ) to obtain the loss function and back propagate. The calculation formula is:

[0067] L ω =E e～U [Q(s,a|θ)] (4)

[0068] Among them, Q(s t ,a t |θ) is the state information s t and action a t The cumulative reward under , ω is the parameter of the Actor evaluation network;

[0069] The gradient ascent method is used to calculate the gradients of these loss functions to update the parameters in the Actor evaluation network, that is, to obtain the state s through back propagation t Next Q(s t ,a t |θ) The action corresponding to the maximum expected value, first calculate the gradient:

[0070]

[0071] Among them, Q(s t ,a t |θ) is the state information s t and action a t The cumulative rewards under is the gradient of the cumulative reward function Q(s,a|θ) with respect to a, is the gradient of the action function a(s|ω) with respect to ω, and a(s|ω) is the output action function when the parameter of the Actor evaluation network is ω and the input is s;

[0072] Then use the gradient ascent method to update the parameters in the Actor evaluation network: Among them, α l The learning rate for updating the Actor evaluation network parameters. In this step, only the parameters in the Actor evaluation network are updated, and the parameters of the Critic evaluation network remain unchanged.

[0073] Since the reward function r in the single-objective DDPG algorithm is a numerical value, its cumulative reward Q is also a numerical value. Similarly, the expected value of the cumulative reward and the loss function are both numerical values. When improving the single-objective DDPG algorithm to a multi-objective DDPG algorithm, the reward function is improved from a single objective to a multi-objective by combining the Pareto theory, that is, from a numerical value to an array, and each numerical value in the array is a target of the reward function. After the improvement, there are two problems that need to be solved, one is how to select the maximum cumulative reward Q, and the other is how to calculate the loss function and perform back propagation.

[0074] First, regarding how to select the maximum cumulative reward Q, that is, how to compare the size of the cumulative reward Q, the Pareto optimal frontier is introduced to select the maximum cumulative reward Q.

[0075] The Pareto optimal frontier means that, assuming that there are i objective functions in the multi-objective problem, A and B are its two feasible solutions. If all the objective function values ​​of solution A are better than solution B, then solution A is said to be better than solution B, that is, solution A dominates solution B; if only some objective functions of solution A are better than solution B, then solution A and solution B are said to be indistinguishable, that is, solution A does not dominate solution B. If the objective function value of solution A is better than any other solution in the feasible space, then A is called the optimal solution; if no other solution is found in the feasible space that is better than solution A, then solution A is called the Pareto optimal solution. For multi-objective optimization problems, there is generally no optimal solution, but there are multiple Pareto optimal solutions, and all Pareto optimal solutions constitute the Pareto optimal frontier.

[0076] Therefore, when selecting the maximum cumulative reward Q, firstly, the Pareto optimal frontier of the optimal action value function is solved based on formula (1), and a rule is formulated on the Pareto optimal frontier to select the maximum cumulative reward Q, such as randomly selecting a Q as the maximum cumulative reward, or selecting a certain target with the smallest Q as the maximum cumulative reward, etc. It should be pointed out that solving the Pareto optimal frontier based on the objective function is a common method used by technicians in this field, and this solution does not involve the improvement of its specific calculation process, so its specific calculation process will not be repeated.

[0077] Secondly, how to find the loss function and perform back propagation. Since multiple targets generate multiple loss functions, the gradient is calculated based on multiple loss functions and back propagated.

[0078] (3) Figure 2 As shown in the dashed box ③, the state information s in the sample e sampled in (2) t Input into the Actor evaluation network to get the corresponding action a after the Actor evaluation network is updated t , and then the status information s t and the corresponding action a t The two are input into the Critic evaluation network together, and the Actor evaluation network is updated in the state information s t and action a t The cumulative reward Q(s) under t ,a t |θ).

[0079] (4) Figure 2 As shown in the dashed box ④, the next state information s in the sample e sampled in (2) is t+1 Input into the Actor target network to get the corresponding action a t+1 , and then the status information s t+1 and the corresponding action a t+1 Input them into the Critic target network together, and solve the Pareto optimal frontier according to the method described in step (2), and obtain the state information s t+1 and actions at +1 The cumulative reward Q(s) under t+1 ,a t+1 |θ'), where θ' is the parameter of the Critic target network.

[0080] (5) Figure 2 As shown in the dotted box ⑤, the loss function Loss is back-propagated. The loss function Loss is the mean square error (MSE), and the calculation formula is:

[0081]

[0082] Among them, E is the number of samples e sampled from the experience pool U, Q(s t+1 ,a t+1 |θ') is the state information s t+1 and action a t+1 The cumulative reward under t ,a t |θ) is the state information s t and action a t The cumulative reward under t ,a t ) is the status information s t The reward under , γ is the discount factor.

[0083] Then, the gradient descent method is used to calculate the gradients using these loss functions to update the parameters in the Critic evaluation network. First, the gradient is calculated:

[0084]

[0085] Among them, E is the number of samples e sampled from the experience pool U, Q(s t+1 ,a t+1 |θ') is the state information s t+1 and action a t+1 The cumulative reward under t ,a t |θ) is the state information s t and action a t The cumulative reward under t ,a t ) is the status information s t The reward under , γ is the discount factor, is the gradient of the cumulative reward function Q(s,a|θ) with respect to θ. Q(s,a|θ) is the cumulative reward function output when the parameter of the Critic evaluation network is θ, the input state is s, and the input action is a;

[0086] Then use the gradient descent method to update the parameters in the Critic evaluation network: Among them, α L The learning rate for updating the Critic evaluation network parameters.

[0087] (6) Loop through steps (1)-(5), and after each C steps, update the parameters of the Actor evaluation network and the Critic evaluation network to the Actor target network and the Critic target network.

[0088] In summary, the IMDDPG algorithm improves the reward function by combining Pareto theory and realizes multi-objective learning of the algorithm.

[0089] On this basis, this embodiment proposes a plug-in hybrid vehicle energy management method based on an improved multi-objective DDPG, such as Figure 2 As shown, the following steps are included:

[0090] S1) Establishing an energy management system model for a plug-in hybrid vehicle, including a vehicle longitudinal dynamics model, an engine fuel consumption model, a battery equivalent circuit model, a battery life model, and a drive / generator model;

[0091] S2) obtaining state information of the plug-in hybrid vehicle during actual driving, and inputting the state information into the energy management system model;

[0092] S3) The energy management system model is used as an intelligent agent of IMDDPG, and the reward function of IMDDPG is configured according to the cumulative fuel consumption of the engine, the deviation degree of the battery SOC, and the change of the battery health status. The speed, acceleration and battery SOC of the plug-in hybrid vehicle are used as the state variables of IMDDPG, and the required power of the plug-in hybrid vehicle is used as the action variable of IMDDPG. IMDDPG is used to perform multi-objective optimization on the energy management system model to obtain a trained reinforcement learning model; the initial state information and driving conditions are input into the reinforcement learning model to obtain the energy management strategy during driving.

[0093] For step S1, the power system structure of the plug-in hybrid vehicle is as follows: Figure 3 As shown, the engine drives the generator EM2 and the drive motor EM1 through the transmission mechanism. The generator EM2 charges the battery, and the battery supplies power to the drive motor EM1. The transmission mechanism also provides power to the drive wheels through the clutch C1, C2, the gear set PG1, PG2, the transmission and the final reducer. According to this structure, the energy management system model is constructed as follows:

[0094] (a) The longitudinal dynamics model of the whole vehicle is established, and the expression is as follows:

[0095]

[0096] Among them, F D is the driving force, P D is the driving power, T D is the driving torque, v is the vehicle speed, F R 、F A 、F G 、F A′ They are rolling resistance, air resistance, slope resistance and acceleration resistance during vehicle driving, A is the frontal area of ​​the vehicle, C D is the air resistance coefficient, ρ is the air density, c Ris the rolling resistance coefficient, m is the total mass of the vehicle, g is the acceleration of gravity, θ is the road slope, δ is the vehicle rotation mass conversion coefficient, is the driving acceleration, r is the wheel radius;

[0097]

[0098] Among them, T EN , T EM1 and T EM2 are the torques of the plug-in hybrid vehicle’s engine, EM1 motor, and EM2 motor, ω EN ,ω EM1 and ω EM2 are the speeds of the plug-in hybrid vehicle engine, EM1 motor and EM2 motor, T Out is the output torque of the transmission, K1 and K2 are the gear ratios of the ring gear and sun gear of PG1 and PG2 of the plug-in hybrid vehicle, and i is the main reducer gear ratio.

[0099] (b) Establishment of the engine fuel consumption model. The engine fuel consumption model is established by looking up and correcting the test data. The instantaneous fuel consumption of the engine can be regarded as a function of the engine torque and speed. The expression is as follows:

[0100]

[0101] in, is the instantaneous fuel consumption of the engine, m f is the cumulative fuel consumption of the engine, T EN ,ω EN They are the torque and speed of the engine of the plug-in hybrid vehicle, respectively.

[0102] (c) The battery equivalent circuit model is established. The internal resistance model is selected to equate the battery model to a circuit with an ideal voltage source and a resistor in series. The expression is as follows:

[0103]

[0104] Among them, U Bat is the battery terminal voltage, I Bat is the battery current, U OC is the open circuit current, R Bat is the internal resistance of the battery, P Bat is the battery power, SOC(0) is the initial SOC value, Q Bat is the battery capacity.

[0105] (d) Battery life model establishment: A semi-empirical model is used to establish a battery life model. Assuming that there is no difference between all battery cells in the battery pack and the operating temperature of the battery remains basically constant, the battery life model expression is as follows:

[0106]

[0107] Among them, Q Loss is the battery capacity attenuation, α and β are constant terms, E A is the activation energy, R is the molar gas constant, T K is the thermodynamic temperature of the environment, Ah is the ampere-hour flux, z is the power exponential factor, Q Bat is the battery capacity, I Bat is the battery current, EOL is the end of battery life, N is the total number of cycles, SOH is the health of the battery, and SOC is the state of charge of the battery.

[0108] (e) Drive / generator model. Both the drive motor EM1 and the generator EM2 are permanent magnet synchronous motors. The comprehensive efficiency of the motor and inverter can be expressed as a function of the motor torque and speed. The expression is as follows:

[0109] η EM =f(T EM ,ω EM ) (13)

[0111] Among them, T EM is the motor torque, ω EM is the motor speed, η EM is the corresponding motor efficiency.

[0112]

[0113] Among them, P EM is the motor mechanical power, P Bat,EM The power delivered from the battery to the motor.

[0114] For step S2, the status information mainly includes two parts: vehicle status information and battery status information, where:

[0115] Vehicle status information mainly includes vehicle mass, frontal area, road slope, ambient temperature, instantaneous vehicle speed, motor speed, motor efficiency, etc.

[0116] The battery status information mainly includes battery current, battery voltage, open circuit voltage, internal resistance, SOC (battery state of charge), battery end of life, etc.

[0117] For step S3, the configuration of IMDDPG is as follows:

[0118] Reward function: In this embodiment, the economy and battery life of the plug-in hybrid vehicle are optimized. Since the IMDDPG algorithm seeks the best by maximizing the cumulative reward, the starting value and the ending value of the battery SOC of the plug-in hybrid vehicle remain equal, both of which are the SOC set values, that is, SOC start =SOC end =SOC Target , then the economic performance index is the engine's cumulative fuel consumption mf and the degree to which the battery SOC deviates from the set value after the driving is completed D = (SOC end -SOC Target ) 2 , the battery life indicator is the change in battery health status ΔSOH.

[0119] In the energy management of plug-in hybrid vehicles, the cumulative fuel consumption of the engine is m f , the deviation degree of battery SOC D = (SOC end -SOC Target ) 2 The smaller the change in battery health status ΔSOH, the better. Therefore, the reward function is r = (-m f ,-D,-ΔSOH).

[0120] State variables: The vehicle speed, acceleration and battery SOC of the plug-in hybrid vehicle are taken as state variables, that is, s = {v, acc, SOC}.

[0121] Action variable: The purpose of the energy management strategy of plug-in hybrid vehicles is to achieve reasonable mode switching and gear switching according to the required power. The key is to determine the required power of the plug-in hybrid vehicle. Therefore, the required power P D As an action variable, that is, a={P D}.

[0122] Based on the above configuration, using IMDDPG to perform multi-objective optimization on the energy management system model includes the following steps:

[0123] S31) As described in step (1) of the IMDDPG algorithm, obtain the state information s of the energy management system model t Input into the Actor evaluation network to get the corresponding action a t And input the energy management system model, and obtain the corresponding reward r(s) under the influence of the environment t ,a t ) and the next state information s t+1 , the current sample data e t =(s t ,a t ,r(s t ,a t),s t+1 ) is stored in the experience pool, and this step is repeated until the number of sample data in the experience pool meets the requirement;

[0124] S32) As described in step (2) of the IMDDPG algorithm, randomly select a sample from the experience pool and store the state information s of the selected sample t Input into the Actor evaluation network to get the corresponding action a t , the status information s t and the corresponding action a t Input into the Critic evaluation network, solve the Pareto optimal frontier to obtain the cumulative reward Q(s) corresponding to the selected sample t ,a t |θ), calculate the cumulative reward Q(s) t ,a t |θ) to obtain the loss function and back-propagate it, and use the gradient ascent method to update the parameters in the Actor evaluation network;

[0125] S33) As described in step (3) of the IMDDPG algorithm, the state information s of the selected sample is t Input into the updated Actor evaluation network to obtain the updated corresponding action a t , the status information s t and the corresponding action a after update t Input into the Critic evaluation network, solve the Pareto optimal frontier to obtain the updated cumulative reward Q(s) corresponding to the selected sample t ,a t |θ);

[0126] S34) As described in step (4) of the IMDDPG algorithm, the next state information s of the selected sample is t+1 Input into the Actor target network to get the corresponding action a t+1 , the status information s t+1 and the corresponding action a t+1 Input the target network of Critic and solve the Pareto optimal frontier to obtain the selected sample in the state information s t+1 and action a t+1 The cumulative reward Q(s) under t+1 ,a t+1 |θ'), as described in step (5) of the IMDDPG algorithm, calculate the cumulative reward Q(s t+1 ,a t+1 |θ') to obtain the loss function and back-propagate it, and use the gradient descent method to update the parameters in the Critic evaluation network;

[0127] S35) As described in step (6) of the IMDDPG algorithm, determine whether the difference between the number of cycles in this cycle and the number of cycles in the last update reaches a preset step size;

[0128] If yes, update the parameters of the Actor evaluation network and the Critic evaluation network to the Actor target network and the Critic target network, and then return to obtain the state information s of the energy management system model t The steps input into the Actor evaluation network until the number of cycles meets the requirement;

[0129] Otherwise, execute and return to obtain the state information s of the energy management system model t The steps input into the Actor evaluation network until the number of cycles meets the requirement.

[0130] For the trained reinforcement learning model, after inputting the initial state information and driving conditions, the model will obtain a series of action information during driving, that is, a series of corresponding required power, so as to realize reasonable mode switching and gear switching according to the required power, and realize multi-objective optimization of the PHEV energy management strategy. A series of action information is the energy management strategy during driving.

[0131] In summary, the energy management method of plug-in hybrid vehicles based on improved multi-objective DDPG in this embodiment establishes an energy management system model of plug-in hybrid vehicles, and realizes the modeling of battery life while modeling the economy of plug-in hybrid vehicles. And based on the continuous actions in actual driving, the IMDDPG algorithm is used to perform multi-objective optimization on the energy management strategy of plug-in hybrid vehicles, getting rid of the dependence of previous energy management strategies on driving conditions, and realizing adaptability to different working conditions through continuous learning of intelligent agents. At the same time, while ensuring the real-time nature of the strategy, the optimality of the strategy is achieved.

[0132] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A plug-in hybrid vehicle energy management method based on improved multi-objective DDPG, characterized in that: The following steps are involved: Establish an energy management system model for plug-in hybrid vehicles, including the vehicle longitudinal dynamics model, engine fuel consumption model, battery equivalent circuit model, battery life model, and drive / generator model; Acquiring state information of the plug-in hybrid vehicle during actual driving, and inputting the state information into the energy management system model; The energy management system model is used as an intelligent agent of IMDDPG, and the reward function of IMDDPG is configured according to the cumulative fuel consumption of the engine, the deviation degree of the battery SOC, and the change of the battery health status. The speed, acceleration, and battery SOC of the plug-in hybrid vehicle are used as the state variables of IMDDPG, and the required power of the plug-in hybrid vehicle is used as the action variable of IMDDPG. IMDDPG is used to perform multi-objective optimization on the energy management system model to obtain a trained reinforcement learning model; Inputting the initial state information and driving conditions into the reinforcement learning model to obtain an energy management strategy during driving; In the above steps, using IMDDPG to perform multi-objective optimization on the energy management system model includes the following steps: Obtaining the state information s of the energy management system model t Input into the Actor evaluation network to get the corresponding action a t And input the energy management system model, and obtain the corresponding reward r(s) under the influence of the environment t ,a t ) and the next state information s t+1 , the current sample data e t =(s t ,a t ,r(s t ,a t ),s t+1 ) is stored in the experience pool, and this step is repeated until the number of sample data in the experience pool meets the requirement; Randomly select samples from the experience pool and write the state information s of the selected samples t Input into the Actor evaluation network to get the corresponding action a t , the status information s t and the corresponding action a t Input into the Critic evaluation network, solve the Pareto optimal frontier to obtain the cumulative reward Q(s) corresponding to the selected sample t ,a t |θ), calculate the cumulative reward Q(s) t ,a t |θ) to obtain the loss function and back-propagate it, and use the gradient ascent method to update the parameters in the Actor evaluation network; The state information s of the sample to be selected t Input into the updated Actor evaluation network to obtain the updated corresponding action a t , the status information s t and the corresponding action a after update t Input into the Critic evaluation network, solve the Pareto optimal frontier to obtain the updated cumulative reward Q(s) corresponding to the selected sample t ,a t |θ); The next state information s of the sample to be selected t+1 Input into the Actor target network to get the corresponding action a t+1 , the status information s t+1 and the corresponding action a t+1 Input the target network of Critic and solve the Pareto optimal frontier to obtain the selected sample in the state information s t+1 and action a t+1 The cumulative reward Q(s) under t+1 ,a t+1 |θ'), calculate the cumulative reward Q(s t+1 ,a t+1 |θ') to obtain the loss function and back propagate it. The gradient descent method is used to update the parameters in the Critic evaluation network. The steps to solve the Pareto optimal frontier include: t and the corresponding action a t Solve for the optimal action value function Q * (s t ,a t ) and randomly select an optimal action value function as the maximum cumulative reward on the Pareto optimal frontier, or select the optimal action value function with the smallest target as the maximum cumulative reward. The optimal action value function Q * (s t ,a t ) is as follows: Among them, R t is the cumulative reward with discount, γ is the discount factor, γ∈[0,1], r i is the reward function at time i, i∈[t,T], T is the termination time; Determine whether the difference between the number of cycles in this cycle and the number of cycles in the last update reaches the preset step size. If so, update the parameters of the Actor evaluation network and the Critic evaluation network to the Actor target network and the Critic target network, and return to obtain the state information s of the energy management system model t The steps input into the Actor evaluation network are executed until the number of cycles meets the requirement, otherwise the execution returns to obtain the state information s of the energy management system model. t Steps that are input into the Actor Evaluation Network; In the above steps, the longitudinal dynamics model of the whole vehicle is expressed as follows: Among them, F D is the driving force, P D is the driving power, T D is the driving torque, v is the vehicle speed, F R 、F A 、F G 、F A′ They are rolling resistance, air resistance, slope resistance and acceleration resistance during vehicle driving, A is the frontal area of ​​the vehicle, C D is the air resistance coefficient, ρ is the air density, c R is the rolling resistance coefficient, m is the total mass of the vehicle, g is the acceleration of gravity, θ is the road slope, δ is the vehicle rotation mass conversion coefficient, is the driving acceleration, r is the wheel radius; Among them, T EN , T EM1 and T EM2 are the torques of the plug-in hybrid vehicle’s engine, EM1 motor, and EM2 motor, ω EN ,ω EM1 and ω EM2 are the speeds of the plug-in hybrid vehicle engine, EM1 motor and EM2 motor, T Out is the output torque of the transmission, K1 and K2 are the gear ratios of the ring gear and sun gear of PG1 and PG2 of the plug-in hybrid vehicle, respectively, and i is the main reducer gear ratio; In the above steps, the engine fuel consumption model expression is as follows: in, is the instantaneous fuel consumption of the engine, m f is the cumulative fuel consumption of the engine, T EN ,ω EN They are the torque and speed of the engine of the plug-in hybrid vehicle; In the above steps, the battery equivalent circuit model expression is as follows: Among them, U Bat is the battery terminal voltage, I Bat is the battery current, U OC is the open circuit current, R Bat is the internal resistance of the battery, P Bat is the battery power, SOC(0) is the initial SOC value, Q Bat is the battery capacity; In the above steps, the battery life model expression is as follows: Among them, Q Loss is the battery capacity attenuation, α and β are constant terms, E A is the activation energy, R is the molar gas constant, T K is the thermodynamic temperature of the environment, Ah is the ampere-hour flux, z is the power exponential factor, Q Bat is the battery capacity, I Bat is the battery current, EOL is the end of battery life, N is the total number of cycles, SOH is the health of the battery, and SOC is the state of charge of the battery; In the above steps, the drive / generator model expression is as follows: or EM =f(T EM ,oh EM ) Among them, T EM is the motor torque, ω EM is the motor speed, η EM is the corresponding motor efficiency; Among them, P EM is the motor mechanical power, P Bat,EM The power delivered from the battery to the motor.

Images