Electric flywheel hybrid electric vehicle energy management method based on multi-agent deep reinforcement learning

Abstract

The invention discloses an electric flywheel hybrid electric vehicle energy management method based on multi-agent deep reinforcement learning, and belongs to the field of new energy commercial vehicle energy management. The invention aims to realize efficient cooperative control among a main driving motor, a control motor and a flywheel by using a multi-agent deep reinforcement learning strategy, reduce the energy consumption of a battery and improve the economical efficiency and the overall performance of a vehicle. The method mainly comprises the following steps: constructing a model of the electric flywheel hybrid electric vehicle; defining six working modes; a multi-agent energy management method is designed, and a state variable, an action variable and a reward function required by a deep reinforcement learning SAC algorithm and a PPO algorithm are set. Under the standard working condition, the multiple agents are trained to obtain the corresponding energy management method, and real-time energy management of the target vehicle is achieved. By applying the multi-agent deep reinforcement learning algorithm to the energy method of the electric flywheel hybrid electric vehicle, the energy consumption of the battery is reduced on the premise of ensuring the performance of the whole vehicle.

Description

Technical Field

[0001] The present invention belongs to the field of energy management of new energy commercial vehicles, and in particular relates to an energy management method for electric flywheel hybrid vehicles based on multi-agent deep reinforcement learning. Background Art

[0002] With the increasing global climate change problem and the continuous growth of energy demand, the use of fossil fuels has brought about serious environmental pollution and greenhouse gas emissions. As the main fossil fuel consumption area, automobile exhaust emissions have become the main source of air pollution. In order to reduce dependence on oil, countries have strengthened regulations on automobile emissions and promoted the promotion of energy-saving and environmentally friendly technologies. The development of electrification and hybrid technologies has gradually become the focus of the automotive industry.

[0003] At present, the single motor systems of many new energy commercial vehicles usually need to be equipped with high-power motors, which not only increases the size and weight of the system, but also has a negative impact on the flexibility and overall performance of the vehicle. As a new type of energy-saving vehicle, electric flywheel hybrid vehicles significantly improve the energy utilization efficiency of vehicles through the efficient energy recovery and release capabilities of the electric flywheel energy storage system. The system is mainly composed of a main drive motor, a control motor and a flywheel, which can flexibly adjust energy distribution under different driving conditions, thereby achieving more efficient energy utilization. However, in the current energy management system of electric flywheel hybrid vehicles, the synergy between multiple systems cannot be fully utilized, which limits the improvement of overall energy utilization efficiency.

[0004] Existing hybrid vehicle energy management methods are mainly divided into three categories: rule-based strategies, optimization-based strategies, and learning-based strategies. Rule-based strategies are simple and easy to implement, suitable for rapid deployment in engineering practice, but they rely on predefined rules and are difficult to flexibly adapt to different driving environments and working conditions. Optimization-based strategies can obtain relatively efficient energy distribution through mathematical optimization models, but the computational complexity of global optimization strategies is high and it is difficult to meet the needs of real-time control; real-time optimization strategies rely on a large amount of prior information and are prone to fall into local optimal solutions, reducing the applicability of the algorithm.

[0005] In contrast, deep reinforcement learning provides a new research direction for the development of vehicle energy management methods, which can autonomously learn and optimize decisions in dynamic and complex environments. However, most existing research focuses on single-agent control methods, which often have difficulty coping with diversified energy inputs and complex operating environments, resulting in insufficient flexibility and adaptability of the system. In contrast, multi-agents can optimize resource allocation and improve overall performance through information sharing and collaborative decision-making. Therefore, in order to achieve efficient development of energy management strategies for electric flywheel hybrid vehicles and reduce battery energy consumption, multi-agent-based energy management strategies are of great significance to new energy commercial vehicle companies. Summary of the invention

[0006] The present invention proposes an energy management method for electric flywheel hybrid electric vehicles based on multi-agent deep reinforcement learning. The method realizes efficient coordinated control between the main drive motor, the control motor and the flywheel by applying a multi-agent deep reinforcement learning strategy, reduces battery energy consumption, and improves the economy and overall performance of the vehicle.

[0007] In order to achieve the above objectives, the present invention adopts the following technical solutions:

[0008] An electric flywheel hybrid vehicle energy management method based on multi-agent deep reinforcement learning comprises the following steps:

[0009] (1) Simulate the actual development situation, establish an electric flywheel hybrid vehicle model, and define six working modes of the vehicle model; the electric flywheel hybrid vehicle model includes a vehicle driving dynamics model, a planetary gear system model, a motor model, a battery model, and a flywheel model. The six working modes are divided into driving mode EV1, driving mode EV2, driving mode EV3, braking mode ERGB1, braking mode ERGB2, and parking mode Stop;

[0010] (2) Taking the electric flywheel hybrid vehicle model as the environment, S t As state variables, a1 and a2 are used as execution actions to build a multi-agent model based on neural network. The multi-agent model includes agent 1 and agent 2. Agent 1 adopts SAC reinforcement learning algorithm, and agent 2 adopts PPO reinforcement learning algorithm to determine the state variables of each agent.

[0011] (3) Design a task-adaptive reward function to guide the agent to optimize its behavior through feedback signals, clarify the action space, and define its executable operations in the environment;

[0012] (4) Perform data preprocessing on the state variables. Subsequently, use the preprocessed data to train multi-agents under CHTC conditions to simulate the energy consumption and performance of the vehicle.

[0013] The action variable a1 of agent 1 is the torque output ratio of the main drive motor, and the action variable a2 of agent 2 is the six working modes. The state variables and action variables are set as follows:

[0014]

[0015] Among them, SOC is the battery state of charge, v is the vehicle speed, U is the battery voltage, Acc is the accelerator pedal opening, Br is the decelerator pedal opening, n dm is the main drive motor speed, T dm is the output torque of the main drive motor, n cm To control the motor speed, T cm To control the motor output torque, n FW is the flywheel speed.

[0016] The specific form of the reward function is as follows:

[0017]

[0018] Among them, r SOC ,r v ,r Fw and r mode Represent the reward functions of SOC, vehicle speed, flywheel speed, and mode switching. β1, β2, β3, β4, β5, and β6 represent the weights of each part of the reward, respectively. t 1 represents the reward function of agent 1, r t 2 Represents the reward function of agent 2.

[0019] Step (4) is as follows: the current neural network interacts with the environment and sets the current state set s t = {SOC,v,U,Acc,Br,n dm ,T dm ,n cm ,T cm ,n FW ,} T Input the current neural network, and Agent 1 and Agent 2 will respond according to the state variable s t Give the corresponding action variables respectively and The power system then performs the corresponding power output and feeds back the state s at the next moment t+1 and reward r t 1 and r t 2 . and The experience data are stored in the experience replay pool and the corresponding algorithms are updated. Through repeated sampling and training, these experience data are used to further optimize the policy network of the multi-agent model.

[0020] The deep reinforcement learning algorithm SAC contains Critic and Actor networks. The algorithm update includes two main steps: updating the Critic network and updating the Actor network.

[0021] In the Critic network, the Q-value function is used to evaluate the t and action variables The total reward that may be obtained in the future, SAC uses a dual Q network to avoid the over-estimation problem. There are usually two Q networks Q1 and Q2.

[0022] The goal of the Q-value function is to minimize the following Bellman error:

[0023]

[0024] Where E is the expected value, is the state s obtained according to the strategy π t and action a t The joint distribution formed is is the output of the Q network, indicating that in state s t Next select action The estimated return after t 1 is the reward function of agent 1, γ is the discount factor, is the minimum value of the two Q networks, as an estimate of the value of the next state, a is the temperature coefficient, is the current strategy in s t+1 The action distribution under

[0025] The goal of the Actor network is to optimize the strategy so that the strategy can not only maximize the reward but also maintain a certain degree of entropy. The strategy optimization in SAC is achieved by maximizing the following objective function:

[0026]

[0027] in, is the entropy term of the policy, is the output of the Q network, indicating that in state s t Next select action The estimated return after

[0028] The deep reinforcement learning PPO algorithm used includes Policy and Value networks. The Policy network strategy update is achieved by maximizing the following objective function:

[0029]

[0030] Among them, E t is the expected value at time t, For the current strategy in state s t Take Action The probability of is the probability that the old strategy performs the same action in the same state, is the advantage function, is the probability ratio of the current strategy to the old strategy, is the clipping function and ε is the clipping parameter.

[0031] The specific description of the driving dynamics model of the vehicle at the starting stage is as follows:

[0032]

[0033] Where m is the vehicle mass; g is the acceleration of gravity; f is the rolling resistance coefficient; θ is the road slope; C D is the air resistance coefficient; A is the frontal area of ​​the vehicle; ρ is the air density; v is the vehicle speed; δ is the rotational mass conversion coefficient; R c is the wheel radius; T b is the required mechanical braking torque; T dem The vehicle demand torque;

[0034] The planetary gear system consists of planetary gear set 1 and planetary gear set 2. In the driving state, the power transmission is calculated as follows:

[0035]

[0036] Among them, T dm 、n dm and T cm 、n cm The torque and speed of the main drive motor and control motor; n c is the vehicle wheel speed; η c is the mechanical transmission efficiency of the power system; r PG1 、r PG2 and r c They are planetary gear set 1, planetary gear set 2 and final reduction ratio,

[0037] In braking state, power is transmitted as follows:

[0038]

[0039] Among them, T Fw 、n Fw are the torque and speed of the flywheel, r Fwis the transmission ratio of the flywheel reducer,

[0040] The motor model is shown in the following formula:

[0041]

[0042] Among them, η m is the motor efficiency, P out is the output power of the motor, P in is the input power, U s is the input voltage of the motor, I s is the input current of the motor, T is the motor torque, n is the motor speed,

[0043] The battery model is shown in the following formula:

[0044]

[0045] Among them, SOC0 is the state of charge at the initial state (t0), SOC is the state of charge at t, Q nom is the nominal capacity of the battery, I b is the battery current, U oc is the open circuit voltage of the battery, R0 is the internal resistance of the battery, P b is the instantaneous output power of the battery, P bat is the battery power,

[0046] The flywheel model is shown in the following formula:

[0047]

[0048] Among them, E Fw is the flywheel energy, ω Fw is the angular velocity of the flywheel, I is the moment of inertia of the flywheel, n Fw is the flywheel speed, P Fw is the instantaneous flywheel power.

[0049] The energy management strategy of the electric flywheel hybrid vehicle of the present invention aims to maintain better vehicle speed following performance and better distribute the required torque between different power sources to obtain better economy and durability during driving. The present invention provides an energy management method for an electric flywheel hybrid vehicle based on multi-agent deep reinforcement learning. The multi-agent deep reinforcement learning strategy is used to solve the energy management problem of the whole vehicle, namely, the efficient coordinated control problem between the main drive motor, the control motor and the flywheel, so as to avoid continuous operation of the battery in a high fuel consumption range, reduce battery energy consumption, and improve the economy and overall performance of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1is a structural diagram of an electric flywheel hybrid vehicle according to an embodiment of the present invention;

[0051] Figure 2 is a schematic diagram of the working mode of an electric flywheel hybrid vehicle in an embodiment of the present invention;

[0052] Figure 3 is a schematic flow chart of a multi-agent-based energy management strategy design method in an embodiment of the present invention;

[0053] Figure 4 It is a SOC comparison chart after the multi-agent control strategy is optimized in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments:

[0055] Example

[0056] According to the structural diagram, the proposed electric flywheel hybrid vehicle powertrain is as follows: Figure 1 The system is mainly composed of a main drive motor, a control motor, a flywheel, a flywheel brake, a flywheel reducer, a planetary gear set 1, a planetary gear set 2, a sensor, etc.

[0057] like Figure 3 As shown, the electric flywheel hybrid vehicle energy management method based on multi-agent deep reinforcement learning includes the following steps:

[0058] (1) Simulate the actual development situation, establish an electric flywheel hybrid vehicle model, and define six working modes of the vehicle model; the electric flywheel hybrid vehicle model includes a vehicle driving dynamics model, a planetary gear system model, a motor model, a battery model, and a flywheel model.

[0059] The specific description of the driving dynamics model of the vehicle at the starting stage is as follows:

[0060]

[0061] Where m is the vehicle mass; g is the acceleration of gravity; f is the rolling resistance coefficient; θ is the road slope; C D is the air resistance coefficient; A is the frontal area of ​​the vehicle; ρ is the air density; v is the vehicle speed; δ is the rotational mass conversion coefficient; R c is the wheel radius; T b is the required mechanical braking torque; T dem The vehicle demand torque;

[0062] The planetary gear system consists of planetary gear set 1 and planetary gear set 2. In the driving state, the power transmission is calculated as follows:

[0063]

[0064] Among them, T dm 、n dm and T cm 、n cm The torque and speed of the main drive motor and control motor; n c is the vehicle wheel speed; η c is the mechanical transmission efficiency of the power system; r PG1 、r PG2 and r c They are planetary gear set 1, planetary gear set 2 and final reduction ratio respectively.

[0065] In braking state, power is transmitted as follows:

[0066]

[0067] Among them, T Fw 、n Fw are the torque and speed of the flywheel, r Fw is the transmission ratio of the flywheel reducer.

[0068] The motor model is shown in the following formula:

[0069]

[0070] Among them, η m is the motor efficiency, P out is the output power of the motor, P in is the input power, U s is the input voltage of the motor, I s is the input current of the motor, T is the motor torque, and n is the motor speed.

[0071] The battery model is shown in the following formula:

[0072]

[0073] Among them, SOC0 is the state of charge at the initial state (t0), SOC is the state of charge at t, Q nom is the nominal capacity of the battery, I b is the battery current, U oc is the open circuit voltage of the battery, R0 is the internal resistance of the battery, P b is the instantaneous output power of the battery, P bat is the battery power.

[0074] The flywheel model is shown in the following formula:

[0075]

[0076] Among them, E Fw is the flywheel energy, ω Fw is the angular velocity of the flywheel, I is the moment of inertia of the flywheel, n Fw is the flywheel speed, P Fw is the instantaneous flywheel power.

[0077] like Figure 2 As shown, according to the structural diagram, it can be seen that the electric flywheel hybrid vehicle has six working modes, the driving state is divided into EV1, EV2, and EV3, the braking state is divided into ERGB1 and ERGB2, and Stop is the parking mode.

[0078] EV1: When the vehicle starts, the main drive motor and the control motor both work as motors, outputting positive torque and positive speed. The flywheel is locked and does not participate in the drive.

[0079] ERGB1: In this mode, the main drive motor idles, and the control motor works as a generator, outputting negative torque and positive speed. The flywheel speed increases, and together with the control motor, it provides partial braking force.

[0080] ERGB2: During braking, when the vehicle speed drops to a critical value, the control motor changes from positive speed to reverse speed, but still outputs negative torque, consuming electrical energy. The flywheel continues to accelerate and rotate, cooperating with the control motor to provide braking force, converting the kinetic energy and electrical energy recovered by braking into mechanical energy of the flywheel.

[0081] EV2: The main drive motor outputs positive torque and positive speed to work as a motor. The motor is controlled to output positive torque and negative speed in generator mode, the flywheel speed decreases, and the stored energy is released.

[0082] EV3: During further acceleration, the motor speed is controlled to change from reverse to forward and enter the motor mode, while the main drive motor remains in the motor state. The flywheel continues to release mechanical energy to accelerate the vehicle.

[0083] Stop: When the vehicle is stationary, the planetary carrier speed is zero. At this time, the motor is controlled to output positive torque, enter reverse deceleration and work in generator mode. At the same time, the flywheel begins to decelerate and release its stored mechanical energy, which is gradually converted into battery power.

[0084] (2) Taking the electric flywheel hybrid vehicle model as the environment, S t The state variables are a1 and a2, and a multi-agent model based on neural network is constructed with a1 and a2 as execution actions. The multi-agent model includes agent 1 and agent 2. Agent 1 adopts SAC reinforcement learning algorithm, and agent 2 adopts PPO reinforcement learning algorithm to determine the state variables of each agent to ensure that it can effectively perceive the environment.

[0085] The state variable S based on the multi-agent model t The battery state of charge SOC, vehicle speed v, battery voltage U, accelerator pedal opening Acc, decelerator pedal opening Br, main drive motor speed n dm , Main drive motor output torque T dm , control the motor speed n cm , control the motor output torque T cm , flywheel speed n FW ; The action variable a1 of agent 1 is the torque output ratio of the main drive motor, and the action variable a2 of agent 2 is the six working modes. The state variables and action variables are set as follows:

[0086]

[0087] (3) Design a task-adaptive reward function to guide the agent to optimize its behavior through feedback signals, clarify the action space, and define its executable operations in the environment.

[0088] The reasonable design of the reward function can effectively guide the behavior of the intelligent agent, and its primary goal is to improve energy efficiency. In the reward function setting, the reward and punishment mechanism of SOC is particularly critical. In addition, the tracking performance of the vehicle speed v cannot be ignored. Combined with the structural characteristics of this model, maximizing the recovery and utilization of flywheel kinetic energy is another important goal. Therefore, the design of the reward function must also include rewards for flywheel speed and mode switching. The specific form is as follows:

[0089]

[0090] Among them, r SOC ,r v ,r Fw and r mode Represent the reward functions of SOC, vehicle speed, flywheel speed, and mode switching. β1, β2, β3, β4, β5, and β6 represent the weights of each part of the reward, respectively. t 1 represents the reward function of agent 1, r t 2 Represents the reward function of agent 2.

[0091] (4) Perform data preprocessing on state variables to ensure data quality and consistency. Then, use the preprocessed data to train multiple agents under CHTC conditions to simulate the energy consumption and performance of the vehicle, thereby optimizing the decision-making ability and behavioral strategy of the agents.

[0092] Training a multi-agent model for an electric flywheel hybrid vehicle consists of the following steps:

[0093] Step A: Initialize the parameters of each agent in the multi-agent model according to the set state variables, action variables and reward functions, and normalize the state variables;

[0094] Step B: The initialized multi-agent model interacts with the electric flywheel hybrid vehicle and is trained under CHTC conditions to obtain a trained multi-agent model.

[0095] Specifically, the current neural network interacts with the environment and sets the current state s t = {SOC,v,U,Acc,Br,n dm ,T dm ,n cm ,T cm ,n FW ,} T Input the current neural network, and Agent 1 and Agent 2 will respond according to the state variable s t Give the corresponding action variables respectively and The power system then performs the corresponding power output and feeds back the state s at the next moment t+1 and reward r t 1 and r t 2 . and The experience data are stored in the experience replay pool and the corresponding algorithms are updated. Through repeated sampling and training, these experience data are used to further optimize the policy network of the multi-agent model, improve the accuracy of decision-making and the overall energy efficiency of the system.

[0096] The deep reinforcement learning algorithm SAC used in step B contains Critic and Actor networks, and the algorithm update includes two main steps: updating the Critic network and updating the Actor network.

[0097] In the Critic network, the Q-value function is used to evaluate the t and action variables The total reward that may be obtained in the future. SAC uses a dual Q network to avoid the over-estimation problem, usually with two Q networks Q1 and Q2.

[0098] The goal of the Q-value function is to minimize the following Bellman error:

[0099]

[0100] Where E is the expected value, is the state s obtained according to the strategy π t and action a tThe joint distribution formed is and are the outputs of the Q1 and Q2 networks, respectively, indicating that in state s t Next select action The estimated return after t 1 is the reward function of agent 1, γ is the discount factor, is the minimum value of the two Q networks, as the value estimate of the next state, i takes the value of 1 or 2, a is the temperature coefficient, is the current strategy in s t+1 The following action distribution.

[0101] The goal of the Actor network is to optimize the policy so that it not only maximizes the reward but also maintains a certain degree of entropy. Policy optimization in SAC is achieved by maximizing the following objective function:

[0102]

[0103] in, is the entropy term of the policy, is the output of the Q network, indicating that in state s t Next select action Estimated return after .

[0104] The deep reinforcement learning PPO algorithm used includes Policy and Value networks. The Policy network strategy update is achieved by maximizing the following objective function:

[0105]

[0106] Among them, E t is the expected value at time t, For the current strategy in state s t Take Action The probability of is the probability that the old strategy performs the same action in the same state, is the advantage function, is the probability ratio of the current strategy to the old strategy, is the clipping function and ε is the clipping parameter.

[0107] The Value network is responsible for providing each state s t Estimate its state value. This estimate of the state value is to calculate the advantage function The core of the function is to measure the selection of an action a in a certain state. t The superiority of the behavior over the average. The estimate of the advantage function is usually calculated via the TD-error.

[0108] Initialize the current policy network parameters and value network parameters, and set the corresponding hyperparameters such as shear coefficient and learning rate to ensure that the strategy of each agent can effectively explore and utilize the environment, and finally obtain the initialized multi-agent model.

[0109] Figure 4 This is a comparison curve of the battery SOC of the electric flywheel hybrid vehicle under the multi-agent strategy of the present invention and the traditional single-motor new energy commercial vehicle model under the same vehicle parameters. According to the simulation results, the energy management strategy improves the vehicle economy, improves the safety and stability of fuel vehicle operation, reduces battery consumption, and greatly improves energy management efficiency.

Claims

1. An energy management method for electric flywheel hybrid electric vehicles based on multi-agent deep reinforcement learning, characterized in that: The following steps are involved: (1) Simulate the actual development situation, establish an electric flywheel hybrid vehicle model, and define six working modes of the vehicle model; the electric flywheel hybrid vehicle model includes a vehicle driving dynamics model, a planetary gear system model, a motor model, a battery model, and a flywheel model. The six working modes are divided into driving mode EV1, driving mode EV2, driving mode EV3, braking mode ERGB1, braking mode ERGB2, and parking mode Stop; (2) Taking the electric flywheel hybrid vehicle model as the environment, S t As state variables, a1 and a2 are used as execution actions to build a multi-agent model based on neural network. The multi-agent model includes agent 1 and agent 2. Agent 1 adopts SAC reinforcement learning algorithm, and agent 2 adopts PPO reinforcement learning algorithm to determine the state variables of each agent. (3) Design a task-adaptive reward function to guide the agent to optimize its behavior through feedback signals, clarify the action space, and define its executable operations in the environment; (4) Perform data preprocessing on the state variables. Then, use the preprocessed data to train multi-agents under CHTC conditions to simulate the energy consumption and performance of the vehicle.

2. The electric flywheel hybrid vehicle energy management method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The action variable a1 of agent 1 is the torque output ratio of the main drive motor, and the action variable a2 of agent 2 is the six working modes. The state variables and action variables are set as follows: s t ={SOC,v,U,Acc,Br,n dm ,T dm ,n cm ,T cm ,n Fw } T Among them, SOC is the battery state of charge, v is the vehicle speed, U is the battery voltage, Acc is the accelerator pedal opening, Br is the decelerator pedal opening, n dm is the main drive motor speed, T dm is the output torque of the main drive motor, n cm To control the motor speed, T cm To control the motor output torque, n FW is the flywheel speed.

3. The electric flywheel hybrid vehicle energy management method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The specific form of the reward function is as follows: Among them, r SOC ,r v ,r Fw and r mode Represent the reward functions of SOC, vehicle speed, flywheel speed, and mode switching. β1, β2, β3, β4, β5, and β6 represent the weights of each part of the reward, respectively. t 1 represents the reward function of agent 1, r t 2 Represents the reward function of agent 2.

4. The electric flywheel hybrid vehicle energy management method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: Step (4) is as follows: the current neural network interacts with the environment and sets the current state set s t = {SOC,v,U,Acc,Br,n dm ,T dm ,n cm ,T cm ,n FW ,} T Input the current neural network, and Agent 1 and Agent 2 will respond according to the state variable s t Give the corresponding action variables respectively and The power system then performs the corresponding power output and feeds back the state s at the next moment t+1 and reward r t 1 and r t 2 . and The experience data are stored in the experience replay pool and the corresponding algorithms are updated. Through repeated sampling and training, these experience data are used to further optimize the policy network of the multi-agent model.

5. The electric flywheel hybrid vehicle energy management method based on multi-agent deep reinforcement learning according to claim 4 is characterized in that: The deep reinforcement learning algorithm SAC contains Critic and Actor networks. The algorithm update includes two main steps: updating the Critic network and updating the Actor network. In the Critic network, the Q-value function is used to evaluate the t and action variables The total reward that may be obtained in the future, SAC uses a dual Q network to avoid the over-estimation problem. There are usually two Q networks Q1 and Q2. The goal of the Q-value function is to minimize the following Bellman error: Where E is the expected value, is the state s obtained according to the strategy π t and action a t The joint distribution formed is is the output of the Q network, indicating that in state s t Next select action The estimated return after t 1 is the reward function of agent 1, γ is the discount factor, is the minimum value of the two Q networks, as an estimate of the value of the next state, a is the temperature coefficient, is the current strategy in s t+1 The action distribution under The goal of the Actor network is to optimize the strategy so that the strategy can not only maximize the reward but also maintain a certain degree of entropy. The strategy optimization in SAC is achieved by maximizing the following objective function: in, is the entropy term of the policy, is the output of the Q network, indicating that in state s t Next select action The estimated return after The deep reinforcement learning PPO algorithm used includes Policy and Value networks. The Policy network strategy update is achieved by maximizing the following objective function: Among them, E t is the expected value at time t, For the current strategy in state s t Take Action The probability of is the probability that the old strategy performs the same action in the same state, is the advantage function, is the probability ratio of the current strategy to the old strategy, is the clipping function and ε is the clipping parameter.

6. The electric flywheel hybrid vehicle energy management method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The specific description of the driving dynamics model of the vehicle at the starting stage is as follows: Where m is the vehicle mass; g is the acceleration of gravity; f is the rolling resistance coefficient; θ is the road slope; C D is the air resistance coefficient; A is the frontal area of ​​the vehicle; ρ is the air density; v is the vehicle speed; δ is the rotational mass conversion coefficient; R c is the wheel radius; T b is the required mechanical braking torque; T dem The vehicle demand torque; The planetary gear system consists of planetary gear set 1 and planetary gear set 2. In the driving state, the power transmission is calculated as follows: Among them, T dm 、n dm and T cm 、n cm The torque and speed of the main drive motor and control motor; n c is the vehicle wheel speed; η c is the mechanical transmission efficiency of the power system; r PG1 、r PG2 and r c They are planetary gear set 1, planetary gear set 2 and final reduction ratio, In braking state, power is transmitted as follows: Among them, T Fw 、n Fw are the torque and speed of the flywheel, r Fw is the transmission ratio of the flywheel reducer, The motor model is shown in the following formula: Among them, η m is the motor efficiency, P out is the output power of the motor, P in is the input power, U s is the input voltage of the motor, I s is the input current of the motor, T is the motor torque, n is the motor speed, The battery model is shown in the following formula: Among them, SOC0 is the state of charge at the initial state (t0), SOC is the state of charge at t, Q nom is the nominal capacity of the battery, I b is the battery current, U oc is the open circuit voltage of the battery, R0 is the internal resistance of the battery, P b is the instantaneous output power of the battery, P bat is the battery power, The flywheel model is shown in the following formula: Among them, E Fw is the flywheel energy, ω Fw is the angular velocity of the flywheel, I is the moment of inertia of the flywheel, n Fw is the flywheel speed, P Fw is the instantaneous flywheel power.

Images