An energy management method for electric flywheel hybrid vehicles based on multi-agent deep reinforcement learning

Abstract

The present invention discloses an energy management method for an electric flywheel hybrid vehicle based on multi-agent deep reinforcement learning, which belongs to the field of energy management for new energy commercial vehicles. The purpose of the present invention is to use a multi-agent deep reinforcement learning strategy to achieve efficient coordinated control between the main drive motor, the control motor and the flywheel, reduce battery energy consumption, and improve the economy and overall performance of the vehicle. The present invention mainly includes: constructing a model of an electric flywheel hybrid vehicle; defining six working modes; designing a multi-agent energy management method, and setting the state variables, action variables and reward functions required by the deep reinforcement learning SAC algorithm and PPO algorithm. Under standard working conditions, the multi-agent is trained to obtain the corresponding energy management method to achieve real-time energy management of the target vehicle. By applying the multi-agent deep reinforcement learning algorithm to the energy method of an electric flywheel hybrid vehicle, the battery energy consumption is reduced while ensuring the performance of the entire vehicle.

Description

Technical Field

[0001] The present invention belongs to the field of energy management of new energy commercial vehicles, and in particular relates to an energy management method for electric flywheel hybrid vehicles based on multi-agent deep reinforcement learning. Background Art

[0002] With the growing global threat of climate change and the continued growth of energy demand, the use of fossil fuels has led to serious environmental pollution and greenhouse gas emissions. Automobiles, as a major consumer of fossil fuels, have become a major source of air pollution through exhaust emissions. To reduce dependence on oil, countries are tightening regulations on vehicle emissions and promoting the adoption of energy-saving and environmentally friendly technologies. The development of electrification and hybrid technologies has gradually become a key focus of the automotive industry.

[0003] Currently, the single-motor systems of many new energy commercial vehicles usually need to be equipped with high-power motors, which not only increases the size and weight of the system, but also has a negative impact on the vehicle's flexibility and overall performance. As a new type of energy-saving vehicle, electric flywheel hybrid vehicles significantly improve the vehicle's energy utilization efficiency through the efficient energy recovery and release capabilities of the electric flywheel energy storage system. The system mainly consists of a main drive motor, a control motor and a flywheel, and can flexibly adjust energy distribution under different driving conditions, thereby achieving more efficient energy utilization. However, in the current energy management system of electric flywheel hybrid vehicles, the synergy between multiple systems cannot be fully utilized, which limits the improvement of overall energy utilization efficiency.

[0004] Existing hybrid vehicle energy management methods are primarily categorized into three main categories: rule-based, optimization-based, and learning-based strategies. Rule-based strategies are simple and easy to implement, suitable for rapid deployment in engineering practice. However, they rely on predefined rules and lack flexibility in adapting to diverse driving environments and operating conditions. Optimization-based strategies can achieve relatively efficient energy allocation through mathematical optimization models, but global optimization strategies are computationally complex and difficult to meet real-time control requirements. Real-time optimization strategies rely on extensive prior information and are prone to falling into local optimal solutions, reducing the algorithm's applicability.

[0005] In contrast, deep reinforcement learning offers a new research direction for the development of vehicle energy management methods, enabling autonomous learning and optimized decision-making in dynamic and complex environments. However, existing research has largely focused on single-agent control methods, which often struggle to cope with diverse energy inputs and complex operating environments, resulting in insufficient system flexibility and adaptability. In contrast, multi-agent systems can optimize resource allocation and improve overall performance through information sharing and collaborative decision-making. Therefore, to achieve efficient development of energy management strategies for electric flywheel hybrid vehicles and reduce battery energy consumption, multi-agent-based energy management strategies are of great significance to new energy commercial vehicle companies. Summary of the Invention

[0006] This paper proposes an energy management method for electric flywheel hybrid vehicles based on multi-agent deep reinforcement learning. By applying a multi-agent deep reinforcement learning strategy, this method achieves efficient coordinated control between the main drive motor, control motor, and flywheel, reducing battery energy consumption and improving vehicle economy and overall performance.

[0007] In order to achieve the above objectives, the present invention adopts the following technical solutions:

[0008] An energy management method for an electric flywheel hybrid vehicle based on multi-agent deep reinforcement learning comprises the following steps:

[0009] (1) Simulate the actual development situation, establish an electric flywheel hybrid vehicle model, and define six working modes of the vehicle model; the electric flywheel hybrid vehicle model includes a vehicle driving dynamics model, a planetary gear system model, a motor model, a battery model, and a flywheel model. The six working modes are divided into driving mode EV1, driving mode EV2, driving mode EV3, braking mode ERGB1, braking mode ERGB2, and parking mode Stop;

[0010] (2) Taking the electric flywheel hybrid vehicle model as the environment, S t As the state variable, a1 and a2 are used as the execution actions to build a multi-agent model based on a neural network. The multi-agent model includes agent 1 and agent 2. Agent 1 uses the SAC reinforcement learning algorithm, and agent 2 uses the PPO reinforcement learning algorithm to determine the state variables of each agent.

[0011] (3) Design a task-adapted reward function to guide the agent to optimize its behavior through feedback signals, and clarify the action space to define its executable operations in the environment;

[0012] (4) Perform data preprocessing on the state variables. Subsequently, the preprocessed data is used to train the multi-agent under CHTC conditions to simulate the energy consumption and performance of the vehicle.

[0013] The action variable a1 of agent 1 is the torque output ratio of the main drive motor, and the action variable a2 of agent 2 is the six working modes. The state variables and action variables are set as follows:

[0014]

[0015] Among them, SOC is the battery state of charge, v is the vehicle speed, U is the battery voltage, Acc is the accelerator pedal opening, Br is the decelerator pedal opening, n dm is the main drive motor speed, T dm The output torque of the main drive motor, n cm To control the motor speed, T cm To control the motor output torque, n FW is the flywheel speed.

[0016] The specific form of the reward function is as follows:

[0017]

[0018] Among them, r SOC ,r v ,r Fw and r mode Represent the reward functions of SOC, vehicle speed, flywheel speed, and mode switching respectively. β1, β2, β3, β4, β5, and β6 represent the weights of each part of the reward, r t 1 represents the reward function of agent 1, r t 2 Represents the reward function of agent 2.

[0019] Step (4) is as follows: the current neural network interacts with the environment and sets the current state set s t ={SOC,v,U,Acc,Br,n dm ,T dm ,n cm ,T cm ,n FW ,} T Input the current neural network, and Agent 1 and Agent 2 will be based on the state variable s t Give the corresponding action variables respectively and The power system then performs the corresponding power output and feeds back the state s at the next moment t+1 and reward r t 1 and r t 2 . and The experience data are stored in the experience replay pool and the corresponding algorithms are updated respectively. Through repeated sampling and training, these experience data are used to further optimize the policy network of the multi-agent model.

[0020] The deep reinforcement learning algorithm SAC contains Critic and Actor networks. The algorithm update includes two main steps: updating the Critic network and updating the Actor network.

[0021] In the Critic network, the Q-value function is used to evaluate the t and action variables Under the total reward that may be obtained in the future, SAC uses a dual Q network to avoid the over-estimation problem. There are usually two Q networks Q1 and Q2.

[0022] The goal of the Q-value function is to minimize the following Bellman error:

[0023]

[0024] Where E is the expected value, is the state s obtained according to the strategy π t and action a t The joint distribution formed, is the output of the Q network, indicating that in state s t Select Action The estimated return after t 1 is the reward function of agent 1, γ is the discount factor, is the minimum value of the two Q networks, which is used as the value estimate of the next state, a is the temperature coefficient, is the current strategy in s t+1 The action distribution under

[0025] The goal of the Actor network is to optimize the policy so that it not only maximizes the reward but also maintains a certain degree of entropy. Policy optimization in SAC is achieved by maximizing the following objective function:

[0026]

[0027] in, is the entropy term of the policy, is the output of the Q network, indicating that in state s t Select Action The estimated return after

[0028] The deep reinforcement learning PPO algorithm used includes Policy and Value networks. The Policy network strategy update is achieved by maximizing the following objective function:

[0029]

[0030] Among them, E t is the expected value at time t, For the current strategy in state s t Take action The probability of is the probability that the old strategy performs the same action in the same state, is the advantage function, is the probability ratio of the current strategy to the old strategy, is the clipping function and ε is the clipping parameter.

[0031] The specific description of the vehicle's driving dynamics model at the starting stage is as follows:

[0032]

[0033] Where m is the vehicle mass; g is the acceleration of gravity; f is the rolling resistance coefficient; θ is the road slope; C D is the air resistance coefficient; A is the frontal area of ​​the vehicle; ρ is the air density; v is the vehicle speed; δ is the rotational mass conversion coefficient; R c is the wheel radius; T b is the required mechanical braking torque; T dem is the vehicle demand torque;

[0034] The planetary gear system consists of planetary gear set 1 and planetary gear set 2. In the driving state, the power transmission is calculated as follows:

[0035]

[0036] Among them, T dm 、n dm and T cm 、n cm The torque and speed of the main drive motor and the control motor; n c is the vehicle wheel speed; η c is the mechanical transmission efficiency of the power system; r PG1 、r PG2 and r c They are planetary gear set 1, planetary gear set 2 and final reduction ratio,

[0037] In the braking state, the power is transmitted as follows:

[0038]

[0039] Among them, T Fw 、n Fw are the torque and speed of the flywheel, r Fwis the flywheel reducer transmission ratio,

[0040] The motor model is shown in the following formula:

[0041]

[0042] Among them, η m is the motor efficiency, P out is the output power of the motor, P in is the input power, U s is the input voltage of the motor, I s is the input current of the motor, T is the motor torque, n is the motor speed,

[0043] The battery model is shown in the following formula:

[0044]

[0045] Among them, SOC0 is the state of charge at the initial state (t0), SOC is the state of charge at t, Q nom is the nominal capacity of the battery, I b is the battery current, U oc is the open circuit voltage of the battery, R0 is the internal resistance of the battery, P b is the instantaneous output power of the battery, P bat is the battery power,

[0046] The flywheel model is shown in the following formula:

[0047]

[0048] Among them, E Fw is the flywheel energy, ω Fw is the angular velocity of the flywheel, I is the moment of inertia of the flywheel, n Fw is the flywheel speed, P Fw is the instantaneous flywheel power.

[0049] The energy management strategy of the electric flywheel hybrid vehicle of the present invention aims to maintain better vehicle speed tracking during driving and better distribute the required torque between different power sources to obtain better economy and durability. The present invention provides an energy management method for electric flywheel hybrid vehicles based on multi-agent deep reinforcement learning. By using a multi-agent deep reinforcement learning strategy, the energy management problem of the entire vehicle is realized, namely, the efficient coordinated control problem between the main drive motor, the control motor and the flywheel, thereby avoiding continuous operation of the battery in the high fuel consumption range, reducing battery energy consumption, and improving the economy and overall performance of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1is a structural diagram of an electric flywheel hybrid vehicle according to an embodiment of the present invention;

[0051] Figure 2 is a schematic diagram of the operating mode of an electric flywheel hybrid vehicle according to an embodiment of the present invention;

[0052] Figure 3 This is a flow chart of a multi-agent-based energy management strategy design method according to an embodiment of the present invention;

[0053] Figure 4 This is a comparison chart of SOC after optimization of the multi-agent control strategy in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments:

[0055] Example

[0056] According to the structural diagram, the proposed electric flywheel hybrid vehicle powertrain is as follows: Figure 1 The system mainly consists of a main drive motor, a control motor, a flywheel, a flywheel brake, a flywheel reducer, a planetary gear set 1, a planetary gear set 2, and sensors.

[0057] like Figure 3 As shown, the energy management method of electric flywheel hybrid electric vehicle based on multi-agent deep reinforcement learning includes the following steps:

[0058] (1) Simulate the actual development situation, establish an electric flywheel hybrid vehicle model, and define six working modes of the vehicle model; among them, the electric flywheel hybrid vehicle model includes a vehicle driving dynamics model, a planetary gear system model, a motor model, a battery model, and a flywheel model.

[0059] The specific description of the vehicle's driving dynamics model at the starting stage is as follows:

[0060]

[0061] Where m is the vehicle mass; g is the acceleration of gravity; f is the rolling resistance coefficient; θ is the road slope; C D is the air resistance coefficient; A is the frontal area of ​​the vehicle; ρ is the air density; v is the vehicle speed; δ is the rotational mass conversion coefficient; R c is the wheel radius; T b is the required mechanical braking torque; T dem is the vehicle demand torque;

[0062] The planetary gear system consists of planetary gear set 1 and planetary gear set 2. In the driving state, the power transmission is calculated as follows:

[0063]

[0064] Among them, T dm 、n dm and T cm 、n cm The torque and speed of the main drive motor and the control motor; n c is the vehicle wheel speed; η c is the mechanical transmission efficiency of the power system; r PG1 、r PG2 and r c They are planetary gear set 1, planetary gear set 2 and final reduction ratio respectively.

[0065] In the braking state, the power is transmitted as follows:

[0066]

[0067] Among them, T Fw 、n Fw are the torque and speed of the flywheel, r Fw is the flywheel reducer transmission ratio.

[0068] The motor model is shown in the following formula:

[0069]

[0070] Among them, η m is the motor efficiency, P out is the output power of the motor, P in is the input power, U s is the input voltage of the motor, I s is the input current of the motor, T is the motor torque, and n is the motor speed.

[0071] The battery model is shown in the following formula:

[0072]

[0073] Among them, SOC0 is the state of charge at the initial state (t0), SOC is the state of charge at t, Q nom is the nominal capacity of the battery, I b is the battery current, U oc is the open circuit voltage of the battery, R0 is the internal resistance of the battery, P b is the instantaneous output power of the battery, P bat is the battery power.

[0074] The flywheel model is shown in the following formula:

[0075]

[0076] Among them, E Fw is the flywheel energy, ω Fw is the angular velocity of the flywheel, I is the moment of inertia of the flywheel, n Fw is the flywheel speed, P Fw is the instantaneous flywheel power.

[0077] like Figure 2 As shown, according to the structural diagram, it can be seen that the electric flywheel hybrid vehicle has six working modes. The driving state is divided into EV1, EV2, and EV3, the braking state is divided into ERGB1 and ERGB2, and Stop is the parking mode.

[0078] EV1: When the vehicle starts, both the main drive motor and the control motor operate as motors, outputting positive torque and positive speed. The flywheel is locked and does not participate in driving.

[0079] ERGB1: In this mode, the main drive motor idles and the control motor operates as a generator, outputting negative torque and positive speed. The flywheel speed increases, and together with the control motor, it provides partial braking force.

[0080] ERGB2: During braking, when the vehicle speed drops to a critical value, the control motor switches from positive rotation to reverse rotation, while still outputting negative torque and consuming electrical energy. The flywheel continues to accelerate, working in conjunction with the control motor to provide braking force, converting the kinetic and electrical energy recovered by braking into mechanical energy for the flywheel.

[0081] EV2: The main drive motor outputs positive torque and positive speed to operate as a motor. The motor is controlled to output positive torque and negative speed in generator mode, causing the flywheel speed to decrease and release stored energy.

[0082] EV3: During further acceleration, the control motor switches from reverse to forward rotation and enters motor mode, while the main drive motor remains in motor mode. The flywheel continues to release mechanical energy, accelerating the vehicle.

[0083] Stop: When the vehicle is stationary, the planetary carrier speed is zero. At this point, the motor is controlled to output positive torque, entering reverse deceleration and operating in generator mode. Simultaneously, the flywheel begins to decelerate, releasing its stored mechanical energy and gradually converting it into battery power.

[0084] (2) Taking the electric flywheel hybrid vehicle model as the environment, S t The state variables are set as a1 and a2 as execution actions, and a multi-agent model based on neural network is constructed. The multi-agent model includes agent 1 and agent 2. Agent 1 adopts SAC reinforcement learning algorithm and agent 2 adopts PPO reinforcement learning algorithm to determine the state variables of each agent to ensure that it can effectively perceive the environment.

[0085] The state variable S based on the multi-agent model t The battery state of charge SOC, vehicle speed v, battery voltage U, accelerator pedal opening Acc, decelerator pedal opening Br, main drive motor speed n dm , Main drive motor output torque T dm , control the motor speed n cm , control the motor output torque T cm , flywheel speed n FW The action variable a1 of agent 1 is the torque output ratio of the main drive motor, and the action variable a2 of agent 2 is the six working modes. The state variables and action variables are set as follows:

[0086]

[0087] (3) Design a task-adapted reward function to guide the agent to optimize its behavior through feedback signals, clarify the action space, and define its executable operations in the environment.

[0088] A well-designed reward function can effectively guide the agent's behavior, with the primary goal being to improve energy efficiency. Within this reward function, the SOC reward and penalty mechanism is particularly crucial. Furthermore, the tracking performance of vehicle speed v is also crucial. Given the structural characteristics of this vehicle model, maximizing flywheel kinetic energy recovery and utilization is another key objective. Therefore, the reward function design must also include rewards for flywheel speed and mode switching, specifically in the following form:

[0089]

[0090] Among them, r SOC ,r v ,r Fw and r mode Represent the reward functions of SOC, vehicle speed, flywheel speed, and mode switching respectively. β1, β2, β3, β4, β5, and β6 represent the weights of each part of the reward, r t 1 represents the reward function of agent 1, r t 2 Represents the reward function of agent 2.

[0091] (4) Data preprocessing is performed on state variables to ensure data quality and consistency. Subsequently, the preprocessed data is used to train multiple agents under CHTC conditions to simulate the energy consumption and performance of the vehicle, thereby optimizing the decision-making ability and behavioral strategy of the agents.

[0092] Training a multi-agent model for an electric flywheel hybrid vehicle involves the following steps:

[0093] Step A: Initialize the parameters of each agent in the multi-agent model according to the set state variables, action variables and reward function, and normalize the state variables;

[0094] Step B: The initialized multi-agent model interacts with the electric flywheel hybrid vehicle and is trained under CHTC conditions to obtain a trained multi-agent model.

[0095] Specifically include: the current neural network interacts with the environment, and the current state set s t ={SOC,v,U,Acc,Br,n dm ,T dm ,n cm ,T cm ,n FW ,} T Input the current neural network, and Agent 1 and Agent 2 will be based on the state variable s t Give the corresponding action variables respectively and The power system then performs the corresponding power output and feeds back the state s at the next moment t+1 and reward r t 1 and r t 2 . and The experience data are stored in the experience replay pool and the corresponding algorithms are updated respectively. Through repeated sampling and training, these experience data are used to further optimize the policy network of the multi-agent model, improve the accuracy of decision-making and the overall energy efficiency of the system.

[0096] The deep reinforcement learning algorithm SAC used in step B contains Critic and Actor networks. The algorithm update includes two main steps: updating the Critic network and updating the Actor network.

[0097] In the Critic network, the Q-value function is used to evaluate the t and action variables The total reward that may be obtained in the future. SAC uses a dual Q network to avoid the over-estimation problem, usually with two Q networks Q1 and Q2.

[0098] The goal of the Q-value function is to minimize the following Bellman error:

[0099]

[0100] Where E is the expected value, is the state s obtained according to the strategy π t and action a tThe joint distribution formed, and are the outputs of the Q1 and Q2 networks, respectively, indicating that in state s t Select Action The estimated return after t 1 is the reward function of agent 1, γ is the discount factor, is the minimum value of the two Q networks, which is used as the value estimate of the next state. i takes the value of 1 or 2, and a is the temperature coefficient. is the current strategy in s t+1 The following action distribution.

[0101] The goal of the Actor network is to optimize the policy so that it not only maximizes the reward but also maintains a certain level of entropy. Policy optimization in SAC is achieved by maximizing the following objective function:

[0102]

[0103] in, is the entropy term of the policy, is the output of the Q network, indicating that in state s t Select Action Estimated return after .

[0104] The deep reinforcement learning PPO algorithm used includes Policy and Value networks. The Policy network strategy update is achieved by maximizing the following objective function:

[0105]

[0106] Among them, E t is the expected value at time t, For the current strategy in state s t Take action The probability of is the probability that the old strategy performs the same action in the same state, is the advantage function, is the probability ratio of the current strategy to the old strategy, is the clipping function and ε is the clipping parameter.

[0107] The Value network is responsible for providing each state s t Estimate its state value. This estimate of state value is to calculate the advantage function The core of the function is to measure the choice of an action a in a certain state. t The superiority of the behavior compared to the average. The estimate of the advantage function is usually calculated using the TD-error.

[0108] Initialize the current policy network parameters and value network parameters, and set the corresponding hyperparameters such as the shear coefficient and learning rate to ensure that the strategy of each agent can effectively explore and utilize the environment, and finally obtain the initialized multi-agent model.

[0109] Figure 4 This graph compares the battery SOC of an electric flywheel hybrid vehicle using the proposed multi-agent strategy with that of a traditional single-motor new energy commercial vehicle model under the same vehicle parameters. Simulation results demonstrate that this energy management strategy improves vehicle economy, enhances the safety and stability of fuel-powered vehicles, reduces battery consumption, and significantly enhances energy management efficiency.

Claims

1. An energy management method for electric flywheel hybrid vehicles based on multi-agent deep reinforcement learning, characterized in that: The following steps are involved: (1) Simulate the actual development situation, establish an electric flywheel hybrid vehicle model, and define six working modes of the vehicle model; the electric flywheel hybrid vehicle model includes a vehicle driving dynamics model, a planetary gear system model, a motor model, a battery model, and a flywheel model. The six working modes are divided into driving mode EV1, driving mode EV2, driving mode EV3, braking mode ERGB1, braking mode ERGB2, and parking mode Stop; EV1: When the vehicle starts, the main drive motor and the control motor both work as electric motors, outputting positive torque and positive speed. The flywheel is locked and does not participate in driving. ERGB1: In this mode, the main drive motor idles, and the control motor works as a generator, outputting negative torque and positive speed. The flywheel speed increases, and together with the control motor, it provides partial braking force. ERGB2: During braking, when the vehicle speed drops to a critical value, the control motor changes from positive speed to reverse speed, but still outputs negative torque, consuming electrical energy. The flywheel continues to accelerate and rotates in coordination with the control motor to provide braking force, converting the kinetic energy and electrical energy recovered by braking into mechanical energy of the flywheel. EV2: The main drive motor outputs positive torque and positive speed to work as an electric motor. The motor is controlled to output positive torque and negative speed in generator mode, and the flywheel speed decreases, releasing stored energy. EV3: During further acceleration, the motor speed is controlled to change from reverse to forward and enter motor mode. The main drive motor remains in motor mode, and the flywheel continues to release mechanical energy to accelerate the vehicle. Stop: When the vehicle is stationary, the planetary carrier speed is zero. At this time, the motor is controlled to output positive torque, enter reverse deceleration and operate in generator mode. At the same time, the flywheel begins to decelerate and release its stored mechanical energy, which is gradually converted into battery energy. (2) Taking the electric flywheel hybrid vehicle model as the environment, S t As the state variable, a1 and a2 are used as the execution actions to build a multi-agent model based on a neural network. The multi-agent model includes agent 1 and agent 2. Agent 1 uses the SAC reinforcement learning algorithm, and agent 2 uses the PPO reinforcement learning algorithm to determine the state variables of each agent. (3) Design a task-adapted reward function to guide the agent to optimize its behavior through feedback signals, and clarify the action space to define its executable operations in the environment; (4) Preprocess the state variables and then use the preprocessed data to train multi-agents under CHTC conditions to simulate the energy consumption and performance of the vehicle; The action variable a1 of agent 1 is the torque output ratio of the main drive motor, and the action variable a2 of agent 2 is the six working modes. The state variables and action variables are set as follows: s t ={SOC,v,U,Acc,Br,n dm ,T dm ,n cm ,T cm ,n Fw } T Among them, SOC is the battery state of charge, v is the vehicle speed, U is the battery voltage, Acc is the accelerator pedal opening, Br is the decelerator pedal opening, n dm is the main drive motor speed, T dm The output torque of the main drive motor, n cm To control the motor speed, T cm To control the motor output torque, n FW is the flywheel speed; The specific form of the reward function is as follows: Among them, r SOC ,r v ,r Fw and r mode Represent the reward functions of SOC, vehicle speed, flywheel speed, and mode switching respectively. β1, β2, β3, β4, β5, and β6 represent the weights of each part of the reward, r t 1 represents the reward function of agent 1, r t 2 Represents the reward function of agent 2.

2. The electric flywheel hybrid vehicle energy management method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: Step (4) is as follows: the current neural network interacts with the environment and sets the current state set s t ={SOC,v,U,Acc,Br,n dm ,T dm ,n cm ,T cm ,n FW ,} T Input the current neural network, and Agent 1 and Agent 2 will be based on the state variable s t Give the corresponding action variables respectively and The power system then performs the corresponding power output and feeds back the state s at the next moment t+1 and reward r t 1 and r t 2 . and The experience data are stored in the experience replay pool and the corresponding algorithms are updated respectively. Through repeated sampling and training, these experience data are used to further optimize the policy network of the multi-agent model.

3. The electric flywheel hybrid vehicle energy management method based on multi-agent deep reinforcement learning according to claim 2 is characterized in that: The deep reinforcement learning algorithm SAC contains Critic and Actor networks. The algorithm update includes two main steps: updating the Critic network and updating the Actor network. In the Critic network, the Q-value function is used to evaluate the t and action variables Under the total reward that may be obtained in the future, SAC uses a dual Q network to avoid the over-estimation problem. There are usually two Q networks Q1 and Q2. The goal of the Q-value function is to minimize the following Bellman error: Where E is the expected value, is the state s obtained according to the strategy π t and action a t The joint distribution formed, is the output of the Q network, indicating that in state s t Select Action The estimated return after t 1 is the reward function of agent 1, γ is the discount factor, is the minimum value of the two Q networks, which is used as the value estimate of the next state, a is the temperature coefficient, is the current strategy in s t+1 The action distribution under The goal of the Actor network is to optimize the policy so that it not only maximizes the reward but also maintains a certain degree of entropy. Policy optimization in SAC is achieved by maximizing the following objective function: in, is the entropy term of the policy, is the output of the Q network, indicating that in state s t Select Action The estimated return after The deep reinforcement learning PPO algorithm used includes Policy and Value networks. The Policy network strategy update is achieved by maximizing the following objective function: Among them, E t is the expected value at time t, For the current strategy in state s t Take action The probability of is the probability that the old strategy performs the same action in the same state, is the advantage function, is the probability ratio of the current strategy to the old strategy, is the clipping function and ε is the clipping parameter.

4. The electric flywheel hybrid vehicle energy management method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The specific description of the vehicle's driving dynamics model at the starting stage is as follows: Where m is the vehicle mass; g is the acceleration of gravity; f is the rolling resistance coefficient; θ is the road slope; C D is the air resistance coefficient; A is the frontal area of ​​the vehicle; ρ is the air density; v is the vehicle speed; δ is the rotational mass conversion coefficient; R c is the wheel radius; T b is the required mechanical braking torque; T dem is the vehicle demand torque; The planetary gear system consists of planetary gear set 1 and planetary gear set 2. In the driving state, the power transmission is calculated as follows: Among them, T dm 、n dm and T cm 、n cm The torque and speed of the main drive motor and the control motor; n c is the vehicle wheel speed; η c is the mechanical transmission efficiency of the power system; r PG1 、r PG2 and r c They are planetary gear set 1, planetary gear set 2 and final reduction ratio, In the braking state, the power is transmitted as follows: Among them, T Fw 、n Fw are the torque and speed of the flywheel, r Fw is the flywheel reducer transmission ratio, The motor model is shown in the following formula: Among them, η m is the motor efficiency, P out is the output power of the motor, P in is the input power, U s is the input voltage of the motor, I s is the input current of the motor, T is the motor torque, n is the motor speed, The battery model is shown in the following formula: Among them, SOC0 is the state of charge at the initial state (t0), SOC is the state of charge at t, Q nom is the nominal capacity of the battery, I b is the battery current, U oc is the open circuit voltage of the battery, R0 is the internal resistance of the battery, P b is the instantaneous output power of the battery, P bat is the battery power, The flywheel model is shown in the following formula: Among them, E Fw is the flywheel energy, ω Fw is the angular velocity of the flywheel, I is the moment of inertia of the flywheel, n Fw is the flywheel speed, P Fw is the instantaneous flywheel power.

Images