A hybrid electric vehicle eco-driving strategy based on multi-agent reinforcement learning

Abstract

This invention provides a hybrid electric vehicle eco-driving strategy based on multi-agent reinforcement learning. Unlike traditional reinforcement learning-based eco-driving strategies, it innovatively designs a multi-agent collaborative training framework, effectively avoiding the decoupling of upper-layer speed planning and lower-layer energy management, achieving global collaborative optimization, and obtaining superior energy-saving performance. Compared to existing technologies, this invention significantly reduces computation time and provides better real-time performance in control.

Description

Technical Field

[0001] This invention belongs to the field of intelligent connected vehicle traffic control technology, specifically relating to an ecological driving control method for intelligent connected hybrid vehicles at signalized intersections based on multi-agent reinforcement learning. Background Technology

[0002] In urban traffic networks, traffic lights cause vehicles from different directions to accelerate and decelerate at intersections, easily leading to congestion and significantly contributing to reduced driving comfort and increased vehicle energy consumption. Intelligent connected vehicles, through vehicle-to-infrastructure (V2I) technology, can receive real-time traffic light timing information and remaining green light time, helping to optimize traffic flow and alleviate congestion. Existing eco-driving strategies for hybrid vehicles at signalized intersections typically employ optimization-based methods, mainly categorized into global optimal and instantaneous optimal approaches. Global optimal methods assume complete knowledge of the vehicle's driving environment and future conditions; dynamic programming is a representative of global optimal methods and is suitable for handling highly nonlinear constraints such as traffic lights. However, due to the constantly changing and highly complex traffic environment, especially at intersections, it is difficult to fully predict future driving conditions. Furthermore, dynamic programming algorithms are computationally intensive and time-consuming, thus failing to meet the requirements of practical applications. To address this issue, eco-driving methods based on instantaneous optimum have emerged. Model Predictive Control (MPC), as a representative of instantaneous optimum methods, replaces the global driving cycle with a sliding short-term driving prediction and successively optimizes the prediction time domain to achieve real-time control. However, the performance of MPC is highly dependent on the accuracy of state transition predictions, and the high uncertainty of traffic conditions limits prediction accuracy, thus affecting its control effect. Summary of the Invention

[0003] In view of this, and to address the technical problems existing in this field, the present invention provides a hybrid electric vehicle ecological driving strategy based on multi-agent reinforcement learning, specifically including the following steps:

[0004] Step 1: For the scenario of hybrid vehicles passing through signalized intersections, establish a signalized traffic environment model consisting of a vehicle powertrain model and a signalized traffic environment model with V2I functionality. This model will serve as the upper-level strategy for handling parameters including speed v, acceleration a, and engine power. Vehicle information, including the distance d between the vehicle and the signalized intersection ahead. inter The state (phase) of the traffic lights p and remaining time l r Traffic information is collected and processed to simulate the driving conditions of vehicles at signalized intersections and calculate the corresponding energy consumption.

[0005] Step 2: Construct a multi-agent environment interaction model by modeling the problem as a Markov decision process. The model consists of two parts: an upper-level speed planning strategy agent and a lower-level energy management strategy agent, each corresponding to a Markov decision process. Based on the various vehicle and traffic information collected and processed in Step 1, define the state space, action space, and reward function of the multi-agent in sequence.

[0006] Step 3: Conduct centralized training of the multi-agent system based on reinforcement learning, define the global value function for centralized training, and enable the upper-layer velocity planning strategy agent and the lower-layer energy management strategy agent to select actions and obtain rewards based on their own and global states by maximizing the value function, thereby realizing the update of the upper and lower layer policies.

[0007] Step 4: After training, conduct simulation and real-world application tests on the multi-agent system to verify its effectiveness, and then apply the strategy to real traffic scenarios.

[0008] Furthermore, in step two, the following state spaces are defined for the upper-layer velocity planning strategy agent and the lower-layer energy management strategy agent, respectively:

[0009]

[0010] In the formula, s1 is the state space of the upper-layer speed planning strategy agent, s2 is the state space of the lower-layer energy management strategy agent, and SoC is the remaining power of the power battery.

[0011] Define the following action space:

[0012] a1={a|a min ≤a≤a max}

[0013] a2={P e |P e,min ≤P e ≤P e,max}

[0014] In the formula, a1 represents the action space of the upper-layer velocity planning strategy agent, a2 represents the action space of the lower-layer energy management strategy agent, and a min and a max These are the minimum and maximum acceleration values, P. e P represents the engine power of a hybrid electric vehicle. e,min and P e,max These are the minimum and maximum engine power values, respectively.

[0015] And define the following reward function:

[0016]

[0017] In the formula, w1 and w2 are weight coefficients, r1 is the reward function of the upper-level speed planning agent, which involves three factors: arrival time, energy consumption, and traffic safety. end v represents the distance the vehicle travels to its destination. red The speed of the vehicle when it runs a red light is used to measure the degree of constraint violation by the agent. r2 is the agent reward function of the lower-level energy management policy, which involves two factors: energy consumption and SoC stability. ref For reference SoC, t represents the current time.

[0018] Furthermore, in step three, the upper-layer velocity planning strategy agent and the lower-layer energy management strategy agent are specifically defined as π1(a1|s1) and π2(a2|s2) respectively, so that they can select corresponding actions in their respective state spaces and access the global state s={s1,s2} and the action and reward information of other agents.

[0019] The global value function is defined in the following form:

[0020]

[0021] In the formula, Q i (s,a1,a2) represents the value function of each agent, γ is the discount factor, and s0 is the initial state.

[0022] Furthermore, when the trained agents are applied in actual eco-driving tasks, they make independent decisions based entirely on their own local information, without relying on the observations of other agents or the distributed form of the global state. This reduces the communication and computational burden between agents and enhances the real-time performance of policy control.

[0023] The hybrid electric vehicle eco-driving strategy based on multi-agent reinforcement learning provided by this invention differs from traditional reinforcement learning-based eco-driving strategies. It innovatively designs a multi-agent collaborative training framework, effectively avoiding the decoupling of upper-layer speed planning and lower-layer energy management, achieving global collaborative optimization, and obtaining superior energy-saving effects. Compared to existing technologies, this invention's strategy significantly reduces computation time and provides better real-time performance in control. Attached Figure Description

[0024] Figure 1 This is a control framework diagram of the strategy provided by the present invention. Detailed Implementation

[0025] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] The present invention provides a hybrid electric vehicle ecological driving strategy based on multi-agent reinforcement learning, such as... Figure 1 As shown, the specific steps include:

[0027] Step 1: For the scenario of hybrid vehicles passing through signalized intersections, establish a signalized traffic environment model consisting of a vehicle powertrain model and a signalized traffic environment model with V2I functionality. This model will serve as the upper-level strategy for handling parameters including speed v, acceleration a, and engine power. Vehicle information, including the distance d between the vehicle and the signalized intersection ahead. inter The state (phase) of the traffic lights p and remaining time l r Traffic information is collected and processed to simulate the driving conditions of vehicles at signalized intersections and calculate the corresponding energy consumption.

[0028] Step 2: Construct a multi-agent environment interaction model by modeling the problem as a Markov decision process. The model consists of two parts: an upper-level speed planning strategy agent and a lower-level energy management strategy agent, each corresponding to a Markov decision process. Based on the various vehicle and traffic information collected and processed in Step 1, define the state space, action space, and reward function of the multi-agent in sequence.

[0029] Step 3: Conduct centralized training of the multi-agent system based on reinforcement learning, define the global value function for centralized training, and enable the upper-layer velocity planning strategy agent and the lower-layer energy management strategy agent to select actions and obtain rewards based on their own and global states by maximizing the value function, thereby realizing the update of the upper and lower layer policies.

[0030] Step 4: After training, conduct simulation and real-world application tests on the multi-agent system to verify its effectiveness, and then apply the strategy to real traffic scenarios.

[0031] In a preferred embodiment of the present invention, the following state spaces are defined in step two for the upper-layer velocity planning strategy agent and the lower-layer energy management strategy agent, respectively:

[0032]

[0033] In the formula, s1 is the state space of the upper-layer speed planning strategy agent, s2 is the state space of the lower-layer energy management strategy agent, and SoC is the remaining power of the power battery.

[0034] Define the following action space:

[0035] a1={a|a min ≤a≤a max}

[0036] a2={P e |P e,min ≤P e ≤P e,max}

[0037] In the formula, a1 represents the action space of the upper-layer velocity planning strategy agent, a2 represents the action space of the lower-layer energy management strategy agent, and a min and a max These are the minimum and maximum acceleration values, P. e P represents the engine power of a hybrid electric vehicle. e,min and P e,max These are the minimum and maximum engine power values, respectively.

[0038] And define the following reward function:

[0039]

[0040] In the formula, w1 and w2 are weight coefficients, r1 is the reward function of the upper-level speed planning agent, which involves three factors: arrival time, energy consumption, and traffic safety. end v represents the distance the vehicle travels to its destination. red The speed of the vehicle when it runs a red light is used to measure the degree of constraint violation by the agent. r2 is the agent reward function of the lower-level energy management policy, which involves two factors: energy consumption and SoC stability. ref For reference SoC, t represents the current time.

[0041] In a preferred embodiment of the present invention, in step three, the upper-layer speed planning strategy agent and the lower-layer energy management strategy agent are specifically defined as π1(a1|s1) and π2(a2|s2) respectively, so that they can select corresponding actions in their respective state spaces and access the global state s={s1,s2} and the action and reward information of other agents.

[0042] The global value function is defined in the following form:

[0043]

[0044] In the formula, Q i(s,a1,a2) represents the value function of each agent, γ is the discount factor, and s0 is the initial state.

[0045] In a preferred embodiment of the present invention, when each agent is trained and applied in actual eco-driving tasks, it makes independent decisions based entirely on its own local information, without relying on the observations of other agents or the distributed form of the global state, so as to reduce the communication and computing burden between agents and enhance the real-time performance of policy control.

[0046] It should be understood that the sequence number of each step in the embodiments of the present invention does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0047] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A hybrid electric vehicle ecological driving strategy control method based on multi-agent reinforcement learning, characterized in that: Specifically, the following steps are included: Step 1: For the scenario of hybrid vehicles passing through signalized intersections, establish a signalized traffic environment model consisting of a vehicle powertrain model and a signalized traffic environment model with V2I functionality. This model will serve as the upper-level strategy for handling parameters including speed v, acceleration a, and engine power. Vehicle information, including the distance d between the vehicle and the signalized intersection ahead. inter Phase l of the traffic lights p and remaining time l r Traffic information is collected and processed to simulate the driving conditions of vehicles at signalized intersections and calculate the corresponding energy consumption. Step 2: Construct a multi-agent environment interaction model by modeling the problem as a Markov decision process. This model consists of two parts: an upper-level velocity planning strategy agent and a lower-level energy management strategy agent. The following state spaces are defined for the upper-level velocity planning strategy agent and the lower-level energy management strategy agent, respectively: In the formula, s1 is the state space of the upper-layer speed planning strategy agent, s2 is the state space of the lower-layer energy management strategy agent, and SoC is the remaining power of the power battery. Define the following action space: a1={a|a min ≤a≤a max } a2={P e |P e,min ≤P e ≤P e,max } In the formula, a1 represents the action space of the upper-layer velocity planning strategy agent, a2 represents the action space of the lower-layer energy management strategy agent, and a min and a max These are the minimum and maximum acceleration values, P. e P represents the engine power of a hybrid electric vehicle. e,min and P e,max These are the minimum and maximum engine power values, respectively. And define the following reward function: In the formula, w1 and w2 are weight coefficients, r1 is the reward function of the upper-level speed planning agent, which involves three factors: arrival time, energy consumption, and traffic safety. end v represents the distance the vehicle travels to its destination. red The speed of the vehicle when it runs a red light is used to measure the degree of constraint violation by the agent. r2 is the agent reward function of the lower-level energy management policy, which involves two factors: energy consumption and SoC stability. ref For reference SoC, t is the current time; Step 3: Conduct centralized training of the multi-agent system based on reinforcement learning. Define the global value function for centralized training. During training, enable the upper-layer velocity planning strategy agent and the lower-layer energy management strategy agent to select actions and obtain rewards based on their own and global states by maximizing the value function, and finally realize the updating of the upper and lower layer policies. Step 4: After training, conduct simulation and real-world application tests on the multi-agent system to verify its effectiveness. Then, apply the multi-agent system to ecological driving tasks in real traffic scenarios.

2. The hybrid electric vehicle ecological driving strategy control method based on multi-agent reinforcement learning as described in claim 1, characterized in that: In step three, the upper-layer velocity planning strategy agent and the lower-layer energy management strategy agent are defined as π1(a1|s1) and π2(a2|s2) respectively, so that they can select corresponding actions in their respective state spaces and access the global state s={s1,s2} and the action and reward information of other agents. The global value function is defined in the following form: In the formula, Q i (s,a1,a2) represents the value function of each agent, γ is the discount factor, and s0 is the initial state.

3. The hybrid electric vehicle ecological driving strategy control method based on multi-agent reinforcement learning as described in claim 2, characterized in that: Once trained, each agent makes independent decisions based entirely on its own local information when applied in actual eco-driving tasks, without relying on the observations of other agents or the distributed form of the global state. This reduces the communication and computational burden between agents and enhances the real-time performance of policy control.

Images