Hybrid electric vehicle ecological driving strategy based on multi-agent reinforcement learning

Abstract

The invention provides a hybrid electric vehicle ecological driving strategy based on multi-agent reinforcement learning, which is different from an ecological driving strategy based on traditional reinforcement learning, and a multi-agent cooperative training framework is innovatively designed, so that decoupling processing of upper-layer speed planning and lower-layer energy management can be effectively avoided, and the reliability of the hybrid electric vehicle is improved. The global collaborative optimization is realized, and a better energy-saving effect can be obtained. Compared with the prior art, the strategy provided by the invention has the advantages that the calculation time is greatly shortened, and better real-time performance is provided in control.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent networked vehicle traffic control, and specifically relates to an ecological driving control method for an intelligent networked hybrid vehicle at a signalized intersection based on multi-agent reinforcement learning. Background Art

[0002] In urban traffic networks, vehicles in different directions accelerate and decelerate at intersections under the control of traffic lights, which makes congestion very likely to occur, and is also an important cause of reduced driving comfort and increased vehicle energy consumption. Intelligent connected vehicles can receive the timing information of traffic lights and the remaining time of green lights in real time through vehicle-road cooperative technology (V2I), which helps to optimize the speed of traffic and alleviate congestion. Existing eco-driving strategies for hybrid vehicles at signalized intersections usually adopt optimization-based methods, which can be mainly divided into two categories: global optimal and instantaneous optimal. Among them, the global optimal method assumes that the driving environment and future conditions of the vehicle are completely known. The dynamic programming algorithm is a representative of the global optimal method and is suitable for dealing with highly nonlinear constraints such as traffic lights. However, due to the constant changes and high complexity of the traffic environment, especially at intersections, it is difficult to fully predict future driving conditions. In addition, the dynamic programming algorithm has a large amount of calculation and a long calculation time, so it cannot meet the requirements of practical applications. To address this problem, an eco-driving method based on instantaneous optimality has emerged. As a representative of the instantaneous optimal method, the model predictive control algorithm (MPC) replaces the global driving cycle by sliding short-term driving predictions and successively optimizes the prediction time domain to achieve real-time control. However, the performance of MPC is highly dependent on the accuracy of state transition predictions, while the high uncertainty of traffic conditions will limit the prediction accuracy, thereby affecting its control effect. Summary of the invention

[0003] In view of this, in order to solve the technical problems existing in the art, the present invention provides an eco-driving strategy for a hybrid electric vehicle based on multi-agent reinforcement learning, which specifically includes the following steps:

[0004] Step 1: For the traffic scenario of hybrid electric vehicles at signalized intersections, a signal light traffic environment model consisting of a vehicle powertrain model and a signal light traffic environment model with V2I function is established as the upper layer of the strategy to include speed v, acceleration a and engine power. The vehicle information, including the distance d between the vehicle and the signalized intersection ahead inter 、State of the signal light (phase) p and the remaining time r The traffic information is collected and processed to simulate the driving conditions of vehicles at signalized intersections and calculate the corresponding energy consumption;

[0005] Step 2: Construct a multi-agent environment interaction model by modeling the problem as a Markov decision process, which consists of an upper-level speed planning strategy agent and a lower-level energy management strategy agent, each of which corresponds to a Markov decision process; based on the various vehicle information and traffic information collected and processed in step 1, define the state space, action space and reward function of the multi-agent in turn;

[0006] Step 3: Conduct centralized training based on reinforcement learning for multiple agents, define the global value function of centralized training, and enable the upper-level speed planning strategy agent and the lower-level energy management strategy agent to select actions and obtain rewards based on their own and global states respectively by maximizing the value function, and finally realize the update of the upper and lower strategies;

[0007] Step 4: Conduct simulation and actual application tests on the trained multi-agent to verify its effectiveness, and then apply the strategy to real traffic scenarios.

[0008] Furthermore, in step 2, the following state spaces are defined for the upper-level speed planning strategy agent and the lower-level energy management strategy agent respectively:

[0009]

[0010] Where s1 is the state space of the upper-level speed planning strategy agent, s2 is the state space of the lower-level energy management strategy agent, and SoC is the remaining power of the power battery;

[0011] Define the following action space:

[0012] a1={a|a min ≤a≤a max}

[0013] a2={P e |P e,min ≤P e ≤P e,max}

[0014] Where a1 is the action space of the upper layer speed planning strategy agent, a2 is the action space of the lower layer energy management strategy agent, and a min and a max are the minimum and maximum acceleration, P e is the engine power of the hybrid vehicle, P e,min and P e,max are the minimum and maximum values ​​of engine power respectively;

[0015] And define the following reward function:

[0016]

[0017] Where w1 and w2 are weight coefficients, r1 is the reward function of the upper-level speed planning strategy agent, which involves three factors: time to destination, energy consumption, and traffic safety. end is the distance from the vehicle to the destination, v red is the speed of the vehicle when running a red light, which is used to measure the degree of constraint violation of the agent. r2 is the reward function of the lower-level energy management strategy agent, which involves two factors: energy consumption and SoC stability. SoC ref is the reference SoC and t is the current time.

[0018] Furthermore, in step 3, the upper-level speed planning strategy agent and the lower-level energy management strategy agent are specifically defined as π1(a1|s1) and π2(a2|s2), respectively, so that they can select corresponding actions in their respective state spaces and can access the global state s={s1,s2} and the actions and reward information of other agents;

[0019] The global value function is defined as follows:

[0020]

[0021] In the formula, Q i (s,a1,a2) is the value function of each agent, γ is the discount factor, and s0 is the initial state.

[0022] Furthermore, when each intelligent agent is used in actual ecological driving tasks after training, it makes independent decisions based entirely on its own local information, without relying on the observation of other intelligent agents or the distributed form of the global state, so as to reduce the communication and computing burden between intelligent agents and enhance the real-time performance of policy control.

[0023] The hybrid electric vehicle eco-driving strategy based on multi-agent reinforcement learning provided by the present invention is different from the eco-driving strategy based on traditional reinforcement learning. It innovatively designs a multi-agent collaborative training framework, which can effectively avoid the decoupling of upper-level speed planning and lower-level energy management, achieve global collaborative optimization, and obtain better energy-saving effects. Compared with the existing technology, the strategy of the present invention greatly reduces the calculation time and provides better real-time performance in control. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a control framework diagram of the strategy provided by the present invention. DETAILED DESCRIPTION

[0025] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The hybrid vehicle ecological driving strategy based on multi-agent reinforcement learning provided by the present invention is as follows: Figure 1 As shown, the specific steps include:

[0027] Step 1: For the traffic scenario of hybrid electric vehicles at signalized intersections, a signal light traffic environment model consisting of a vehicle powertrain model and a signal light traffic environment model with V2I function is established as the upper layer of the strategy to include speed v, acceleration a and engine power. The vehicle information, including the distance d between the vehicle and the signalized intersection ahead inter 、State of the signal light (phase) p and the remaining time r The traffic information is collected and processed to simulate the driving conditions of vehicles at signalized intersections and calculate the corresponding energy consumption;

[0028] Step 2: Construct a multi-agent environment interaction model by modeling the problem as a Markov decision process, which consists of an upper-level speed planning strategy agent and a lower-level energy management strategy agent, each of which corresponds to a Markov decision process; based on the various vehicle information and traffic information collected and processed in step 1, define the state space, action space and reward function of the multi-agent in turn;

[0029] Step 3: Conduct centralized training based on reinforcement learning for multiple agents, define the global value function of centralized training, and enable the upper-level speed planning strategy agent and the lower-level energy management strategy agent to select actions and obtain rewards based on their own and global states respectively by maximizing the value function, and finally realize the update of the upper and lower strategies;

[0030] Step 4: Conduct simulation and actual application tests on the trained multi-agent to verify its effectiveness, and then apply the strategy to real traffic scenarios.

[0031] In a preferred embodiment of the present invention, in step 2, the following state spaces are defined for the upper layer speed planning strategy agent and the lower layer energy management strategy agent respectively:

[0032]

[0033] Where s1 is the state space of the upper-level speed planning strategy agent, s2 is the state space of the lower-level energy management strategy agent, and SoC is the remaining power of the power battery;

[0034] Define the following action space:

[0035] a1={a|a min ≤a≤a max}

[0036] a2={P e |P e,min ≤P e ≤P e,max}

[0037] Where a1 is the action space of the upper layer speed planning strategy agent, a2 is the action space of the lower layer energy management strategy agent, and a min and a max are the minimum and maximum acceleration, P e is the engine power of the hybrid vehicle, P e,min and P e,max are the minimum and maximum values ​​of engine power respectively;

[0038] And define the following reward function:

[0039]

[0040] Where w1 and w2 are weight coefficients, r1 is the reward function of the upper-level speed planning strategy agent, which involves three factors: time to destination, energy consumption, and traffic safety. end is the distance from the vehicle to the destination, v red is the speed of the vehicle when running a red light, which is used to measure the degree of constraint violation of the agent. r2 is the reward function of the lower-level energy management strategy agent, which involves two factors: energy consumption and SoC stability. SoC ref is the reference SoC and t is the current time.

[0041] In a preferred embodiment of the present invention, in step 3, the upper-level speed planning strategy agent and the lower-level energy management strategy agent are specifically defined as π1(a1|s1) and π2(a2|s2), respectively, so that they select corresponding actions in their respective state spaces and can access the global state s={s1,s2} and the actions and reward information of other agents;

[0042] The global value function is defined as follows:

[0043]

[0044] In the formula, Q i(s,a1,a2) is the value function of each agent, γ is the discount factor, and s0 is the initial state.

[0045] In a preferred embodiment of the present invention, when each intelligent agent after training is used in an actual ecological driving task, it makes independent decisions based entirely on its own local information, without relying on the observation of other intelligent agents or a distributed form of the global state, so as to reduce the communication and computing burden between intelligent agents and enhance the real-time performance of strategy control.

[0046] It should be understood that the size of the serial numbers of the steps in the embodiment of the present invention does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0047] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An eco-driving strategy for hybrid electric vehicles based on multi-agent reinforcement learning, characterized by: The specific steps include: Step 1: For the traffic scenario of hybrid electric vehicles at signalized intersections, a signal light traffic environment model consisting of a vehicle powertrain model and a signal light traffic environment model with V2I function is established as the upper layer of the strategy to include speed v, acceleration a and engine power. The vehicle information, as well as the distance d between the vehicle and the signalized intersection ahead inter , the phase of the signal light p and the remaining time r The traffic information is collected and processed to simulate the driving conditions of vehicles at signalized intersections and calculate the corresponding energy consumption; Step 2: Construct a multi-agent environment interaction model by modeling the problem as a Markov decision process, which consists of an upper-level speed planning strategy agent and a lower-level energy management strategy agent, each of which corresponds to a Markov decision process; Based on the various vehicle information and traffic information collected and processed in step 1, define the state space, action space and reward function of the multi-agent in turn; Step 3: Conduct centralized training based on reinforcement learning for multiple agents, define the global value function of centralized training, and enable the upper-level speed planning strategy agent and the lower-level energy management strategy agent to select actions and obtain rewards based on their own and global states respectively during training, by maximizing the value function, and finally realize the update of the upper and lower strategies; Step 4: Conduct simulation and actual application tests on the trained multi-agent to verify its effectiveness, and then apply the multi-agent to ecological driving tasks in real traffic scenarios.

2. The hybrid electric vehicle eco-driving strategy based on multi-agent reinforcement learning as claimed in claim 1, characterized in that: In step 2, the following state spaces are defined for the upper-level speed planning strategy agent and the lower-level energy management strategy agent respectively: Where s1 is the state space of the upper-level speed planning strategy agent, s2 is the state space of the lower-level energy management strategy agent, and SoC is the remaining power of the power battery; Define the following action space: a1={a|a min ≤a≤a max } a2={P e |P e,min ≤P e ≤P e,max } Where a1 is the action space of the upper layer speed planning strategy agent, a2 is the action space of the lower layer energy management strategy agent, and a min and a max are the minimum and maximum acceleration, P e is the engine power of the hybrid vehicle, P e,min and P e,max are the minimum and maximum values ​​of engine power respectively; And define the following reward function: Where w1 and w2 are weight coefficients, r1 is the reward function of the upper-level speed planning strategy agent, which involves three factors: time to destination, energy consumption, and traffic safety. end is the distance from the vehicle to the destination, v red is the speed of the vehicle when running a red light, which is used to measure the degree of constraint violation of the agent. r2 is the reward function of the lower-level energy management strategy agent, which involves two factors: energy consumption and SoC stability. SoC ref is the reference SoC and t is the current time.

3. The hybrid electric vehicle eco-driving strategy based on multi-agent reinforcement learning as claimed in claim 1, characterized in that: In step 3, the upper-level speed planning strategy agent and the lower-level energy management strategy agent are defined as π1(a1|s1) and π2(a2|s2) respectively, so that they can select corresponding actions in their respective state spaces and can access the global state s={s1,s2} and the actions and reward information of other agents; The global value function is defined as follows: In the formula, Q i (s,a1,a2) is the value function of each agent, γ is the discount factor, and s0 is the initial state.

4. The hybrid electric vehicle eco-driving strategy based on multi-agent reinforcement learning as claimed in claim 3, characterized in that: When each intelligent agent after training is applied in the actual ecological driving task, it makes independent decisions based entirely on its own local information, without relying on the observation of other intelligent agents or the distributed form of the global state, so as to reduce the communication and computing burden between intelligent agents and enhance the real-time performance of policy control.

Images