Economical driving method for connected new energy vehicles in complex traffic scenarios

Abstract

The present invention relates to an economical driving strategy for hybrid vehicles in complex traffic scenarios based on deep reinforcement learning, and belongs to the field of new energy vehicles. The method comprises: constructing an interactive multi-lane and multi-traffic light training scenario: using a vehicle kinematic model to describe the longitudinal motion of the vehicle in the training scenario, simplifying the lane changing process of the vehicle into a transient process, and using a rule-based decision model to control other vehicles, so that the environment is interactive; constructing a decision model based on maximum entropy deep reinforcement learning, including state space, action space, reward function, strategy model, evaluation model and experience replay pool; constructing target vehicle safety constraints, including longitudinal acceleration safety constraints and lateral lane change decision safety constraints, which are used to avoid dangerous behaviors such as vehicle collisions and violations of traffic rules; training a decision model based on maximum entropy deep reinforcement learning. The present invention improves the economy of autonomous driving vehicles by using deep reinforcement learning.

Description

Technical Field

[0001] The present invention belongs to the field of new energy vehicles and relates to an economical driving method for networked new energy vehicles in complex traffic scenarios based on deep reinforcement learning. Background Art

[0002] With the rapid development of urban transportation, traffic congestion and environmental pollution are becoming increasingly serious. Hybrid electric vehicles have become an important means to solve these problems due to their high fuel efficiency and low emissions. However, in the complex urban traffic environment with multiple lanes and multiple traffic lights, how to fully utilize the energy-saving potential of hybrid electric vehicles remains a challenge.

[0003] At present, the economic driving strategy of hybrid vehicles mainly relies on pre-set driving rules. The advantages are simple implementation, small amount of calculation, and suitable for real-time application, but lack flexibility and cannot dynamically adapt to complex and changeable urban traffic conditions, resulting in poor fuel efficiency and driving comfort. In addition, mathematical models and optimization algorithms are used to find the best driving strategy to maximize fuel economy and minimize emissions. Although the effect is better than the method of pre-set rules, it usually requires a lot of computing resources and is difficult to apply in real-time driving. It also relies on accurate traffic prediction models. If the model is not accurate enough, the optimization effect will also be affected. Although these methods have improved the economic driving performance of hybrid vehicles to a certain extent, there are still problems such as insufficient flexibility, high computational complexity, and strong dependence on traffic prediction. Learning-based methods can use driving data to automatically generate generalized driving experience, showing obvious advantages in adaptability and robustness in complex environments. However, current learning-based decision-making methods are mostly focused on relatively simple highway scenes, or mainly consider safety issues in the process of autonomous driving. They will no longer be applicable to driving economy in multi-lane and multi-traffic light scenes that interact with other vehicles.

[0004] Therefore, a new hybrid vehicle economic driving strategy for complex traffic scenarios is urgently needed to solve the above problems. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide an economical driving method for networked new energy vehicles in complex traffic scenarios, through the interactive training data provided by the training environment, combined with the characteristics of multi-lane and multi-traffic light roads, to improve the economy, comfort and stability of hybrid vehicle driving strategies based on deep reinforcement learning methods.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] An economical driving method for a networked new energy vehicle in complex traffic scenarios, the method comprising:

[0008] S1. Construct an interactive multi-lane multi-traffic light simulation training scenario, in which the longitudinal motion of the vehicle is described by the vehicle kinematic model, the lane changing process of the vehicle is simplified to a transient process, and the traffic light consists of multiple phases; at the same time, a rule-based longitudinal acceleration decision model for other vehicles and a lane changing decision model for other vehicles are built, so that other vehicles can actively respond to changes in the traffic environment;

[0009] S2. Construct a decision model based on maximum entropy deep reinforcement learning, including state space, action space, and reward function, and set the structure of the strategy model and the evaluation model. The strategy model represents the mapping relationship between state and action, and the evaluation model is used to evaluate the action of the maximum entropy deep reinforcement learning model.

[0010] S3. Constructing a safety constraint model of the target vehicle, including the longitudinal motion safety constraint and lane-changing action safety constraint of the target vehicle;

[0011] S4. Train the decision model based on maximum entropy deep reinforcement learning.

[0012] Furthermore, in step S1, the vehicle kinematic model is expressed as:

[0013]

[0014] Where x and v represent the longitudinal position and velocity of the vehicle, respectively; x′ and v′ represent the first-order derivatives of the longitudinal position and velocity, respectively; and a represents acceleration.

[0015] In step S1, the other vehicle longitudinal acceleration decision models include:

[0016] 1) Output the maximum safe speed v based on the information of other vehicles in front and the traffic light ahead safe :

[0017]

[0018] In the formula, v l represents the speed of the preceding vehicle, b max Indicates the maximum deceleration of the vehicle, d gap Denotes the workshop distance, d tl Indicates the distance from the vehicle to the traffic light ahead;

[0019] 2) Output the vehicle acceleration a based on the maximum safe speed, the maximum road speed and the vehicle's maximum acceleration information:

[0020] v des =min(v max ,v+a max Δt,vsafe )

[0021] v'=min(0,v des )

[0022] a=v'-v

[0023] In the formula, v des represents the expected speed, v max represents the maximum speed limit of the road, v represents the current speed of the vehicle, and a max represents the maximum acceleration of the vehicle, Δt represents the time step, and v' represents the final target speed.

[0024] The other vehicle lane-changing decision models include:

[0025] 1) Obtain lane information and surrounding vehicle information, and select the lane set L that meets the lane change conditions through conditional judgment val :

[0026] L val = {l i ,…}

[0027]

[0028] In the formula, l i represents the lane number, δ i Indicates lane l i Whether there are lane change conditions, d max Indicates the maximum deceleration of the vehicle, v f Indicates lane l i Speed ​​of the following vehicle;

[0029] 2) Obtain available lane information L according to the destination tar , combined with lane L with lane-changing conditions val The final lane L that can be changed is obtained:

[0030] L=L tar ∩L val

[0031] 3) Based on the average speed of each lane Determine the final lane change action l tar ;

[0032]

[0033] Where S represents the number of each lane. i The corresponding average speed A collection of .

[0034] Further, in step S2, the state space is expressed as:

[0035] S=[S e ,S others ,S tl ,S flow ]

[0036] st

[0037] S e =[l e ,v e ,a e ]

[0038] S others =[d0,l0,v0,d1,l1,v1,...,d5,l5,v5]

[0039] S tl =[d tl ,t red ,t green ,t yellow ]

[0040]

[0041] In the formula, S e Indicates the target vehicle information, S others Indicates surrounding vehicle information, S tl Indicates the traffic light information ahead, S flow Indicates the traffic flow information ahead; l e 、v e 、a e Respectively represent the lane, speed, and acceleration of the target vehicle; d i , l i 、v i They represent the distance between the surrounding vehicles and the target vehicle, the lane where the surrounding vehicles are located, and the speed of the surrounding vehicles respectively; d tl Indicates the distance to the traffic light ahead, t red ,t green ,t yellow Respectively represent the remaining time of the red light, the remaining time of the green light, and the remaining time of the yellow light; ρ j Indicates the average traffic flow speed and traffic density of each lane;

[0042] The action space is expressed as:

[0043] A=[a,l]

[0044] sta∈[-4.5,2.6]m / s 2 ,l∈[0,2]

[0045] Where l represents the target lane;

[0046] The reward function is expressed as:

[0047] R=w v R v +w a R a +w a' R a' +w eco R eco +w lc R lc

[0048] In the formula, w v 、w a 、w a' 、w eco 、w lc Both represent weight coefficients, R v represents the traffic efficiency reward, R a , R a′ Respectively represent the vehicle's acceleration and acceleration change rate reward, R eco represents the operating cost, R lc represents the lane changing cost.

[0049] Furthermore, in step S3, the longitudinal motion safety constraint includes:

[0050] 1) Calculate the maximum safe speed v safe ,Right now:

[0051]

[0052] 2) Calculate the maximum safe acceleration a safe ,Right now:

[0053] a safe =v safe -v

[0054] 3) The acceleration a output by the learning-based decision model policy With the maximum safe acceleration a safe In comparison, the smaller value is taken as the final acceleration control amount a, that is:

[0055] a=min(a policy ,a safe ).

[0056] The lane changing action safety constraints include:

[0057] The target lane l output by the learning-based decision model policy Make a judgment:

[0058]

[0059] In the formula, δ policy Indicates lane l policy Whether there are lane-changing conditions; if δ policy If it is 1, the lane change is performed, otherwise it is not performed.

[0060] Further, step S4 includes:

[0061] S41. Initialize the decision model based on maximum entropy deep reinforcement learning, including the hyperparameters of the strategy model and the evaluation model;

[0062] S42. Add the target vehicle to the training environment and generate interactive training data under the joint action of safety constraints (s t ,a t ,r t ,s t+1 ), add the training data to the experience replay pool;

[0063] S43, extracting training data from the experience replay pool, and updating the two evaluation models of the evaluation function respectively by the gradient descent method, and the descent gradient is:

[0064]

[0065] In the formula, M represents the number of samples, |M| represents the size of the sample set, and s t 、a t 、r t They represent the state of the vehicle at time t, the action taken, and the reward obtained, respectively. i represents the i-th evaluation model, θ i Denotes the evaluation model Q i , y(·) represents the prediction of the evaluation model value, Q tar-j represents the jth objective function, π(·|s t ) represents the policy function, Represents the state s at the next moment t+1 The behavior at the next moment sampled according to the strategy function; α represents the temperature coefficient; γ represents the discount factor;

[0066] S44. Update the policy function using the gradient descent method, and the descent gradient is:

[0067]

[0068] In the formula, ψ represents the parameters of the policy function, is in state s t Next, the behavior sampled according to the policy function;

[0069] S45, update the temperature coefficient using the gradient descent method, the descent gradient is:

[0070]

[0071] In the formula, H0 represents the target entropy;

[0072] S46, respectively updating the two target evaluation models of the evaluation model:

[0073] θ tar,i =ρθ tar,i +(1-ρ)θ i ,for i=1,2

[0074] Where ρ represents the soft update coefficient, θ tar,i Denotes the objective function Q tar-i The parameter θ i Denotes the evaluation model Q i Parameters;

[0075] S47. Iteratively update the maximum entropy deep reinforcement learning model. After the maximum entropy deep reinforcement learning model converges, test the maximum entropy deep reinforcement learning model. If it does not meet expectations, optimize the hyperparameters and reward function of the maximum entropy reinforcement learning model, and return to step S41.

[0076] The beneficial effects of the present invention are:

[0077] 1) The present invention designs a multi-lane and multi-traffic light training environment with high randomness and interactivity, so that the training data is more in line with the characteristics of real traffic scenes, which is conducive to improving the decision-making performance of the reinforcement learning decision model in real traffic scenes.

[0078] 2) The present invention designs an economical driving strategy for hybrid vehicles in complex traffic scenarios based on maximum entropy deep reinforcement learning. It extracts key environmental information as input for complex traffic scenarios and takes acceleration and target lane as output. The resulting learning-based decision model is beneficial to improving the economy and comfort of hybrid vehicles during driving.

[0079] 3) The present invention designs a rule-based safety constraint that can ensure the safety of the vehicle during driving and avoid dangerous behaviors such as collisions and violations of traffic regulations.

[0080] 4) The present invention designs an effective hybrid vehicle reward function for complex traffic scenarios, which can ensure that the vehicle has certain efficiency, economy and comfort when driving.

[0081] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0083] Figure 1 A logical schematic diagram of the economical driving strategy of the present invention;

[0084] Figure 2 This is a schematic diagram of the structure of the reinforcement learning decision planning model;

[0085] Figure 3 This is a schematic diagram of the interactive multi-lane multi-traffic light training environment;

[0086] Figure 4 This is a schematic diagram of a multi-lane and multi-traffic light scenario;

[0087] Figure 5 Update the training process diagram for the reinforcement learning decision planning model;

[0088] Figure 6 The figure is a schematic diagram of the process of the method of the present invention. DETAILED DESCRIPTION

[0089] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0090] See also Figure 1 to Figure 6 The present invention provides a method for economical driving of networked new energy vehicles in complex traffic scenarios based on reinforcement learning. Considering the interactive behavior between vehicles in a real traffic environment, an interactive training environment is constructed to provide interactive training data. At the same time, considering the requirements of economical driving of autonomous driving vehicles for economy and driving efficiency, a decision-making method based on maximum entropy deep reinforcement learning with better stability, higher efficiency and higher sample utilization is proposed. Figure 1and Figure 6 As shown, the method specifically comprises the following steps:

[0091] Step S1: Construct Figure 4 The interactive multi-lane multi-traffic light training scenario shown is as follows:

[0092] S11: Figure 3 As shown in the figure, in the interactive multi-lane multi-traffic light training scenario, the longitudinal motion of the vehicle is described by the vehicle kinematic model, namely:

[0093]

[0094] Where x and v are the longitudinal position and velocity of the vehicle, respectively; x' and v' represent the first-order derivatives of the longitudinal position and velocity, respectively; and a is the acceleration;

[0095] S12: In the interactive multi-lane and multi-traffic light training scenario, the lane-changing process of the vehicle is simplified into a transient process;

[0096] S13: In the interactive multi-lane multi-traffic light training scenario, the traffic lights are composed of multiple phases, and different states of different lanes in different phases represent different traffic rights;

[0097] S14: Building a rule-based longitudinal acceleration decision model for other vehicles: In order to enable other vehicles to actively respond to changes in the traffic environment and make the traffic environment interactive, a rule-based longitudinal acceleration decision model is given to other vehicles, which specifically includes the following steps:

[0098] S141: The decision-making planning model of other vehicles outputs the maximum safe speed v according to the information of the preceding vehicle and the traffic light ahead. safe :

[0099]

[0100] In the formula, v l is the speed of the preceding vehicle, b max is the maximum deceleration of the vehicle, d gap is the workshop distance, d tl is the distance from the vehicle to the traffic light ahead. It should be noted that, in this embodiment, the length of all vehicles is 5m, and the acceleration range is [-4.5, 2.6] m / s 2 At the same time, when there is no vehicle ahead, no traffic light or the traffic light is green, the corresponding item is taken to obtain the maximum safe speed v safe ;

[0101] S142: Output acceleration control instruction a according to the maximum safe speed, the maximum road speed and the maximum vehicle acceleration information:

[0102] v des =min(v max ,v+a max Δt,v safe )

[0103] v'=min(0,v des )

[0104] a=v'-v

[0105] In the formula, v des is the expected speed, v max is the maximum speed limit of the road, v is the current speed of the vehicle, a max is the maximum acceleration of the vehicle, Δt is the time step, v safe is the maximum safe speed, v' is the final target speed;

[0106] S15: Building a rule-based lane-changing decision model for other vehicles: In order to enable other vehicles to actively respond to changes in the traffic environment and make the environment interactive, a rule-based lane-changing decision model is also given to other vehicles;

[0107] S151: Obtain lane information and surrounding vehicle information, and select a lane set L with lane change conditions through conditional judgment val :

[0108] L val = {l i ,…}

[0109]

[0110] In the formula, l i represents the lane number, δ i Indicates lane l i Whether there are lane change conditions, v l Indicates lane l i Speed ​​of the preceding vehicle, d max Indicates the maximum deceleration of the vehicle, d gap represents the workshop distance, v f Indicates lane l i Speed ​​of the following vehicle;

[0111] S152: Obtaining available lane information L according to the destination tar , combined with lane L with lane-changing conditions val The final lane L that can be changed is obtained:

[0112] L=L tar ∩Lval

[0113] S153: Based on the average speed of each lane Determine the final lane change action l tar ;

[0114]

[0115] Where S represents the number of each lane. i The corresponding average speed A collection of;

[0116] S16: Randomly initialize the initial position, initial speed and expected speed of other vehicles on the road, as well as the initial position, initial speed and phase cycle of the traffic light of the target vehicle to ensure the randomness of the traffic environment and avoid collisions and conflicts between vehicles.

[0117] Step S2: Construct Figure 2 The decision model based on maximum entropy deep reinforcement learning shown in Figure 1 specifically includes:

[0118] S21: Construct state space S: Construct state space based on key information of traffic environment, including speed, acceleration and lane of target vehicle, speed, lane and relative distance of other vehicles within a certain range to target vehicle, relative distance of traffic lights ahead and phase information of traffic lights, and traffic flow speed and density information of each lane ahead.

[0119] The state space S can be expressed as follows:

[0120] S=[S e ,S others ,S tl ,S flow ]

[0121] st

[0122] S e =[l e ,v e ,a e ]

[0123] S others =[d0,l0,v0,d1,l1,v1,...,d5,l5,v5]

[0124] S tl =[d tl ,t red ,t green ,t yellow ]

[0125]

[0126] In the formula, S e Indicates the target vehicle information, S others Indicates surrounding vehicle information, S tl Indicates the traffic light information ahead, S flow Represents the traffic flow information ahead; where l e 、v e 、a e Respectively represent the lane, speed, and acceleration of the target vehicle; d i , l i 、v i They represent the distance between the surrounding vehicles and the target vehicle, the lane where the surrounding vehicles are located, and the speed of the surrounding vehicles respectively; d tl ,t red ,t green ,t yellow They represent the distance to the traffic light ahead and the remaining duration of each state (i.e. the remaining time t red 、Green light lasts for the remaining time t green and the yellow light lasts for the remaining time t yellow ; ρ j Indicates the average traffic flow speed and traffic density of each lane;

[0127] S22: Determine action space A: The action space consists of acceleration and target lane. The acceleration obtained from the action space controls the longitudinal motion of the vehicle, and the target lane obtained from the action space and the lane where the target vehicle is currently located determine the lane change action to control the lateral motion of the vehicle. The action space A can be expressed as:

[0128] A=[a,l]

[0129] sta∈[-4.5,2.6]m / s 2 ,l∈[0,2]

[0130] Wherein, a is the output acceleration, l is the target lane, and in this embodiment, three lanes are used, which are numbered 0, 1, and 2 from right to left;

[0131] S23: Construct reward function R: The reward function is the traffic efficiency R v , comfort (including R a and R a' ), operating cost R eco and lane-changing cost R lc The weighted sum of the five indicators is:

[0132] R=w v R v +w a R a +wa' R a' +w eco R eco +w lc R lc

[0133] In the formula, w v 、w a 、w a' 、w eco 、w lc Represent different weight coefficients, traffic efficiency R v The target vehicle is required to be close to the expected speed when driving, and the comfort requirement is the acceleration R of the vehicle. a and the rate of change of acceleration R a Smaller, running cost R eco The equivalent fuel consumption of the vehicle is required to be small during driving, and the lane change cost R lc It is required to minimize unnecessary lane changing;

[0134] Among them, the traffic efficiency R v It is expressed by the following formula:

[0135] R v =(v max -v) 2

[0136] In the formula, v max represents the maximum speed limit of the road, and v represents the speed of the target vehicle;

[0137] Comfort (including R a and R a' ) is represented by the following formula:

[0138] R a =a 2

[0139] R a' =a' 2

[0140] In the formula, a represents the acceleration of the target vehicle, and a' represents the rate of change of acceleration;

[0141] Running cost eco It is expressed by the following formula:

[0142] R eco =b1·v+b2·v 2 +b3·P+b4·P 2 +b5

[0143] In the formula, b represents the fitting coefficient, v represents the target vehicle speed, and P represents the power required by the target vehicle. This function is the equivalent fuel consumption function of the hybrid vehicle, which can reflect the energy consumption to a certain extent.

[0144] Lane changing cost R lc It is expressed by the following formula:

[0145]

[0146] The above formula shows that if lane change is executed, R lc is 1, otherwise it is 0;

[0147] S24: Given the structure of the strategy model and the evaluation model: Use a neural network to fit the strategy model and the evaluation model. The strategy model represents the mapping relationship between state and action. The evaluation model uses the weighted sum of maximizing reward and strategy entropy as the evaluation criterion to evaluate the actions taken by the maximum entropy deep reinforcement learning model.

[0148] S25: The experience replay pool stores the experience data of the target vehicle interacting with the environment. During model training, a certain amount of data is randomly extracted from the experience replay pool, and new data of the target vehicle interacting with the environment is also stored in the experience replay pool.

[0149] Step S3: constructing the target vehicle safety constraints, specifically including the following steps:

[0150] S31: constructing the longitudinal motion safety constraint of the target vehicle, obtaining the upper limit of the acceleration of the target vehicle under different conditions through a rule-based method, comparing the acceleration output by the learning-based decision model with the upper limit of the acceleration, and taking the smaller value as the final acceleration control amount;

[0151] S311: Calculate the maximum safe speed v safe ,Right now:

[0152]

[0153] In the formula, v l is the speed of the preceding vehicle, b max is the maximum deceleration of the vehicle, d gap is the workshop distance, d tl is the distance from the target vehicle to the traffic light ahead. It should be noted that when there is no vehicle ahead, no traffic light ahead, or the traffic light is green, another item is used to obtain the maximum safe speed v safe ;

[0154] S312: Calculate the maximum safe acceleration a safe ,Right now:

[0155] a safe =vsafe -v

[0156] S313: The acceleration a output by the decision model based on learning policy With the maximum safe acceleration a safe In comparison, the smaller value is taken as the final acceleration control amount a, that is:

[0157] a=min(a policy ,a safe )

[0158] S32: constructing a safety constraint for the lane-changing action of the target vehicle, and judging the target lane output by the learning-based decision model through a rule-based method. If the target lane is within the safety constraint range of the lane-changing action, the lane is changed; otherwise, the lane is not changed;

[0159] S321: Target lane l output by the learning-based decision model policy Make a judgment:

[0160]

[0161] In the formula, δ policy Indicates lane l policy Whether there are lane change conditions, v l Indicates lane l policy Speed ​​of the preceding vehicle, d max Indicates the maximum deceleration of the vehicle, d gap represents the workshop distance, v f Indicates lane l policy Speed ​​of the following vehicle;

[0162] S322: Determine δ policy Is it 1? If yes, then change lanes; otherwise, do not change lanes;

[0163] Step S4: Figure 5 As shown, the training is based on the maximum entropy deep reinforcement learning decision model, which specifically includes the following steps:

[0164] S41: Initialize the decision model based on maximum entropy deep reinforcement learning, including the hyperparameters of the strategy model and the evaluation model;

[0165] S42: Add the target vehicle to the training environment and generate interactive training data under the joint action of safety constraints (s t ,a t ,r t ,s t+1 ), add the training data to the experience replay pool;

[0166] S43: Extract training data from the experience replay pool, and use the gradient descent method to update the two evaluation models of the evaluation function respectively. The descent gradient is:

[0167]

[0168] In the formula, M is the number of samples, |M| represents the size of the sample set, and s t 、a t 、r t They are the state of the vehicle at time t, the action taken, and the reward obtained, Q i is the i-th evaluation model, θ i is the evaluation model Q i The parameters of the evaluation model, y(·) is the prediction of the evaluation model value, Q tar-j is the jth objective function, π(·|s t ) represents the policy function, is the state s at the next moment t+1 , the behavior at the next moment sampled according to the policy function; α is the temperature coefficient; γ is the discount factor;

[0169] S44: Update the policy function using the gradient descent method, and the descent gradient is:

[0170]

[0171] Where ψ is the parameter of the policy function, is in state s t Next, the behavior sampled according to the policy function;

[0172] S45: Update the temperature coefficient using the gradient descent method, and the descent gradient is:

[0173]

[0174] In the formula, α is the temperature coefficient, H0 is the target entropy;

[0175] S46: Update two target evaluation models of the evaluation model respectively:

[0176] θ tar,i =ρθ tar,i +(1-ρ)θ i ,for i=1,2

[0177] Where ρ is the soft update coefficient, θ tar,i is the objective function Q tar-i The parameter θ i is the evaluation model Q i Parameters;

[0178] S47: Iteratively update the maximum entropy deep reinforcement learning model: After the maximum entropy deep reinforcement learning model converges, test the maximum entropy deep reinforcement learning model. If it does not meet expectations, optimize the hyperparameters and reward function of the maximum entropy reinforcement learning model, and return to step S41. The hyperparameters of the final model are shown in Table 1.

[0179] Table 1 Model hyperparameter values

[0180]

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. An economical driving method for a networked new energy vehicle in complex traffic scenarios, characterized by: The method comprises the following steps: S1. Construct an interactive multi-lane multi-traffic light simulation training scenario, in which the longitudinal motion of the vehicle is described by the vehicle kinematic model, the lane changing process of the vehicle is simplified to a transient process, and the traffic light consists of multiple phases; at the same time, a rule-based longitudinal acceleration decision model for other vehicles and a lane changing decision model for other vehicles are built, so that other vehicles can actively respond to changes in the traffic environment; S2. Construct a decision model based on maximum entropy deep reinforcement learning, including state space, action space, and reward function, and set the structure of the strategy model and the evaluation model. The strategy model represents the mapping relationship between state and action, and the evaluation model is used to evaluate the action of the maximum entropy deep reinforcement learning model. The state space is represented as: S=[S e ,S others ,S tl ,S flow ] st S e =[l e ,v e ,a e ] S others =[d0,l0,v0,d1,l1,v1,...,d5,l5,v5] S tl =[d tl ,t red ,t green ,t yellow ] In the formula, S e Indicates the target vehicle information, S others Indicates surrounding vehicle information, S tl Indicates the traffic light information ahead, S flow Indicates the traffic flow information ahead; l e 、v e 、a e Respectively represent the lane, speed, and acceleration of the target vehicle; d i , l i 、v i They represent the distance between the surrounding vehicles and the target vehicle, the lane where the surrounding vehicles are located, and the speed of the surrounding vehicles respectively; d tl Indicates the distance to the traffic light ahead, t red ,t green ,t yellow Respectively represent the remaining time of the red light, the remaining time of the green light, and the remaining time of the yellow light; ρ j Indicates the average traffic flow speed and traffic density of each lane; The action space is expressed as: A=[a,l] s.t.a∈[-4.5,2.6]m / s 2 ,l∈[0,2] Where l represents the target lane; The reward function is expressed as: R=w v R v +w a R a +w a′ R a′ +w eco R eco +w lc R lc In the formula, w v 、w a 、w a′ 、w eco 、w lc Both represent weight coefficients, R v represents the traffic efficiency reward, R a , R a′ Respectively represent the vehicle's acceleration and acceleration change rate reward, R eco represents the operating cost, R lc represents the lane-changing cost; S3. Constructing a safety constraint model of the target vehicle, including the longitudinal motion safety constraint and lane-changing action safety constraint of the target vehicle; S4. Train the decision model based on maximum entropy deep reinforcement learning.

2. The economical driving method of a networked new energy vehicle according to claim 1, characterized in that: In step S1, the vehicle kinematic model is expressed as: Where x and v represent the longitudinal position and velocity of the vehicle, respectively; x′ and v′ represent the first-order derivatives of the longitudinal position and velocity, respectively; and a represents acceleration.

3. The economical driving method of a networked new energy vehicle according to claim 1, characterized in that: In step S1, the other vehicle longitudinal acceleration decision models include: 1) Output the maximum safe speed v based on the information of other vehicles in front and the traffic light ahead safe : In the formula, v l represents the speed of the preceding vehicle, b max Indicates the maximum deceleration of the vehicle, d gap Denotes the workshop distance, d tl Indicates the distance from the vehicle to the traffic light ahead; 2) Output the vehicle acceleration a based on the maximum safe speed, the maximum road speed and the vehicle's maximum acceleration information: v des =min(v max ,v+a max Δt,v safe ) v'=min(0,v des ) a=v'-v In the formula, v des represents the expected speed, v max represents the maximum speed limit of the road, v represents the current speed of the vehicle, and a max represents the maximum acceleration of the vehicle, Δt represents the time step, and v' represents the final target speed.

4. The economical driving method of a networked new energy vehicle according to claim 1, characterized in that: In step S1, the other vehicle lane change decision model includes: 1) Obtain lane information and surrounding vehicle information, and select the lane set L that meets the lane change conditions through conditional judgment val : L val ＝{l i ,…} In the formula, l i represents the lane number, δ i Indicates lane l i Whether there are lane change conditions, d max Indicates the maximum deceleration of the vehicle, v f Indicates lane l i Speed ​​of the following vehicle; v self Indicates the current speed of the vehicle; 2) Obtain available lane information L according to the destination tar , combined with lane L with lane-changing conditions val The final lane L that can be changed is obtained: L=L tar ∩L val 3) Based on the average speed of each lane Determine the final lane change action l tar ; Where S represents the number of each lane. i The corresponding average speed A collection of .

5. The economical driving method of a networked new energy vehicle according to claim 1, characterized in that: In step S3, the longitudinal motion safety constraint includes: 1) Calculate the maximum safe speed v safe ,Right now: 2) Calculate the maximum safe acceleration a safe ,Right now: and safe =in safe -v 3) The acceleration a output by the learning-based decision model policy With the maximum safe acceleration a safe In comparison, the smaller value is taken as the final acceleration control amount a, that is: a = min(a) policy ,in safe )。 6. The economical driving method of a networked new energy vehicle according to claim 1, characterized in that: In step S3, the lane changing action safety constraints include: The target lane l output by the learning-based decision model policy Make a judgment: In the formula, δ policy Indicates lane l policy Whether there are lane-changing conditions; if δ policy If v is 1, the lane change is performed, otherwise the lane change is not performed; self Indicates the current speed of the vehicle.

7. The economical driving method of a networked new energy vehicle according to claim 1, characterized in that: Step S4 includes: S41. Initialize the decision model based on maximum entropy deep reinforcement learning, including the hyperparameters of the strategy model and the evaluation model; S42. Add the target vehicle to the training environment and generate interactive training data under the joint action of safety constraints (s t ,a t ,r t ,s t+1 ), add the training data to the experience replay pool; S43, extracting training data from the experience replay pool, and updating the two evaluation models of the evaluation function respectively by the gradient descent method, and the descent gradient is: In the formula, M represents the number of samples, |M| represents the size of the sample set, and s t 、a t 、r t They represent the state of the vehicle at time t, the action taken, and the reward obtained, respectively. i represents the i-th evaluation model, θ i Denotes the evaluation model Q i , y(·) represents the prediction of the evaluation model value, Q tar-j represents the jth objective function, π(·|s t ) represents the policy function, Represents the state s at the next moment t+1 The behavior at the next moment sampled according to the strategy function; α represents the temperature coefficient; γ represents the discount factor; S44. Update the policy function using the gradient descent method, and the descent gradient is: In the formula, ψ represents the parameters of the policy function, is in state s t Next, the behavior sampled according to the policy function; S45, update the temperature coefficient using the gradient descent method, the descent gradient is: In the formula, H0 represents the target entropy; S46, respectively updating the two target evaluation models of the evaluation model: i tar,i =ρθ tar,i +(1-p)θ i ,for i=1,2 Where ρ represents the soft update coefficient, θ tar,i Denotes the objective function Q tar-i The parameter θ i Denotes the evaluation model Q i Parameters; S47. Iteratively update the maximum entropy deep reinforcement learning model. After the maximum entropy deep reinforcement learning model converges, test the maximum entropy deep reinforcement learning model. If it does not meet expectations, optimize the hyperparameters and reward function of the maximum entropy reinforcement learning model, and return to step S41.

Images