Hierarchical control method for hybrid electric vehicle platooning energy-saving driving based on deep reinforcement learning

Abstract

This invention relates to a hierarchical control method for hybrid electric vehicle (HEV) platooning energy-saving driving based on deep reinforcement learning, belonging to the field of new energy vehicle technology. The method comprises the following steps: S1: constructing a vehicle platoon motion model and establishing platoon motion constraints based on the vehicle longitudinal dynamics model; S2: acquiring vehicle operating data using vehicle-to-vehicle (V2V) communication technology; S3: constructing an upper-layer collaborative adaptive cruise control model based on a dual-delay deep deterministic policy gradient algorithm to provide vehicle speed data to a lower-layer energy management model; S4: constructing a lower-layer energy management model based on a deep Q-network that considers cabin temperature control; and S5: training the lower-layer energy management model. This method can effectively improve the safety and energy efficiency of HEV platooning, while also enhancing cabin thermal comfort.

Description

Technical Field

[0001] The present invention belongs to the technical field of new energy vehicles and relates to a hierarchical control method for energy-saving driving of a hybrid vehicle platoon based on deep reinforcement learning. Background Art

[0002] With the rapid development and gradual implementation of intelligent connected transportation systems, vehicle connectivity and automation have become a research hotspot in the new energy vehicle sector. Platooning can effectively reduce following distances and significantly reduce the aerodynamic drag coefficient of following vehicles, offering significant energy-saving potential. In a platoon composed of connected vehicles, not only can vehicles share information but also set common goals for unified planning, significantly improving vehicle energy consumption, ensuring platoon stability, and minimizing collisions.

[0003] Current vehicle platoon control technologies have achieved a series of results in improving platoon stability and safety. However, existing research has largely neglected the energy consumption of the cabin temperature control system, which, to a certain extent, affects the energy performance of the vehicle platoon. Cabin temperature control energy consumption is related to the power output of the powertrain, and the powertrain's operating mode is related to vehicle speed, which in turn determines the power demand of the powertrain. This means that cabin temperature control, speed planning, and powertrain energy management are three interrelated sub-problems. Traditional vehicle platoon control methods mostly rely on pre-defined rules and models. These methods suffer from poor optimality and real-time performance when dealing with multiple coupled optimization objectives. In recent years, reinforcement learning (RL) technology has garnered widespread attention in autonomous driving decision-making. RL continuously learns optimal policies through interaction with the environment, enabling more flexible and efficient decision-making in complex tasks. However, single-agent RL approaches struggle to cope with complex, multi-objective collaborative optimization tasks, and the resulting decisions lack generalizability. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a hierarchical control method for energy-saving driving in hybrid electric vehicle platoons based on deep reinforcement learning. This method plans vehicle driving speed using a dual-delay deep deterministic policy gradient algorithm model, and implements energy management of the vehicle power system and cabin temperature control system based on a deep Q-network algorithm model. This ensures the safety of vehicle platooning, reduces vehicle energy consumption, and improves cabin thermal comfort.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning comprises the following steps:

[0007] S1: Construct a vehicle platoon motion model based on the vehicle longitudinal dynamics model and establish vehicle platoon motion constraints;

[0008] S2: Use V2V communication technology to obtain vehicle operation data;

[0009] S3: Build an upper-level collaborative adaptive cruise control model based on a double-delayed deep deterministic policy gradient algorithm to provide vehicle speed data for the lower-level energy management model;

[0010] S4: Construct a lower-level energy management model based on a deep Q network that considers cabin temperature control;

[0011] S5: Training the lower-layer energy management model.

[0012] Furthermore, the vehicle longitudinal dynamics model is as follows:

[0013]

[0014] in, is the vehicle i and its position, v i ,a i ,F a,i are the speed, driving force and acceleration of vehicle i, respectively, m i ,δ i are the mass and rotational mass conversion coefficient of vehicle i, respectively.

[0015] Furthermore, the vehicle platoon motion constraints include spacing constraints, speed constraints, acceleration constraints, string stability constraints, and headway constraints:

[0016] Δs min,i ≤Δs i

[0017] 0≤v i ≤v max,i

[0018] a min,i ≤a i ≤a max,i

[0019]

[0020] Where Δs i is the distance between vehicle i and the preceding vehicle; Δs min,i is the minimum distance between vehicle i and the preceding vehicle; v max,i is the maximum speed of vehicle i; a min,i ,a max,i are the minimum acceleration and maximum acceleration of vehicle i respectively; l i Indicates the string stability of vehicle i, and sets the l between the leading vehicle and its following vehiclei Similarly, the l of the rear vehicle and the vehicle in front i Same; d i ,d i-1 ,d i+1 are the travel distances per unit time of vehicle i, preceding vehicle i-1, and following vehicle i+1 respectively; h i is the headway; h min is the limit headway; s i ,s i+1 ,l v ,v i are the distance traveled between vehicle i and the following vehicle i+1, the vehicle length, and the speed of vehicle i.

[0021] Furthermore, the step S2 of obtaining the vehicle's operating data by using the V2V communication technology includes:

[0022] The leading vehicle in the platoon uses V2V technology to obtain the operating data of the pilot vehicle, including: the pilot vehicle's speed, acceleration, and distance traveled per unit time;

[0023] Vehicles other than the lead vehicle in a vehicle queue use V2V technology to obtain the operating data of the leading and trailing vehicles, including their speeds and travel distances per unit time. The tail vehicle in the vehicle queue obtains the operating data of the leading vehicle.

[0024] Furthermore, step S3 specifically includes the following steps:

[0025] S31: Create a vehicle platoon operating environment model: Construct a vehicle platoon consisting of a lead vehicle and multiple follower vehicles. Each follower vehicle in the platoon is a control vehicle, and the sequence number is set to 1, 2, ..., n. The trajectory of the lead vehicle is set automatically, and the sequence number of the lead vehicle is set to 0. Set the initial distance between the lead vehicle and the lead vehicle, the initial distance of the platoon, and the initial speed of the platoon.

[0026] S32: Based on the motion simulation model of the vehicle platoon and the vehicle operating status, construct the state space S1 of vehicle 1:

[0027] S1={v0,a0,d i}

[0028] Among them, v0 and a0 are the velocity and acceleration of the pilot car respectively;

[0029] Build the state space S of the remaining vehicles 2,...,n :

[0030] S 2,...,n ={v i-1 ,v i ,d i ,h i ,li}

[0031] Among them, v i-1 is the speed of the preceding vehicle;

[0032] S33: Building Action Space A i , its action variables are extracted from the acceleration calculation formula, as follows:

[0033] a i =-k x,i (s i -s i-1 +l v )-k v,i (v i -v i-1 ),

[0034] A i ={k v,i |0≤k v,i ≤10}

[0035] Among them, k x,i is the position error coefficient between vehicle i and the preceding vehicle, set as a constant; k v,i is the speed error coefficient between vehicle i and the preceding vehicle;

[0036] S34: Construct reward function R i , reward function R i Including the vehicle layer reward function R v,i and the dynamical system layer reward function R p,i Two parts, namely:

[0037] R i =-R v,i -R p,i

[0038] The vehicle layer reward function R v,i is the acceleration index r Acc,i , acceleration index r J,i , headway index r h,i and string stability index r l,i The weighted sum of , that is:

[0039] R v,i =α 1,i r Acc,i +α 2,i r J,i +α 3,i r h,i +α 4,i r l,i

[0040] Among them, α 1,i ,α 2,i,α 3,i ,α 4,i They are acceleration index r Acc,i , acceleration index r J,i , headway index r h,i and string stability index r l,i The weight of the acceleration index r Acc,i Vehicle i is required to meet the acceleration constraint condition; the acceleration index r J,i The vehicle i is required to have a minimum jerk to ensure comfort; the headway index r h,i It is required that vehicle i maintain a certain safe distance from the preceding vehicle, and vehicle i has sufficient reaction time when the preceding vehicle brakes; the string stability index r l,i It is required that the distance error between vehicle i and the preceding and following vehicles in the platoon will not propagate along the platoon and be amplified, and the distance errors between all vehicles in the platoon should maintain the same accuracy to avoid collisions.

[0041] The dynamic system layer reward function R p,i is the fuel consumption index r f,i , battery SOC index r SOC,i and battery SOH index r SOH,i The weighted sum of , that is:

[0042] R p,i =β 1,i r f,i +β 2,i r SOC,i +β 3,i r SOH,i

[0043] Among them, β 1,i ,β 2,i ,β 3,i They are fuel consumption index r f,i , battery SOC index r SOC,i and battery SOH index r SOH,i The weight of the fuel consumption index r f,i The fuel consumption rate of vehicle i is required to be as low as possible as possible below the maximum fuel consumption rate; wherein the battery SOC index r SOC,i The battery SOC of vehicle i is required to be kept within a certain range; wherein the battery SOH index r SOH,i The capacity loss of the battery of vehicle i is required to be as small as possible;

[0044] S35: Construct the network structure of the dual-delayed deep deterministic policy gradient algorithm: the policy network Actor determines the action taken in the current state, and the two evaluation networks Critic1 and Critic2 are used to evaluate the current state and action; the target policy network Target-Actor and the target evaluation networks Target-Critic1 and Target-Critic2 have the same network structure as the corresponding main network to improve learning stability;

[0045] S36: The experience replay pool of the double-delayed deep deterministic policy gradient algorithm stores the experience data of the interaction between the intelligent agent and the vehicle queue operation environment model. During model training, a certain amount of data is randomly extracted from the experience replay pool, and new data from the continuous interaction between the intelligent agent and the vehicle queue operation environment model is also stored in the experience replay pool.

[0046] Further, step S4 specifically includes the following steps:

[0047] S41: Create an energy management environment model: Build a powertrain model and cabin thermal management model for each vehicle in the platoon; set initial values ​​for the powertrain operating mode, battery SOC, battery SOH, and cabin temperature;

[0048] S42: Build the state space of vehicle i

[0049]

[0050] Among them, T cab,i ,e cab,i ,Δe cab,i are the cabin temperature of vehicle i, the error between the actual cabin temperature and the set temperature, and the error change rate; SOC i ,P d,i ,P cab,i are the battery SOC value, power system demand power and cabin temperature control system power of vehicle i respectively; v fi is the speed of vehicle i at the next moment; a i is the acceleration of vehicle i, provided by the upper-layer cooperative adaptive cruise control model; Δt is the time step;

[0051] S43: Building an Action Space It consists of two parts: the control instructions of the cabin temperature control system and the operating mode of the power system, as follows:

[0052]

[0053] Among them, a h,i ,a AC,i are the opening degrees of the heating system and air conditioning system of vehicle i, respectively, which are discrete variables, and am,i is the operating mode number of the vehicle i power system; the vehicle has five operating modes, namely driving mode (-1): the engine is off, but the transmission keeps the engine idling; parking mode (0): the engine is off, the speed is zero, and the torque is zero; starting mode (1): the transmission starts the engine to idle speed, and no fuel is consumed; idle mode (2): the engine is on but idling, and no torque is generated; starting mode (3): the engine is on, and fuel is supplied to the engine;

[0054] S44: Constructing the reward function Reward Function Including cabin temperature reward function R cab,i and the power system energy consumption reward function R EMS,i Two parts, namely:

[0055]

[0056] The cabin temperature reward function R cab,i is the temperature error e cab,i The absolute value of the temperature error change rate Δe cab,i The absolute value of the temperature control system power P cab,i and the temperature penalty term l cab,i The weighted sum of , that is:

[0057]

[0058] in, are the temperature errors e cab,i The absolute value of the temperature error change rate Δe cab,i The absolute value of the temperature control system power P cab,i and the temperature penalty term l cab,i The weight of the temperature penalty term l cab,i The actual cabin temperature of vehicle i must not exceed the set upper and lower temperature limits;

[0059] The power system energy consumption reward function R EMS,i Is the battery SOC indicator c SOC,i , fuel consumption index c f,i and battery SOH index c SOH,i and battery SOC penalty term l e,i The weighted sum of , that is:

[0060]

[0061] in, They are the battery SOC index c SOC,i , fuel consumption index c f,i and battery SOH index cSOH,i and battery SOC penalty term l e,i The weight of the battery SOC index c SOC,i The battery SOC change of vehicle i is required to be as small as possible when the battery is discharged, and the battery is required to be as small as possible when the battery is charged. SOC,i =0; wherein the fuel consumption index c f,i is the fuel consumption rate of vehicle i; wherein the battery SOH index c SOH,i The capacity loss of the battery of vehicle i is required to be as small as possible; the battery SOC penalty term l e,i The battery SOC of vehicle i is required not to exceed the set upper and lower limits of the battery SOC;

[0062] S45: Construct a rule-based power allocation model: Using the working mode output by the deep reinforcement learning agent as input, the set power allocation rules are used to complete the power distribution of the power system under the selected working mode; no power allocation rules are set for parking mode (0) and starting mode (1);

[0063] S46: Construct the network structure of the deep Q network algorithm to generate the Q values ​​of all possible actions in the current state, and the target network to generate the target Q value;

[0064] S47: The experience replay pool of the deep Q network algorithm stores the experience data of the interaction between the intelligent agent and the energy management environment model; during model training, a certain amount of data is randomly extracted from the experience replay pool, and new data of the continuous interaction between the intelligent agent and the energy management environment model is also stored in the experience replay pool.

[0065] Furthermore, the training of the energy-saving driving hierarchical control model based on deep reinforcement learning in step S5 specifically includes the following steps:

[0066] S51: Initialize the upper-layer collaborative adaptive cruise control model based on the double-delayed deep deterministic policy gradient algorithm and the lower-layer energy management model based on the deep Q network considering cabin temperature control, including the hyperparameters of each neural network and the experience replay pool;

[0067] S52: Upper-layer collaborative adaptive cruise control model based on double-delay deep deterministic policy gradient algorithm: Connect the environment model and the algorithm model, and generate interactive data during training (S i,t ,A i,t ,R i,t ,S i,t+1 ) is added to the experience replay pool D1;

[0068] At each time step, the gradient descent algorithm is used to minimize the loss function L to update the evaluation networks Critic1 and Critic2:

[0069] L=N-1 ∑(yQ θi (S i,t ,A i,t )) 2

[0070]

[0071] Where y is the target value; are the evaluation value function and the target evaluation value function, θ i ,θ i ′ are the parameters of the evaluation network and the target evaluation network respectively; S i,t ,A i,t ,R i,t S is the vehicle’s current state, actions taken, and rewards obtained; i,t+1 is the state of the vehicle at the next moment; γ is the discount factor; π φ′ is the target policy function;∈ is the action noise; State S i,t+1 The sum of the actions taken according to the target policy function and the action noise ∈;

[0072] After the evaluation networks Critic1 and Critic2 are updated for d steps, the policy network Actor is updated using deterministic policy gradients:

[0073]

[0074] Among them, φ is the parameter of the policy network;

[0075] Update the two target evaluation networks and target policy networks separately:

[0076] θ i ′=τθ i +(1-τ)θ i ′,for i=1,2

[0077] φ′=τφ+(1-τ)φ′

[0078] Among them, τ is the soft update coefficient, φ′ is the parameter of the target policy network;

[0079] S53: Lower-level energy management model considering cabin temperature control based on deep Q network: connecting the environmental model and the algorithm model, and generating interactive data during the training process Added to experience replay pool D2;

[0080] At each time step, the loss function L is minimized using the gradient descent method E To update the evaluation network:

[0081]

[0082] in, are the evaluation value function and the target evaluation value function, θ E,i ,θ E,i ′ are the parameters of the evaluation network and the target network respectively; The vehicle's current state, actions taken, rewards obtained, and state at the next moment; Status possible actions to take;

[0083] The target network parameters are updated by hard updating, and the network parameters θ are evaluated every C steps. E,i Copy to the target network;

[0084] S54: Iteratively update the energy-saving driving hierarchical control model based on deep reinforcement learning: After the energy-saving driving hierarchical control model based on deep reinforcement learning converges, test the model. If it does not meet expectations, optimize the model's hyperparameters and reward function, and return to step S51.

[0085] The beneficial effects of the present invention are:

[0086] 1) This paper designs an adaptive cruise control method for vehicle platoons. By incorporating a string stability indicator and a headway indicator into the reward function, the method safely reduces the distance between vehicles, lowers aerodynamic drag, and thus reduces energy consumption.

[0087] 2) The present invention designs an energy management strategy that takes cabin temperature control into consideration, which synergistically optimizes the energy consumption of the cabin temperature control system and the power system, ensures cabin thermal comfort, and improves the economy of the entire vehicle.

[0088] 3) This paper designs a hierarchical control method for energy-saving driving of hybrid electric vehicle platoons. Based on a multi-agent deep reinforcement learning framework, it achieves the coordinated optimization of vehicle speed, cabin temperature, and vehicle energy consumption, effectively improving the safety, comfort, and economy of vehicle platoon operation.

[0089] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0091] Figure 1This is a logical structure diagram of the hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning according to the present invention;

[0092] Figure 2 Schematic diagram of the deep reinforcement learning vehicle platoon operating environment model;

[0093] Figure 3 Schematic diagram of the energy management environment model for deep reinforcement learning;

[0094] Figure 4 Updated diagram of the training process for a multi-agent deep reinforcement learning model;

[0095] Figure 5 This is the overall flow chart of the hierarchical control method for hybrid electric vehicle platoon energy-saving driving based on deep reinforcement learning. DETAILED DESCRIPTION

[0096] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0097] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0098] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0099] like Figures 1 to 5As shown, the present invention provides a hierarchical control method for energy-saving driving in hybrid electric vehicle platoons based on deep reinforcement learning. Considering the issue of information sharing between vehicles in real-world platoons, V2V communication technology is used to acquire vehicle operating data. Furthermore, considering the coupling relationship between vehicle speed planning, powertrain energy distribution, and cabin temperature control during platooning, a hierarchical control method for energy-saving driving based on deep reinforcement learning is proposed, which offers enhanced safety, economy, and generalizability. The method specifically includes the following steps:

[0100] Step S1: construct a vehicle platoon motion model based on the vehicle longitudinal dynamics model and establish vehicle platoon motion constraints. This specifically includes the following steps:

[0101] S11: The vehicle adopts the following longitudinal dynamics model, as follows:

[0102]

[0103] S12: To ensure the comfort and safety of vehicles during driving, the vehicle platoon motion constraints including spacing constraints, speed constraints, acceleration constraints, string stability constraints and headway constraints are constructed, namely:

[0104] Δs min,i ≤Δs i

[0105] 0≤v i ≤v max,i

[0106] a min,i ≤a i ≤a max,i

[0107]

[0108] Where Δs i is the distance between vehicle i and the preceding vehicle; Δs min,i is the minimum distance between vehicle i and the preceding vehicle; v max,i is the maximum speed of vehicle i; a min,i ,a max,i are the minimum acceleration and maximum acceleration of vehicle i respectively; l i represents the string stability of vehicle i, specifically, the l of the first and last vehicles in the vehicle queue i It is impossible to calculate according to the above formula. To simplify the constraints, set the l between the leading vehicle and the following vehicle i Similarly, the l of the rear vehicle and the vehicle in front i Same; d i ,d i-1 ,d i+1are the travel distances per unit time of vehicle i, preceding vehicle i-1, and following vehicle i+1 respectively; h i is the headway; h min is the limit headway; s i ,s i+1 ,l v ,v i are the distance traveled between vehicle i and the following vehicle i+1, the vehicle length, and the speed of vehicle i.

[0109] Step S2, using V2V communication technology to obtain vehicle operation data, specifically includes the following steps:

[0110] S21: The leading vehicle in the platoon uses V2V technology to obtain the operating data of the pilot vehicle, including the pilot vehicle's speed, acceleration, and distance traveled per unit time.

[0111] S22: The other vehicles in the vehicle queue, except the head vehicle, use V2V technology to obtain the operating data of the leading and following vehicles, including: the speed of the leading and following vehicles and the distance traveled per unit time; the tail vehicle in the vehicle queue only needs to obtain the operating data of the leading vehicle.

[0112] Step S3, constructing a collaborative adaptive cruise control model based on a dual-delay deep deterministic policy gradient algorithm, specifically including the following steps:

[0113] S31: Create a vehicle platoon operation environment model: In this example, a simulation environment is constructed that includes a lead vehicle and a platoon of three following vehicles. Figure 2 As shown in the figure, each following vehicle in the vehicle platoon is a control vehicle, and its serial number is set as 1, 2, and 3. The speed sequence of the pilot vehicle is set, and the pilot vehicle serial number is set to 0. The initial distance between the lead vehicle and the pilot vehicle, the initial distance of the vehicle platoon, and the initial speed of the vehicle platoon are set. It is necessary to ensure that the vehicle distance and the initial speed of the vehicle platoon meet the vehicle platoon operation constraints to avoid vehicle collisions.

[0114] S32: Based on the motion simulation model of the vehicle platoon and the vehicle operating status, the state space S is constructed. During the operation of the vehicle platoon, the main task of the lead vehicle is to follow the pilot vehicle. It is necessary to focus on the operating data of the pilot vehicle. Therefore, the state space S1 of vehicle 1 is constructed:

[0115] S1={v0,a0,d i}

[0116] Among them, v0 and a0 are the velocity and acceleration of the pilot car respectively;

[0117] For other vehicles in the vehicle queue except the head vehicle, they need to follow the front vehicle under the premise of ensuring the queue string stability index and the headway index. Therefore, the state space S of vehicles 2 and 3 is constructed. 2,3 :

[0118] S 2,3 ={v i-1 ,v i ,d i ,h i ,l i}

[0119] Among them, v i-1 is the speed of the preceding vehicle;

[0120] S33: Determine the action space A i This method achieves safe and energy-efficient following by controlling the vehicle's acceleration. Therefore, its action variables can be extracted from the acceleration calculation formula. The acceleration formula contains two parts: the position error and the velocity error of the leading and trailing vehicles. To simplify the action space, the coefficient of either part can be selected as the action variable, while the coefficient of the other part is set to a fixed value. Specifically, as follows:

[0121] a i =-k x,i (s i -s i-1 +l v )-k v,i (v i -v i-1 ),

[0122] A i ={k v,i |0≤k v,i ≤10}

[0123] Among them, k x,i is the position error coefficient between vehicle i and the preceding vehicle, constant, k v,i is the speed error coefficient between vehicle i and the preceding vehicle;

[0124] S34: Construct reward function R i , reward function R i Including the vehicle layer reward function R v,i and the dynamical system layer reward function R p,i Two parts, namely:

[0125] R i =-R v,i -R p,i

[0126] The vehicle layer reward function R v,i is the acceleration index r Acc,i , acceleration index rJ,i , headway index r h,i and string stability index r l,i The weighted sum of , that is:

[0127] R v,i =α 1,i r Acc,i +α 2,i r J,i +α 3,i r h,i +α 4,i r l,i

[0128] Among them, α 1,i ,α 2,i ,α 3,i ,α 4,i They are acceleration index r Acc,i , acceleration index r J,i , headway index r h,i and string stability index r l,i The weight of the acceleration index r Acc,i Vehicle i is required to meet the acceleration constraint condition; the acceleration index r J,i The vehicle i is required to have a minimum jerk as possible to ensure comfort; the headway index r h,i It is required that vehicle i maintain a certain safe distance from the preceding vehicle, and vehicle i has sufficient reaction time when the preceding vehicle brakes; the string stability index r l,i It is required that the distance error between vehicle i and the preceding and following vehicles in the queue will not propagate along the queue and be amplified, and the distance errors between all vehicles in the queue should maintain the same accuracy to avoid collisions.

[0129] The dynamic system layer reward function R p,i is the fuel consumption index r f,i , battery SOC index r SOC,i and battery SOH index r SOH,i The weighted sum of , that is:

[0130] R p,i =β 1,i r f,i +β 2,i r SOC,i +β 3,i r SOH,i

[0131] Among them, β 1,i ,β 2,i ,β 3,i They are fuel consumption index r f,i , battery SOC index r SOC,i and battery SOH index rSOH,i The weight of the fuel consumption index r f,i The fuel consumption rate of vehicle i is required to be as low as possible as possible below the maximum fuel consumption rate; wherein the battery SOC index r SOC,i The battery SOC of vehicle i is required to be kept within a certain range; wherein the battery SOH index r SOH,i The capacity loss of the battery of vehicle i is required to be as small as possible;

[0132] S341: In this embodiment, the acceleration index r Acc,i The longitudinal acceleration of the vehicle is required to be kept small, and its expression is:

[0133]

[0134] Among them, a max,i Set to 1m / s 2 ;

[0135] S342: In this embodiment, to ensure comfort, the acceleration index r J,i The vehicle's jerk is required to be kept small, and its expression is:

[0136]

[0137] Among them, J com,i To meet the maximum jerk required for comfort, set it to 0.5 m / s 3 ; J com,i is the maximum acceleration of the vehicle, set to 4m / s 3 ;

[0138] S343: In this embodiment, the headway index r h,i When the leading vehicle brakes, the vehicle must have sufficient reaction time and maintain a minimum distance between vehicles. The expression is:

[0139]

[0140] Among them, h max,i is the maximum headway;

[0141] S344: In this embodiment, the string stability index r l,i It is required that the distance error between vehicles in a vehicle queue will not propagate along the queue and be amplified, and the distance error between all vehicles in the platoon maintains the same accuracy to avoid collisions. The expression is:

[0142]

[0143] S345: In this embodiment, the fuel consumption index r f,i It can be expressed as:

[0144]

[0145] in, is the fuel consumption rate of vehicle i at the current moment; is the maximum fuel consumption rate of vehicle i;

[0146] S346: In this embodiment, the battery SOC indicator r SOC,i The battery SOC of vehicle i is required to be kept within a certain range, which is expressed as:

[0147]

[0148] S347: In this embodiment, the battery SOH indicator r SOH,i The capacity loss of the battery of vehicle i is required to be as small as possible, and its expression is:

[0149]

[0150] S35: Constructing the network structure of the dual-delay deep deterministic policy gradient algorithm: The dual-delay deep deterministic policy gradient algorithm is a deterministic deep reinforcement learning algorithm under the Actor-Critic (AC) framework. The policy network Actor determines the action taken in the current state, and the two evaluation networks Critic1 and Critic2 are used to evaluate the current state and action. The network structure of the target policy network Target-Actor and the target evaluation networks Target-Critic1 and Target-Critic2 are the same as the corresponding main network structure, which is used to improve learning stability.

[0151] S36: The experience replay pool of the double-delayed deep deterministic policy gradient algorithm stores the experience data of the interaction between the intelligent agent and the vehicle queue operation environment model. During model training, a certain amount of data is randomly extracted from the experience replay pool. At the same time, new data from the continuous interaction between the intelligent agent and the vehicle queue operation environment model is also stored in the experience replay pool.

[0152] Step S4: constructing a lower-layer energy management model based on a deep Q network that considers cabin temperature control, specifically including the following steps:

[0153] S41: Create energy management environment model: Construct the power system model and cabin temperature control model of each vehicle in the vehicle queue, such as Figure 3 As shown; set the initial values ​​of the power system operating mode, battery SOC, battery SOH and cabin temperature;

[0154] S42: Building the State Space In order to control the cabin temperature more energy-efficiently and reduce vehicle energy consumption, the cabin temperature control related parameters and the power system energy consumption related parameters are combined to form a state space. That is:

[0155]

[0156] Among them, T cab,i ,e cab,i ,Δe cab,i are the cabin temperature of vehicle i, the error between the actual cabin temperature and the set temperature, and the error change rate; SOC i ,P d,i ,P cab,i are the battery SOC value, power system demand power and cabin temperature control system power of vehicle i respectively.

[0157] S43: Determine the action space The agent's output actions need to control both the cabin temperature and the power distribution of the power system. Therefore, the action space consists of two parts: the control instructions for the cabin temperature control system and the operating mode of the power system. Specifically,

[0158]

[0159] Among them, a h,i ,a AC,i are the opening degrees of the heating system and air conditioning system of vehicle i, respectively, which are discrete variables, and a m,i is the operating mode number of the vehicle i power system; the vehicle has five operating modes, namely driving mode (-1): the engine is turned off, but the transmission keeps the engine idling; parking mode (0): the engine is turned off, the speed is zero, and the torque is zero; starting mode (1): the transmission starts the engine to idle speed, and no fuel is consumed; idle mode (2): the engine is on but idling, and no torque is generated; starting mode (3): the engine is on, and fuel is supplied to the engine.

[0160] S44: Constructing the reward function Reward Function Including cabin temperature control reward function R cab,i and the power system energy consumption reward function R EMS,i Two parts, namely:

[0161]

[0162] The cabin temperature reward function R cab,i is the temperature error e cab,i The absolute value of the temperature error change rate Δe cab,i The absolute value of the temperature control system power P cab,i and the temperature penalty term l cab,iThe weighted sum of , that is:

[0163]

[0164] in, are the temperature errors e cab,i The absolute value of the temperature error change rate Δe cab,i The absolute value of the temperature control system power P cab,i and the temperature penalty term l cab,i The weight of the temperature penalty term l cab,i The actual cabin temperature of vehicle i must not exceed the set upper and lower temperature limits;

[0165] The power system energy consumption reward function R EMS,i Is the battery SOC indicator c SOC,i , fuel consumption index c f,i and battery SOH index c SOH,i and battery SOC penalty term l e,i The weighted sum of , that is:

[0166]

[0167] in, They are the battery SOC index c SOC,i , fuel consumption index c f,i and battery SOH index c SOH,i and battery SOC penalty term l e,i The weight of the battery SOC index c SOC,i The battery SOC change of vehicle i is required to be as small as possible when the battery is discharged, and the battery is required to be as small as possible when the battery is charged. SOC,i =0; wherein the fuel consumption index c f,i is the fuel consumption rate of vehicle i; wherein the battery SOH index c SOH,i The capacity loss of the battery of vehicle i is required to be as small as possible; the battery SOC penalty term l e,i The battery SOC of vehicle i is required not to exceed the set upper and lower limits of the battery SOC;

[0168] S441: In this embodiment, the temperature penalty term l cab,i The actual cabin temperature of vehicle i is required not to exceed the set upper and lower temperature limits. When the temperature exceeds the range, a negative reward related to the driving time will be received, which is expressed as:

[0169]

[0170] Among them, T cab,min ,T cab,maxare the minimum and maximum cabin temperatures, respectively, T is the total duration of the driving task, and t is the current time step;

[0171] S442: In this embodiment, the energy recovery during vehicle braking is not considered, and the battery SOC indicator c SOC,i It can be expressed as:

[0172]

[0173] Among them, ΔSOC i is the SOC change at the current moment;

[0174] S443: In this embodiment, the battery SOH indicator c SOH,i The capacity loss of the battery of vehicle i is required to be as small as possible, and its expression is:

[0175] c SOH,i =1-SOH i

[0176] S444: In this embodiment, the battery SOC penalty item l e,i The battery SOC of vehicle i is required not to exceed the set upper and lower limits of the battery SOC. When the battery SOC exceeds the range, it will receive a negative reward related to the driving time, which is expressed as:

[0177]

[0178] Among them, SOC min ,SOC max are the minimum and maximum values ​​of battery SOC respectively;

[0179] S45: Construct a rule-based power allocation model: Using the working mode output by the deep reinforcement learning agent as input, the set power allocation rules are used to complete the power distribution of the power system under the selected working mode; since there is no energy consumption in the parking mode (0) and only a small amount of power consumption in the starting mode (1), no power allocation rules are set in these two working modes.

[0180] S46: Constructing the network structure of the Deep Q Network algorithm: The Deep Q Network algorithm is an algorithm that combines deep learning and reinforcement learning. It has two neural networks with the same structure. The evaluation network is used to generate the Q value of all possible actions in the current state, and the target network is used to generate the target Q value.

[0181] S47: The experience replay pool of the deep Q network algorithm stores the experience data of the interaction between the intelligent agent and the energy management environment model. During model training, a certain amount of data is randomly extracted from the experience replay pool. At the same time, new data from the continuous interaction between the intelligent agent and the energy management environment model will also be stored in the experience replay pool.

[0182] Step S5, training the energy-saving driving hierarchical control model based on deep reinforcement learning, specifically includes the following steps:

[0183] S51: Initialize a collaborative adaptive cruise control model based on a dual-delayed deep deterministic policy gradient algorithm and an energy management model considering cabin temperature control based on a deep Q-network, including the hyperparameters of each neural network and the experience replay pool;

[0184] S52: Collaborative Adaptive Cruise Control Model Based on Dual-Delay Deep Deterministic Policy Gradient Algorithm: Connecting the environment model with the algorithm model, generating interactive data during training (S i,t ,A i,t ,R i,t ,S i,t+1 ) is added to the experience replay pool D1;

[0185] At each time step, the gradient descent algorithm is used to minimize the loss function L to update the evaluation networks Critic1 and Critic2:

[0186]

[0187] Where y is the target value; are the evaluation value function and the target evaluation value function, θ i ,θ i ′ are the parameters of the evaluation network and the target evaluation network respectively; S i,t ,A i,t ,R i,t S is the vehicle’s current state, actions taken, and rewards obtained; i,t+1 is the state of the vehicle at the next moment; γ is the discount factor; π φ′ is the target policy function;∈ is the action noise; State S i,t+1 The sum of the actions taken according to the target policy function and the action noise ∈;

[0188] After the evaluation networks Critic1 and Critic2 are updated for d steps, the policy network Actor is updated using deterministic policy gradients:

[0189]

[0190] Here, φ is the parameter of the policy network.

[0191] Update the two target evaluation networks and target policy networks separately:

[0192] θ i ′=τθ i +(1-τ)θ i′,for i=1,2

[0193] φ′=τφ+(1-τ)φ′

[0194] Among them, τ is the soft update coefficient, φ′ is the parameter of the target policy network;

[0195] S53: Energy management model considering cabin temperature control based on deep Q network: Connecting the environmental model and algorithm model to generate interactive data during training Added to experience replay pool D2;

[0196] At each time step, the loss function L is minimized using the gradient descent method E To update the evaluation network:

[0197]

[0198] in, are the evaluation value function and the target evaluation value function, θ E,i ,θ E,i ′ are the parameters of the evaluation network and the target network respectively; The vehicle's current state, actions taken, rewards obtained, and state at the next moment; Status possible actions to take;

[0199] The target network parameters are updated by hard updating, and the network parameters θ are evaluated every C steps. E,i Copy to the target network;

[0200] S54: Iteratively update the energy-saving driving hierarchical control model based on deep reinforcement learning: After the energy-saving driving hierarchical control model based on deep reinforcement learning converges, test the model. If it does not meet expectations, optimize the model's hyperparameters and reward function, and return to step S51.

[0201] In the above embodiments, references to "this embodiment" in the specification indicate that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment" do not necessarily refer to the same embodiment.

[0202] In the above embodiments, although the invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory structures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed. The embodiments of the present invention are intended to encompass all such alternatives, modifications, and variations that fall within the broad scope of the appended claims.

[0203] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the methods in this embodiment is implemented.

[0204] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0205] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes any one of the methods in this embodiment.

[0206] Regarding the computer-readable storage medium in this embodiment, those skilled in the art will appreciate that all or part of the steps in the aforementioned method embodiments can be implemented using hardware associated with the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps in the aforementioned method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0207] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used for communication, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes the various steps of the above method.

[0208] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0209] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0210] The present invention can be used in a wide variety of general-purpose or special-purpose computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments that include any of the above.

[0211] The present invention may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0212] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A hierarchical control method for hybrid electric vehicle platooning energy-saving driving based on deep reinforcement learning, characterized by: The following steps are involved: S1: Construct a vehicle platoon motion model based on the vehicle longitudinal dynamics model and establish vehicle platoon motion constraints; S2: Use V2V communication technology to obtain vehicle operation data; S3: Build an upper-level collaborative adaptive cruise control model based on a double-delayed deep deterministic policy gradient algorithm to provide vehicle speed data for the lower-level energy management model; S4: Construct a lower-level energy management model based on a deep Q network that considers cabin temperature control; S5: training the lower-layer energy management model; Step S4 specifically includes the following steps: S41: Create an energy management environment model: Build a powertrain model and cabin thermal management model for each vehicle in the platoon; set initial values ​​for the powertrain operating mode, battery SOC, battery SOH, and cabin temperature; S42: Build the state space of vehicle i : in, are the cabin temperature of vehicle i, the error between the actual cabin temperature and the set temperature, and the error change rate; are the battery SOC value, power system demand power and cabin temperature control system power of vehicle i respectively; is the speed of vehicle i at the next moment; is the acceleration of vehicle i, provided by the upper-layer cooperative adaptive cruise control model; is the time step; S43: Building an Action Space , which consists of two parts: the control instructions of the cabin temperature control system and the working mode of the power system, as follows: in, are the opening degrees of the heating system and air conditioning system of vehicle i, which are discrete variables. is the operating mode number of the vehicle i power system; the vehicle has five operating modes, namely driving mode (-1): the engine is off, but the transmission keeps the engine idling; parking mode (0): the engine is off, the speed is zero, and the torque is zero; starting mode (1): the transmission starts the engine to idle speed, and no fuel is consumed; idle mode (2): the engine is on but idling, and no torque is generated; starting mode (3): the engine is on, and fuel is supplied to the engine; S44: Constructing the reward function , the reward function Including cabin temperature reward function and the power system energy consumption reward function Two parts, namely: The cabin temperature reward function is the temperature error Absolute value and temperature error change rate The absolute value of the temperature control system power and temperature penalty The weighted sum of , that is: in, Temperature error Absolute value and temperature error change rate The absolute value of the temperature control system power and temperature penalty The weight of the temperature penalty term The actual cabin temperature of vehicle i must not exceed the set upper and lower temperature limits; The power system energy consumption reward function Is the battery SOC indicator , fuel consumption index and battery SOH indicators and battery SOC penalty The weighted sum of , that is: in, They are battery SOC indicators , fuel consumption index and battery SOH indicators and battery SOC penalty The weight of the battery SOC indicator The battery SOC change of vehicle i is required to be as small as possible when the battery is discharged, and the battery is required to be as small as possible when the battery is charged. ; The fuel consumption index is the fuel consumption rate of vehicle i; wherein the battery SOH index The battery capacity loss of vehicle i is required to be as small as possible; the battery SOC penalty term The battery SOC of vehicle i is required not to exceed the set upper and lower limits of the battery SOC; S45: Construct a rule-based power allocation model: Using the working mode output by the deep reinforcement learning agent as input, the set power allocation rules are used to complete the power distribution of the power system under the selected working mode; no power allocation rules are set for parking mode (0) and starting mode (1); S46: Construct the network structure of the deep Q network algorithm to generate the Q values ​​of all possible actions in the current state, and the target network to generate the target Q value; S47: The experience replay pool of the deep Q network algorithm stores the experience data of the interaction between the intelligent agent and the energy management environment model; during model training, a certain amount of data is randomly extracted from the experience replay pool, and new data of the continuous interaction between the intelligent agent and the energy management environment model is also stored in the experience replay pool.

2. The hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning according to claim 1 is characterized by: The vehicle longitudinal dynamics model is as follows: in, is the position of vehicle i, are the speed, driving force and acceleration of vehicle i, are the mass and rotational mass conversion coefficient of vehicle i, respectively.

3. The hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning according to claim 1 is characterized by: The vehicle platoon motion constraints include spacing constraints, speed constraints, acceleration constraints, string stability constraints, and headway constraints: in, is the distance between vehicle i and the preceding vehicle; is the minimum distance between vehicle i and the preceding vehicle; is the maximum speed of vehicle i; are the minimum acceleration and maximum acceleration of vehicle i respectively; Indicates the string stability of vehicle i, setting the distance between the leading vehicle and its following vehicle Similarly, the rear vehicle and the vehicle in front same; are the travel distances per unit time of vehicle i, preceding vehicle i-1, and following vehicle i+1 respectively; is the headway; is the limit headway; are the distance traveled between vehicle i and the following vehicle i+1, the vehicle length, and the speed of vehicle i.

4. The hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning according to claim 1 is characterized by: The step S2 of obtaining the vehicle's operating data by using V2V communication technology includes: The leading vehicle in the platoon uses V2V technology to obtain the operating data of the pilot vehicle, including: the pilot vehicle's speed, acceleration, and distance traveled per unit time; Vehicles other than the lead vehicle in a vehicle queue use V2V technology to obtain the operating data of the leading and trailing vehicles, including their speeds and travel distances per unit time. The tail vehicle in the vehicle queue obtains the operating data of the leading vehicle.

5. The hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning according to claim 1 is characterized by: Step S3 specifically includes the following steps: S31: Create a vehicle platoon operating environment model: Construct a vehicle platoon consisting of a lead vehicle and multiple follower vehicles. Each follower vehicle in the platoon is a control vehicle, and the sequence number is set to 1, 2, ..., n. The trajectory of the lead vehicle is set automatically, and the sequence number of the lead vehicle is set to 0. Set the initial distance between the lead vehicle and the lead vehicle, the initial distance of the platoon, and the initial speed of the platoon. S32: Based on the motion simulation model of the vehicle platoon and the vehicle operation status, build the state space of vehicle 1 : in, are the speed and acceleration of the pilot car respectively; Build the state space for the remaining vehicles : in, is the speed of the preceding vehicle; S33: Building the Action Space , its action variables are extracted from the acceleration calculation formula, as follows: in, is the position error coefficient between vehicle i and the preceding vehicle, which is set as a constant; is the speed error coefficient between vehicle i and the preceding vehicle; S34: Constructing reward function , the reward function Including vehicle-level reward function and the dynamical system layer reward function Two parts, namely: The vehicle-level reward function Is the acceleration index , acceleration index , headway index and string stability index The weighted sum of , that is: in, Acceleration index , acceleration index , headway index and string stability index The weight of the acceleration index Vehicle i is required to meet the acceleration constraint condition; the acceleration index The vehicle jerk is required to be as small as possible to ensure comfort; the headway index It is required that vehicle i maintain a certain safe distance from the vehicle in front, and vehicle i has sufficient reaction time when the vehicle in front brakes; the string stability index It is required that the distance error between vehicle i and the preceding and following vehicles in the platoon will not propagate along the platoon and be amplified, and the distance errors between all vehicles in the platoon should maintain the same accuracy to avoid collisions. The dynamical system layer reward function Is the fuel consumption indicator , battery SOC indicator and battery SOH indicators The weighted sum of , that is: in, Fuel consumption index , battery SOC indicator and battery SOH indicators The weight of the fuel consumption index The fuel consumption rate of vehicle i is required to be as low as possible below the maximum fuel consumption rate; wherein the battery SOC index The battery SOC of vehicle i is required to be kept within a certain range; wherein the battery SOH indicator The capacity loss of the battery of vehicle i is required to be as small as possible; S35: Construct the network structure of the dual-delayed deep deterministic policy gradient algorithm: the policy network Actor determines the action taken in the current state, and the two evaluation networks Critic1 and Critic2 are used to evaluate the current state and action; the target policy network Target-Actor and the target evaluation networks Target-Critic1 and Target-Critic2 have the same network structure as the corresponding main network to improve learning stability; S36: The experience replay pool of the double-delayed deep deterministic policy gradient algorithm stores the experience data of the interaction between the intelligent agent and the vehicle queue operation environment model. During model training, a certain amount of data is randomly extracted from the experience replay pool, and new data from the continuous interaction between the intelligent agent and the vehicle queue operation environment model is also stored in the experience replay pool.

6. The hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning according to claim 1 is characterized by: The training of the energy-saving driving hierarchical control model based on deep reinforcement learning in step S5 specifically includes the following steps: S51: Initialize the upper-layer collaborative adaptive cruise control model based on the double-delayed deep deterministic policy gradient algorithm and the lower-layer energy management model based on the deep Q network considering cabin temperature control, including the hyperparameters of each neural network and the experience replay pool; S52: Upper-layer collaborative adaptive cruise control model based on dual-delay deep deterministic policy gradient algorithm: Connecting the environment model and the algorithm model to generate interactive data during training Add to experience replay pool ; At each time step, the loss function is minimized using the gradient descent algorithm To update the evaluation networks Critic1 and Critic2: in, is the target value; are the evaluation value function and the target evaluation value function, respectively. are the parameters of the evaluation network and the target evaluation network respectively; The vehicle's current state, actions taken, and rewards obtained; is the state of the vehicle at the next moment; is the discount factor; is the target policy function; It is the action noise; Status The actions taken according to the target policy function and the action noise The sum of After evaluating the networks Critic1 and Critic2 for d steps, the policy network Actor is updated using deterministic policy gradients: in, are the parameters of the policy network; Update the two target evaluation networks and target policy networks separately: in, is the soft update coefficient, are the parameters of the target policy network; S53: Lower-level energy management model considering cabin temperature control based on deep Q network: connecting the environmental model and the algorithm model, and generating interactive data during the training process Add to experience replay pool ; At each time step, the loss function is minimized using gradient descent To update the evaluation network: in, are the evaluation value function and the target evaluation value function, respectively. are the parameters of the evaluation network and the target network respectively; The vehicle's current state, actions taken, rewards obtained, and state at the next moment; Status possible actions to take; The target network parameters are updated by hard updating, and the network parameters are evaluated every C steps. Copy to the target network; S54: Iteratively update the energy-saving driving hierarchical control model based on deep reinforcement learning: After the energy-saving driving hierarchical control model based on deep reinforcement learning converges, test the model. If it does not meet expectations, optimize the model's hyperparameters and reward function, and return to step S51.

Images