Hybrid electric vehicle queue energy-saving driving hierarchical control method based on deep reinforcement learning

Abstract

The invention relates to a hybrid electric vehicle queue energy-saving driving hierarchical control method based on deep reinforcement learning, and belongs to the technical field of new energy automobiles, and the method comprises the following steps: S1, constructing a vehicle queue motion model according to a vehicle longitudinal dynamics model, and constructing vehicle queue motion constraint conditions; s2, acquiring operation data of the vehicle by using a V2V communication technology; s3, constructing an upper-layer cooperative adaptive cruise control model based on a double-delay depth deterministic strategy gradient algorithm, and providing vehicle speed data for a lower-layer energy management model; s4, constructing a lower layer energy management model based on the deep Q network and considering cabin temperature control; and S5, training the lower-layer energy management model. According to the invention, the safety and the energy-saving property of the vehicle queue of the hybrid electric vehicle can be effectively improved, and meanwhile, the thermal comfort of a cabin is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of new energy vehicles and relates to a hybrid vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning. Background Art

[0002] With the rapid development and gradual implementation of intelligent networked transportation systems, vehicle networking and automation have become a research hotspot in the field of new energy vehicles. Vehicle platooning can effectively reduce the following distance and greatly reduce the aerodynamic drag coefficient of the following vehicle, which has a high energy-saving potential. In a vehicle platoon composed of networked vehicles, vehicles can not only share information, but also set common goals for unified planning, which significantly improves the effects of reducing vehicle energy consumption, ensuring the stability of vehicle platoons, and reducing collisions.

[0003] Current vehicle platoon control technology has achieved a series of results in terms of vehicle platoon stability and safety. However, most existing studies have ignored the energy consumption of the cabin temperature control system, which has affected the energy consumption performance of the vehicle platoon to a certain extent. The energy consumption of cabin temperature control is related to the power output of the power system. The operating mode of the power system is related to the vehicle speed, and the vehicle speed determines the power demand of the power system. That is, cabin temperature control, vehicle speed planning and energy management of the power system are three interrelated sub-problems. Most traditional vehicle platoon control methods rely on pre-set rules and models. These methods have poor optimality and real-time performance when dealing with multiple coupled optimization objectives. In recent years, reinforcement learning technology has received widespread attention in autonomous driving decision-making. Reinforcement learning continuously learns the optimal strategy by interacting with the environment, and can achieve more flexible and efficient decision-making in complex tasks. However, single-agent reinforcement learning is difficult to cope with complex, multi-objective collaborative optimization tasks, and the generalization ability of the decisions made is insufficient. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a hierarchical control method for hybrid electric vehicle platoon energy-saving driving based on deep reinforcement learning, which plans the vehicle driving speed through a double-delay deep deterministic policy gradient algorithm model, and realizes the energy management of the vehicle power system and the cabin temperature control system based on the deep Q network algorithm model, thereby ensuring the safety of vehicle platoon driving, reducing the energy consumption of the vehicle, and improving the thermal comfort of the cabin.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning comprises the following steps:

[0007] S1: Construct a vehicle platoon motion model based on the vehicle longitudinal dynamics model, and construct vehicle platoon motion constraints;

[0008] S2: Use V2V communication technology to obtain vehicle operation data;

[0009] S3: Build an upper-layer collaborative adaptive cruise control model based on a double-delayed deep deterministic policy gradient algorithm to provide vehicle speed data for the lower-layer energy management model;

[0010] S4: Construct a lower-level energy management model based on deep Q network considering cabin temperature control;

[0011] S5: Training the lower-layer energy management model.

[0012] Furthermore, the vehicle longitudinal dynamics model is as follows:

[0013]

[0014] in, is the vehicle i and position, v i ,a i ,F a,i are the speed, driving force and acceleration of vehicle i, respectively, m i ,δ i are the mass and rotational mass conversion factor of vehicle i respectively.

[0015] Furthermore, the vehicle queue motion constraints include spacing constraints, speed constraints, acceleration constraints, string stability constraints and headway constraints:

[0016] Δs min,i ≤Δs i

[0017] 0≤v i ≤v max,i

[0018] a min,i ≤a i ≤a max,i

[0019]

[0020] Where, Δs i is the distance between vehicle i and the preceding vehicle; Δs min,i is the minimum distance between vehicle i and the vehicle in front; v max,i is the maximum speed of vehicle i; a min,i ,a max,i are the minimum acceleration and maximum acceleration of vehicle i respectively; l i represents the string stability of vehicle i, and sets the l between the leading vehicle and its following vehiclei Similarly, the l of the rear vehicle and the vehicle in front of it i Same;d i ,d i-1 ,d i+1 h are the travel distances per unit time of vehicle i, the preceding vehicle i-1, and the following vehicle i+1 respectively; i is the headway; h min is the limit headway; s i ,s i+1 ,l v ,v i are the distance between vehicle i and the front of vehicle i+1, the vehicle length and the speed of vehicle i respectively.

[0021] Further, the step S2 of obtaining the vehicle operation data by using the V2V communication technology includes:

[0022] The leading vehicle in the vehicle queue uses V2V technology to obtain the operating data of the pilot vehicle, including: the pilot vehicle's speed, acceleration, and travel distance per unit time;

[0023] The other vehicles in the vehicle queue, except the head vehicle, use V2V technology to obtain the operating data of the front and rear vehicles, including: the speed of the front and rear vehicles and the driving distance per unit time; the tail vehicle in the vehicle queue obtains the operating data of the front vehicle.

[0024] Further, step S3 specifically includes the following steps:

[0025] S31: Create a vehicle platoon operation environment model: construct a leading vehicle and a vehicle platoon consisting of multiple following vehicles, each following vehicle in the vehicle platoon is a control vehicle, and the serial number is set to 1, 2, ..., n in sequence; the trajectory of the leading vehicle is set automatically, and the serial number of the leading vehicle is set to 0; set the initial distance between the head vehicle and the leading vehicle, the initial distance of the vehicle platoon, and the initial speed of the vehicle platoon;

[0026] S32: Based on the motion simulation model of the vehicle platoon and the vehicle operation status, build the state space S1 of vehicle 1:

[0027] S1={v0,a0,d i}

[0028] Among them, v0 and a0 are the speed and acceleration of the pilot car respectively;

[0029] Build the state space S of the remaining vehicles 2,...,n :

[0030] S 2,...,n = {v i-1 ,v i ,d i ,h i ,li}

[0031] Among them, v i-1 is the speed of the preceding vehicle;

[0032] S33: Building Action Space A i , whose action variables are extracted from the acceleration calculation formula, as follows:

[0033] a i =-k x,i (s i -s i-1 +l v )-k v,i (v i -v i-1 ),

[0034] A i ={k v,i |0≤k v,i ≤10}

[0035] Among them, k x,i is the position error coefficient between vehicle i and the preceding vehicle, set as a constant; k v,i is the speed error coefficient between vehicle i and the preceding vehicle;

[0036] S34: Construct reward function R i , the reward function R i Including the vehicle layer reward function R v,i and the dynamical system layer reward function R p,i Two parts, namely:

[0037] R i =-R v,i -R p,i

[0038] The vehicle layer reward function R v,i is the acceleration index r Acc,i , acceleration index r J,i , headway index r h,i And string stability index r l,i The weighted sum of is:

[0039] R v,i =α 1,i r Acc,i +α 2,i r J,i +α 3,i r h,i +α 4,i r l,i

[0040] Among them, α 1,i ,α 2,i,α 3,i ,α 4,i They are acceleration index r Acc,i , acceleration index r J,i , headway index r h,i And string stability index r l,i The weight of the acceleration index r Acc,i The vehicle i is required to meet the acceleration constraint condition; the acceleration index r J,i The vehicle i is required to have a minimum jerk as possible to ensure comfort; the headway index r h,i It is required that vehicle i maintain a certain safe distance from the vehicle in front, and vehicle i has sufficient reaction time when the vehicle in front brakes; the string stability index r l,i It is required that the distance error between vehicle i and the vehicles in front and behind in the queue will not propagate along the queue and be amplified, and the distance errors between all vehicles in the platoon should maintain the same accuracy to avoid collisions;

[0041] The dynamical system layer reward function R p,i is the fuel consumption index r f,i 、Battery SOC index r SOC,i and battery SOH index SOH,i The weighted sum of is:

[0042] R p,i =β 1,i r f,i +β 2,i r SOC,i +β 3,i r SOH,i

[0043] Among them, β 1,i ,β 2,i ,β 3,i They are respectively the fuel consumption index r f,i 、Battery SOC index r SOC,i and battery SOH index SOH,i The weight of the fuel consumption index r f,i The fuel consumption rate of vehicle i is required to be as low as possible from the maximum fuel consumption rate; wherein the battery SOC index r SOC,i The battery SOC of vehicle i is required to be kept within a certain range; wherein the battery SOH index r SOH,i The capacity loss of the battery of vehicle i is required to be as small as possible;

[0044] S35: Construct the network structure of the double-delayed deep deterministic policy gradient algorithm: the policy network Actor determines the action taken in the current state, and the two evaluation networks Critic1 and Critic2 are used to evaluate the current state and action; the network structure of the target policy network Target-Actor, the target evaluation network Target-Critic1 and Target-Critic2 is the same as the corresponding main network structure, which is used to improve the stability of learning;

[0045] S36: The experience replay pool of the double-delay deep deterministic policy gradient algorithm stores the experience data of the interaction between the intelligent agent and the vehicle queue operation environment model; during model training, a certain amount of data is randomly extracted from the experience replay pool, and new data of the continuous interaction between the intelligent agent and the vehicle queue operation environment model is also stored in the experience replay pool.

[0046] Further, step S4 specifically includes the following steps:

[0047] S41: Create an energy management environment model: construct a power system model and a cabin thermal management model for each vehicle in the vehicle queue; set the initial values ​​of the power system operating mode, battery SOC, battery SOH and cabin temperature;

[0048] S42: Build the state space of vehicle i

[0049]

[0050] Among them, T cab,i ,e cab,i ,Δe cab,i are the cabin temperature of vehicle i, the error between the actual cabin temperature and the set temperature, and the error change rate; SOC i ,P d,i ,P cab,i are the battery SOC value, power system demand power and cabin temperature control system power of vehicle i respectively; v fi is the speed of vehicle i at the next moment; a i is the acceleration of vehicle i, provided by the upper-layer cooperative adaptive cruise control model; Δt is the time step;

[0051] S43: Building an Action Space It consists of two parts: the control instructions of the cabin temperature control system and the working mode of the power system, as follows:

[0052]

[0053] Among them, a h,i ,a AC,i are the opening degrees of the heating system and air conditioning system of vehicle i, respectively, are discrete variables, and am,i is the working mode number of the power system of vehicle i; the vehicle has five working modes, namely driving mode (-1): the engine is turned off, but the transmission keeps the engine idling; parking mode (0): the engine is turned off, the speed is zero, and the torque is zero; starting mode (1): the transmission starts the engine to idle speed, and no fuel is consumed; idle mode (2): the engine is in the on state but idling, and no torque is generated; starting mode (3): the engine is on, and fuel is supplied to the engine;

[0054] S44: Constructing the reward function Reward Function Including cabin temperature reward function R cab,i and the power system energy consumption reward function R EMS,i Two parts, namely:

[0055]

[0056] The cabin temperature reward function R cab,i is the temperature error e cab,i The absolute value of the temperature error change rate Δe cab,i The absolute value of the temperature control system power P cab,i and the temperature penalty term l cab,i The weighted sum of is:

[0057]

[0058] in, They are temperature errors e cab,i The absolute value of the temperature error change rate Δe cab,i The absolute value of the temperature control system power P cab,i and the temperature penalty term l cab,i The weight of the temperature penalty term l cab,i It is required that the actual cabin temperature of vehicle i cannot exceed the set upper and lower temperature limits;

[0059] The power system energy consumption reward function R EMS,i Is the battery SOC indicator c SOC,i 、Fuel consumption indexc f,i And battery SOH index c SOH,i and battery SOC penalty term l e,i The weighted sum of is:

[0060]

[0061] in, They are the battery SOC index c SOC,i 、Fuel consumption indexc f,i And battery SOH index cSOH,i and battery SOC penalty term l e,i The weight of the battery SOC index c SOC,i The battery SOC change of vehicle i during battery discharge is required to be as small as possible, and the battery is required to be as stable as possible during battery charging. SOC,i =0; wherein the fuel consumption index c f,i is the fuel consumption rate of vehicle i; wherein the battery SOH index c SOH,i The capacity loss of the battery of vehicle i is required to be as small as possible; the battery SOC penalty term l e,i It is required that the battery SOC of vehicle i cannot exceed the set upper and lower limits of the battery SOC;

[0062] S45: Construct a rule-based power allocation model: take the working mode output by the deep reinforcement learning agent as input, and use the set power allocation rule to complete the power allocation of the power system under the selected working mode; no power allocation rule is set for the parking mode (0) and the starting mode (1);

[0063] S46: constructing a network structure of a deep Q network algorithm for generating Q values ​​of all possible actions in the current state, and a target network for generating a target Q value;

[0064] S47: The experience replay pool of the deep Q network algorithm stores the experience data of the interaction between the intelligent agent and the energy management environment model. During model training, a certain amount of data is randomly extracted from the experience replay pool, and new data of the continuous interaction between the intelligent agent and the energy management environment model is also stored in the experience replay pool.

[0065] Further, the training of the energy-saving driving hierarchical control model based on deep reinforcement learning in step S5 specifically includes the following steps:

[0066] S51: Initialize the upper-layer collaborative adaptive cruise control model based on the double-delayed deep deterministic policy gradient algorithm and the lower-layer energy management model considering cabin temperature control based on the deep Q network, including the hyperparameters of each neural network and the experience replay pool;

[0067] S52: Upper-layer collaborative adaptive cruise control model based on double-delayed deep deterministic policy gradient algorithm: Connect the environment model with the algorithm model, and generate interactive data during training (S i,t ,A i,t ,R i,t ,S i,t+1 ) is added to the experience replay pool D1;

[0068] At each time step, the gradient descent algorithm is used to minimize the loss function L to update the evaluation networks Critic1 and Critic2:

[0069] L=N-1 ∑(yQ θi (S i,t ,A i,t )) 2

[0070]

[0071] Among them, y is the target value; are the evaluation value function and the target evaluation value function, θ i ,θ i ′ are the parameters of the evaluation network and the target evaluation network respectively; S i,t ,A i,t ,R i,t S is the vehicle’s current state, actions taken, and rewards obtained; i,t+1 is the state of the vehicle at the next moment; γ is the discount factor; π φ′ is the target policy function; ∈ is the action noise; For state S i,t+1 The sum of the actions taken according to the target policy function and the action noise ∈;

[0072] After evaluating the networks Critic1 and Critic2 for d steps, the policy network Actor is updated using deterministic policy gradients:

[0073]

[0074] Among them, φ is the parameter of the policy network;

[0075] Update the two target evaluation networks and the target policy network separately:

[0076] θ i ′=τθ i +(1-τ)θ i ′,for i=1,2

[0077] φ′=τφ+(1-τ)φ′

[0078] Among them, τ is the soft update coefficient, φ′ is the parameter of the target policy network;

[0079] S53: Lower-level energy management model considering cabin temperature control based on deep Q network: connect the environmental model and the algorithm model to generate interactive data during the training process Added to experience replay pool D2;

[0080] At each time step, the loss function L is minimized using the gradient descent method E To update the evaluation network:

[0081]

[0082] in, are the evaluation value function and the target evaluation value function, θ E,i ,θ E,i ′ are the parameters of the evaluation network and the target network respectively; The vehicle’s current state, actions taken, rewards obtained, and state at the next moment; Status possible actions to be taken;

[0083] The target network parameters are updated by hard updating, and the network parameters θ are evaluated every C steps. E,i Copy to the target network;

[0084] S54: Iteratively update the energy-saving driving hierarchical control model based on deep reinforcement learning: After the energy-saving driving hierarchical control model based on deep reinforcement learning converges, test the model. If it does not meet expectations, optimize the model's hyperparameters and reward function, and return to step S51.

[0085] The beneficial effects of the present invention are:

[0086] 1) The present invention designs an adaptive cruise control method for vehicle queues, adds a string stability index and a headway index into the reward function, safely shortens the distance between vehicles, reduces aerodynamic drag, and thus reduces energy consumption.

[0087] 2) The present invention designs an energy management strategy that takes cabin temperature control into consideration, which synergistically optimizes the energy consumption of the cabin temperature control system and the power system, ensures the thermal comfort of the cabin, and improves the economy of the entire vehicle.

[0088] 3) The present invention designs a hierarchical control method for energy-saving driving of hybrid electric vehicle platoons. Based on a multi-agent deep reinforcement learning framework, it realizes the coordinated optimization of vehicle speed, cabin temperature and vehicle energy consumption, effectively improving the safety, comfort and economy of vehicle platoon operation.

[0089] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0091] Figure 1This is a logical structure diagram of the hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning according to the present invention;

[0092] Figure 2 This is a schematic diagram of the deep reinforcement learning vehicle platoon operating environment model;

[0093] Figure 3 Schematic diagram of the energy management environment model for deep reinforcement learning;

[0094] Figure 4 Update the training process diagram for the multi-agent deep reinforcement learning model;

[0095] Figure 5 This is the overall flow chart of the hierarchical control method for hybrid electric vehicle platoon energy-saving driving based on deep reinforcement learning. DETAILED DESCRIPTION

[0096] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0097] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0098] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0099] like Figures 1 to 5As shown, the present invention provides a hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning. Considering the information sharing problem between vehicles in a real vehicle platoon, V2V communication technology is used to obtain vehicle operation data. At the same time, considering the coupling relationship between vehicle speed planning, power system energy distribution and cabin temperature control during vehicle platoon driving, a higher safety, better economy and stronger generalization based energy-saving driving hierarchical control method is proposed. The method specifically includes the following steps:

[0100] Step S1, constructing a vehicle platoon motion model based on the vehicle longitudinal dynamics model, and constructing vehicle platoon motion constraint conditions. Specifically, the following steps are included:

[0101] S11: The vehicle adopts the following longitudinal dynamics model, as follows:

[0102]

[0103] S12: In order to ensure the comfort and safety of vehicles during driving, the vehicle queue motion constraints including spacing constraints, speed constraints, acceleration constraints, string stability constraints and headway constraints are constructed, namely:

[0104] Δs min,i ≤Δs i

[0105] 0≤v i ≤v max,i

[0106] a min,i ≤a i ≤a max,i

[0107]

[0108] Where, Δs i is the distance between vehicle i and the preceding vehicle; Δs min,i is the minimum distance between vehicle i and the vehicle in front; v max,i is the maximum speed of vehicle i; a min,i ,a max,i are the minimum acceleration and maximum acceleration of vehicle i respectively; l i represents the string stability of vehicle i, in particular, the l of the first and last vehicles in the vehicle queue i It is impossible to calculate according to the above formula. To simplify the constraints, the distance between the leading vehicle and the following vehicle is set to i Similarly, the l of the rear vehicle and the vehicle in front of it i Same;d i ,d i-1 ,d i+1h are the travel distances per unit time of vehicle i, the preceding vehicle i-1, and the following vehicle i+1 respectively; i is the headway; h min is the limit headway; s i ,s i+1 ,l v ,v i are the distance between vehicle i and the front of vehicle i+1, the vehicle length and the speed of vehicle i respectively.

[0109] Step S2, using V2V communication technology to obtain vehicle operation data, specifically includes the following steps:

[0110] S21: The head vehicle of the vehicle queue uses V2V technology to obtain the operating data of the pilot vehicle, including: the speed, acceleration, and travel distance per unit time of the pilot vehicle;

[0111] S22: The other vehicles in the vehicle queue, except the head vehicle, use V2V technology to obtain the operating data of the front and rear vehicles, including: the speed of the front and rear vehicles and the driving distance per unit time; the tail vehicle in the vehicle queue only needs to obtain the operating data of the front vehicle.

[0112] Step S3, constructing a collaborative adaptive cruise control model based on a double-delayed deep deterministic policy gradient algorithm, specifically comprising the following steps:

[0113] S31: Create a vehicle platoon operation environment model: In this example, an operation simulation environment including a pilot vehicle and a vehicle platoon consisting of three following vehicles is constructed, such as Figure 2 As shown in the figure, each following vehicle in the vehicle queue is a control vehicle, and the serial numbers are set to 1, 2, and 3 in sequence; the speed sequence of the pilot vehicle is set, and the serial number of the pilot vehicle is set to 0; the initial distance between the head vehicle and the pilot vehicle, the initial distance of the vehicle queue, and the initial speed of the vehicle queue are set. It is necessary to ensure that the vehicle distance and the initial speed of the vehicle queue meet the vehicle queue operation constraints to avoid vehicle collisions;

[0114] S32: Based on the motion simulation model of the vehicle platoon and the vehicle operation status, the state space S is constructed: During the operation of the vehicle platoon, the main task of the head vehicle is to follow the pilot vehicle, and it is necessary to focus on the operation data of the pilot vehicle, so the state space S1 of vehicle 1 is constructed:

[0115] S1={v0,a0,d i}

[0116] Among them, v0 and a0 are the speed and acceleration of the pilot car respectively;

[0117] For other vehicles in the vehicle queue except the head vehicle, they need to follow the front vehicle under the premise of ensuring the queue string stability index and the headway index. Therefore, the state space S of vehicles 2 and 3 is constructed. 2,3 :

[0118] S 2,3 = {v i-1 ,v i ,d i ,h i ,l i}

[0119] Among them, v i-1 is the speed of the preceding vehicle;

[0120] S33: Determine the action space A i :This method achieves safe and energy-saving following behavior by controlling the acceleration of the vehicle. Therefore, its action variables can be extracted from the acceleration calculation formula. The acceleration formula contains two parts: the position error and speed error of the front and rear vehicles. To simplify the action space, the coefficient of any part can be selected as the action variable, and the coefficient of the other part is set to a fixed value, as follows:

[0121] a i =-k x,i (s i -s i-1 +l v )-k v,i (v i -v i-1 ),

[0122] A i ={k v,i |0≤k v,i ≤10}

[0123] Among them, k x,i is the position error coefficient between vehicle i and the preceding vehicle, constant, k v,i is the speed error coefficient between vehicle i and the preceding vehicle;

[0124] S34: Construct reward function R i , the reward function R i Including the vehicle layer reward function R v,i and the dynamical system layer reward function R p,i Two parts, namely:

[0125] R i =-R v,i -R p,i

[0126] The vehicle layer reward function R v,i is the acceleration index r Acc,i , acceleration index rJ,i , headway index r h,i And string stability index r l,i The weighted sum of is:

[0127] R v,i =α 1,i r Acc,i +α 2,i r J,i +α 3,i r h,i +α 4,i r l,i

[0128] Among them, α 1,i ,α 2,i ,α 3,i ,α 4,i They are acceleration index r Acc,i , acceleration index r J,i , headway index r h,i And string stability index r l,i The weight of the acceleration index r Acc,i The vehicle i is required to meet the acceleration constraint condition; the acceleration index r J,i The vehicle i is required to have a minimum jerk as possible to ensure comfort; the headway index r h,i It is required that vehicle i maintain a certain safe distance from the vehicle in front, and vehicle i has sufficient reaction time when the vehicle in front brakes; the string stability index r l,i It is required that the distance error between vehicle i and the front and rear vehicles in the queue will not propagate along the queue and be amplified, and the distance errors between all vehicles in the platoon maintain the same accuracy to avoid collisions.

[0129] The dynamical system layer reward function R p,i is the fuel consumption index r f,i 、Battery SOC index r SOC,i and battery SOH index SOH,i The weighted sum of is:

[0130] R p,i =β 1,i r f,i +β 2,i r SOC,i +β 3,i r SOH,i

[0131] Among them, β 1,i ,β 2,i ,β 3,i They are respectively the fuel consumption index r f,i 、Battery SOC index r SOC,i and battery SOH indexSOH,i The weight of the fuel consumption index r f,i The fuel consumption rate of vehicle i is required to be as low as possible from the maximum fuel consumption rate; wherein the battery SOC index r SOC,i The battery SOC of vehicle i is required to be kept within a certain range; wherein the battery SOH index r SOH,i The capacity loss of the battery of vehicle i is required to be as small as possible;

[0132] S341: In this embodiment, the acceleration index r Acc,i The longitudinal acceleration of the vehicle is required to be kept small, and its expression is:

[0133]

[0134] Among them, a max,i Set to 1m / s 2 ;

[0135] S342: In this embodiment, to ensure comfort, the acceleration index r J,i The vehicle's jerk is required to be kept small, and its expression is:

[0136]

[0137] Among them, J com,i To meet the maximum jerk required for comfort, set it to 0.5 m / s 3 ; J com,i is the maximum acceleration of the vehicle, set to 4m / s 3 ;

[0138] S343: In this embodiment, the headway index r h,i It is required that when the front vehicle brakes, the vehicle has sufficient reaction time and keeps the distance between vehicles as small as possible. The expression is:

[0139]

[0140] Among them, h max,i is the maximum headway;

[0141] S344: In this embodiment, the string stability index r l,i It is required that the distance error between vehicles in a vehicle queue will not propagate along the queue and be amplified, and the distance error between all vehicles in the convoy should maintain the same accuracy to avoid collision. The expression is:

[0142]

[0143] S345: In this embodiment, the fuel consumption index r f,i It can be expressed as:

[0144]

[0145] in, is the fuel consumption rate of vehicle i at the current moment; is the maximum fuel consumption rate of vehicle i;

[0146] S346: In this embodiment, the battery SOC indicator r SOC,i The battery SOC of vehicle i is required to be kept within a certain range, and its expression is:

[0147]

[0148] S347: In this embodiment, the battery SOH indicator r SOH,i The capacity loss of the battery of vehicle i is required to be as small as possible, and its expression is:

[0149]

[0150] S35: Constructing the network structure of the double-delayed deep deterministic policy gradient algorithm: The double-delayed deep deterministic policy gradient algorithm is a deterministic deep reinforcement learning algorithm under the Actor-Critic (AC) framework. The policy network Actor determines the action taken in the current state, and the two evaluation networks Critic1 and Critic2 are used to evaluate the current state and action; the network structure of the target policy network Target-Actor, the target evaluation network Target-Critic1 and Target-Critic2 is the same as the corresponding main network structure, which is used to improve the stability of learning;

[0151] S36: The experience replay pool of the double-delay deep deterministic policy gradient algorithm stores the experience data of the interaction between the intelligent agent and the vehicle queue operation environment model; during model training, a certain amount of data is randomly extracted from the experience replay pool, and new data of the continuous interaction between the intelligent agent and the vehicle queue operation environment model will also be stored in the experience replay pool.

[0152] Step S4, constructing a lower-layer energy management model based on a deep Q network considering cabin temperature control, specifically including the following steps:

[0153] S41: Create an energy management environment model: Construct a power system model and a cabin temperature control model for each vehicle in the vehicle queue, such as Figure 3 As shown; set the initial values ​​of the power system operating mode, battery SOC, battery SOH and cabin temperature;

[0154] S42: Building the State Space In order to control the cabin temperature more energy-efficiently and reduce the energy consumption of the vehicle, the cabin temperature control related parameters and the power system energy consumption related parameters are combined to form a state space. That is:

[0155]

[0156] Among them, T cab,i ,e cab,i ,Δe cab,i are the cabin temperature of vehicle i, the error between the actual cabin temperature and the set temperature, and the error change rate; SOC i ,P d,i ,P cab,i They are the battery SOC value, power system demand power and cabin temperature control system power of vehicle i respectively.

[0157] S43: Determine the action space The output action of the agent needs to control the cabin temperature and the power distribution of the power system at the same time. Therefore, the action space consists of two parts: the control instructions of the cabin temperature control system and the working mode of the power system, as follows:

[0158]

[0159] Among them, a h,i ,a AC,i are the opening degrees of the heating system and air conditioning system of vehicle i, respectively, are discrete variables, and a m,i is the operating mode number of the vehicle i power system; the vehicle has five operating modes, namely, driving mode (-1): the engine is turned off, but the transmission keeps the engine idling; parking mode (0): the engine is turned off, the speed is zero, and the torque is zero; starting mode (1): the transmission starts the engine to idle, and no fuel is consumed; idle mode (2): the engine is in the on state but idling, and no torque is generated; starting mode (3): the engine is on, and fuel is supplied to the engine.

[0160] S44: Constructing the reward function Reward Function Including the cabin temperature control reward function R cab,i and the power system energy consumption reward function R EMS,i Two parts, namely:

[0161]

[0162] The cabin temperature reward function R cab,i is the temperature error e cab,i The absolute value of the temperature error change rate Δe cab,i The absolute value of the temperature control system power P cab,i and the temperature penalty term l cab,iThe weighted sum of is:

[0163]

[0164] in, They are temperature errors e cab,i The absolute value of the temperature error change rate Δe cab,i The absolute value of the temperature control system power P cab,i and the temperature penalty term l cab,i The weight of the temperature penalty term l cab,i It is required that the actual cabin temperature of vehicle i cannot exceed the set upper and lower temperature limits;

[0165] The power system energy consumption reward function R EMS,i Is the battery SOC indicator c SOC,i 、Fuel consumption indexc f,i And battery SOH index c SOH,i and battery SOC penalty term l e,i The weighted sum of is:

[0166]

[0167] in, They are the battery SOC index c SOC,i 、Fuel consumption indexc f,i And battery SOH index c SOH,i and battery SOC penalty term l e,i The weight of the battery SOC index c SOC,i The battery SOC change of vehicle i during battery discharge is required to be as small as possible, and the battery is required to be as stable as possible during battery charging. SOC,i =0; wherein the fuel consumption index c f,i is the fuel consumption rate of vehicle i; wherein the battery SOH index c SOH,i The capacity loss of the battery of vehicle i is required to be as small as possible; the battery SOC penalty term l e,i It is required that the battery SOC of vehicle i cannot exceed the set upper and lower limits of the battery SOC;

[0168] S441: In this embodiment, the temperature penalty term l cab,i The actual cabin temperature of vehicle i is required not to exceed the set temperature upper and lower boundaries. When the temperature exceeds the range, it will receive a negative reward related to the driving time, and its expression is:

[0169]

[0170] Among them, T cab,min ,T cab,maxare the minimum and maximum values ​​of cabin temperature, respectively, T is the total duration of the driving task, and t is the current time step;

[0171] S442: In this embodiment, the energy recovery during the vehicle braking process is not considered, and the battery SOC indicator c SOC,i It can be expressed as:

[0172]

[0173] Among them, ΔSOC i is the SOC change at the current moment;

[0174] S443: In this embodiment, the battery SOH indicator c SOH,i The capacity loss of the battery of vehicle i is required to be as small as possible, and its expression is:

[0175] c SOH,i =1-SOH i

[0176] S444: In this embodiment, the battery SOC penalty item l e,i It is required that the battery SOC of vehicle i cannot exceed the set upper and lower limits of the battery SOC. When the battery SOC exceeds the range, it will receive a negative reward related to the driving time, and its expression is:

[0177]

[0178] Among them, SOC min ,SOC max are the minimum and maximum values ​​of battery SOC respectively;

[0179] S45: Construct a rule-based power allocation model: Take the working mode output by the deep reinforcement learning agent as input, and use the set power allocation rules to complete the power distribution of the power system under the selected working mode; since the parking mode (0) has no energy consumption and the starting mode (1) only consumes a small amount of electricity, no power allocation rules are set in these two working modes.

[0180] S46: Constructing the network structure of the deep Q network algorithm: The deep Q network algorithm is an algorithm that combines deep learning and reinforcement learning. It has two neural networks with the same structure. The evaluation network is used to generate the Q value of all possible actions in the current state, and the target network is used to generate the target Q value.

[0181] S47: The experience replay pool of the deep Q network algorithm stores the experience data of the interaction between the intelligent agent and the energy management environment model. During model training, a certain amount of data is randomly extracted from the experience replay pool. At the same time, new data from the continuous interaction between the intelligent agent and the energy management environment model is also stored in the experience replay pool.

[0182] Step S5, training the energy-saving driving hierarchical control model based on deep reinforcement learning, specifically includes the following steps:

[0183] S51: Initialize the collaborative adaptive cruise control model based on the double-delayed deep deterministic policy gradient algorithm and the energy management model considering cabin temperature control based on the deep Q network, including the hyperparameters of each neural network and the experience replay pool;

[0184] S52: Collaborative Adaptive Cruise Control Model Based on Dual-Delay Deep Deterministic Policy Gradient Algorithm: Connecting the environment model with the algorithm model, generating interactive data during training (S i,t ,A i,t ,R i,t ,S i,t+1 ) is added to the experience replay pool D1;

[0185] At each time step, the gradient descent algorithm is used to minimize the loss function L to update the evaluation networks Critic1 and Critic2:

[0186]

[0187] Among them, y is the target value; are the evaluation value function and the target evaluation value function, θ i ,θ i ′ are the parameters of the evaluation network and the target evaluation network respectively; S i,t ,A i,t ,R i,t S is the vehicle’s current state, actions taken, and rewards obtained; i,t+1 is the state of the vehicle at the next moment; γ is the discount factor; π φ′ is the target policy function; ∈ is the action noise; For state S i,t+1 The sum of the actions taken according to the target policy function and the action noise ∈;

[0188] After evaluating the networks Critic1 and Critic2 for d steps, the policy network Actor is updated using deterministic policy gradients:

[0189]

[0190] Among them, φ is the parameter of the policy network.

[0191] Update the two target evaluation networks and the target policy network separately:

[0192] θ i ′=τθ i +(1-τ)θ i′,for i=1,2

[0193] φ′=τφ+(1-τ)φ′

[0194] Among them, τ is the soft update coefficient, φ′ is the parameter of the target policy network;

[0195] S53: Energy management model considering cabin temperature control based on deep Q network: connect the environmental model and algorithm model to generate interactive data during training Added to experience replay pool D2;

[0196] At each time step, the loss function L is minimized using the gradient descent method E To update the evaluation network:

[0197]

[0198] in, are the evaluation value function and the target evaluation value function, θ E,i ,θ E,i ′ are the parameters of the evaluation network and the target network respectively; The vehicle’s current state, actions taken, rewards obtained, and state at the next moment; Status possible actions to be taken;

[0199] The target network parameters are updated by hard updating, and the network parameters θ are evaluated every C steps. E,i Copy to the target network;

[0200] S54: Iteratively update the energy-saving driving hierarchical control model based on deep reinforcement learning: After the energy-saving driving hierarchical control model based on deep reinforcement learning converges, test the model. If it does not meet expectations, optimize the model's hyperparameters and reward function, and return to step S51.

[0201] In the above embodiments, the description's reference to "this embodiment" indicates that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment" do not necessarily all refer to the same embodiment.

[0202] In the above-described embodiments, although the invention has been described in conjunction with specific embodiments of the invention, many substitutions, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other storage structures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed. Embodiments of the invention are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of the appended claims.

[0203] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the methods in this embodiment is implemented.

[0204] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0205] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes any one of the methods in this embodiment.

[0206] The computer-readable storage medium in this embodiment can be understood by ordinary technicians in this field: all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0207] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used to communicate, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes each step of the above method.

[0208] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0209] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0210] The present invention can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0211] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0212] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning, characterized by: The following steps are involved: S1: Construct a vehicle platoon motion model based on the vehicle longitudinal dynamics model, and construct vehicle platoon motion constraints; S2: Use V2V communication technology to obtain vehicle operation data; S3: Build an upper-layer collaborative adaptive cruise control model based on a double-delayed deep deterministic policy gradient algorithm to provide vehicle speed data for the lower-layer energy management model; S4: Construct a lower-level energy management model based on deep Q network considering cabin temperature control; S5: Training the lower-layer energy management model.

2. The hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning according to claim 1 is characterized by: The vehicle longitudinal dynamics model is as follows: in, is the position of vehicle i, v i ,a i ,F a,i are the speed, driving force and acceleration of vehicle i, respectively, m i ,δ i are the mass and rotational mass conversion factor of vehicle i respectively.

3. The hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning according to claim 1 is characterized by: The vehicle queue motion constraints include spacing constraints, speed constraints, acceleration constraints, string stability constraints and headway constraints: Δs min,i ≤Δs i 0≤v i ≤v max,i a min,i ≤a i ≤a max,i Among them, Δs i is the distance between vehicle i and the preceding vehicle; Δs min,i is the minimum distance between vehicle i and the vehicle in front; v max,i is the maximum speed of vehicle i; a min,i ,a max,i are the minimum acceleration and maximum acceleration of vehicle i respectively; l i represents the string stability of vehicle i, and sets the l between the leading vehicle and its following vehicle i Similarly, the l of the rear vehicle and the vehicle in front of it i Same;d i ,d i-1 ,d i+1 h are the travel distances per unit time of vehicle i, the preceding vehicle i-1, and the following vehicle i+1 respectively; i is the headway; h min is the limit headway; s i ,s i+1 ,l v ,v i are the distance between vehicle i and the front of vehicle i+1, the vehicle length and the speed of vehicle i respectively.

4. The hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning according to claim 1 is characterized by: The step S2 of obtaining the vehicle operation data by using the V2V communication technology includes: The leading vehicle in the vehicle queue uses V2V technology to obtain the operating data of the pilot vehicle, including: the pilot vehicle's speed, acceleration, and travel distance per unit time; The other vehicles in the vehicle queue, except the head vehicle, use V2V technology to obtain the operating data of the front and rear vehicles, including: the speed of the front and rear vehicles and the driving distance per unit time; the tail vehicle in the vehicle queue obtains the operating data of the front vehicle.

5. The hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning according to claim 1 is characterized by: Step S3 specifically includes the following steps: S31: Create a vehicle platoon operation environment model: construct a leading vehicle and a vehicle platoon consisting of multiple following vehicles, each following vehicle in the vehicle platoon is a control vehicle, and the serial number is set to 1, 2, ..., n in sequence; the trajectory of the leading vehicle is set automatically, and the serial number of the leading vehicle is set to 0; set the initial distance between the head vehicle and the leading vehicle, the initial distance of the vehicle platoon, and the initial speed of the vehicle platoon; S32: Based on the motion simulation model of the vehicle platoon and the vehicle operation status, build the state space S1 of vehicle 1: S1={v0,a0,d i } Among them, v0 and a0 are the speed and acceleration of the pilot car respectively; Build the state space S of the remaining vehicles 2,...,n : S 2,...,n ={v i-1 ,v i ,d i ,h i ,l i } Among them, v i-1 is the speed of the preceding vehicle; S33: Building Action Space A i , whose action variables are extracted from the acceleration calculation formula, as follows: a i =-k x,i (s i -s i-1 +l v )-k v,i (v i -v i-1 ), A i ={k v,i |0≤k v,i ≤10} Among them, k x,i is the position error coefficient between vehicle i and the preceding vehicle, set as a constant; k v,i is the speed error coefficient between vehicle i and the preceding vehicle; S34: Construct reward function R i , the reward function R i Including the vehicle layer reward function R v,i and the dynamical system layer reward function R p,i Two parts, namely: R i =-R v,i -R p,i The vehicle layer reward function R v,i is the acceleration index r Acc,i , acceleration index r J,i , headway index r h,i And string stability index r l,i The weighted sum of is: R v,i ＝α 1,i r Acc,i +α 2,i r J,i +α 3,i r h,i +α 4,i r l,i Among them, α 1,i ,α 2,i ,α 3,i ,α 4,i They are acceleration index r Acc,i , acceleration index r J,i , headway index r h,i And string stability index r l,i The weight of the acceleration index r Acc,i The vehicle i is required to meet the acceleration constraint condition; the acceleration index r J,i The vehicle i is required to have a minimum jerk as possible to ensure comfort; the headway index r h,i It is required that vehicle i maintain a certain safe distance from the vehicle in front, and vehicle i has sufficient reaction time when the vehicle in front brakes; the string stability index r l,i It is required that the distance error between vehicle i and the vehicles in front and behind in the queue will not propagate along the queue and be amplified, and the distance errors between all vehicles in the platoon should maintain the same accuracy to avoid collisions; The dynamical system layer reward function R p,i is the fuel consumption index r f,i 、Battery SOC index r SOC,i and battery SOH index SOH,i The weighted sum of is: R p,i ＝β 1,i r f,i +β 2,i r SOC,i +β 3,i r SOH,i Among them, β 1,i ,β 2,i ,β 3,i They are respectively the fuel consumption index r f,i 、Battery SOC index r SOC,i and battery SOH index SOH,i The weight of the fuel consumption index r f,i The fuel consumption rate of vehicle i is required to be as low as possible from the maximum fuel consumption rate; wherein the battery SOC index r SOC,i The battery SOC of vehicle i is required to be kept within a certain range; wherein the battery SOH index r SOH,i The capacity loss of the battery of vehicle i is required to be as small as possible; S35: Construct the network structure of the double-delayed deep deterministic policy gradient algorithm: the policy network Actor determines the action taken in the current state, and the two evaluation networks Critic1 and Critic2 are used to evaluate the current state and action; the network structure of the target policy network Target-Actor, the target evaluation network Target-Critic1 and Target-Critic2 is the same as the corresponding main network structure, which is used to improve the stability of learning; S36: The experience replay pool of the double-delay deep deterministic policy gradient algorithm stores the experience data of the interaction between the intelligent agent and the vehicle queue operation environment model; during model training, a certain amount of data is randomly extracted from the experience replay pool, and new data of the continuous interaction between the intelligent agent and the vehicle queue operation environment model is also stored in the experience replay pool.

6. The hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning according to claim 1 is characterized by: Step S4 specifically includes the following steps: S41: Create an energy management environment model: construct a power system model and a cabin thermal management model for each vehicle in the vehicle queue; set the initial values ​​of the power system operating mode, battery SOC, battery SOH and cabin temperature; S42: Build the state space of vehicle i v fi =v i +a i Δt Among them, T cab,i ,e cab,i ,Δe cab,i are the cabin temperature of vehicle i, the error between the actual cabin temperature and the set temperature, and the error change rate; SOC i ,P d,i ,P cab,i are the battery SOC value, power system demand power and cabin temperature control system power of vehicle i respectively; v fi is the speed of vehicle i at the next moment; a i is the acceleration of vehicle i, provided by the upper-layer cooperative adaptive cruise control model; Δt is the time step; S43: Building an Action Space It consists of two parts: the control instructions of the cabin temperature control system and the working mode of the power system, as follows: Among them, a h,i ,a AC,i are the opening degrees of the heating system and air conditioning system of vehicle i, respectively, are discrete variables, and a m,i is the working mode number of the power system of vehicle i; the vehicle has five working modes, namely driving mode (-1): the engine is turned off, but the transmission keeps the engine idling; parking mode (0): the engine is turned off, the speed is zero, and the torque is zero; starting mode (1): the transmission starts the engine to idle speed, and no fuel is consumed; idle mode (2): the engine is in the on state but idling, and no torque is generated; starting mode (3): the engine is on, and fuel is supplied to the engine; S44: Constructing the reward function Reward Function Including cabin temperature reward function R cab,i and the power system energy consumption reward function R EMS,i Two parts, namely: The cabin temperature reward function R cab,i is the temperature error e cab,i The absolute value of the temperature error change rate Δe cab,i The absolute value of the temperature control system power P cab,i and the temperature penalty term l cab,i The weighted sum of is: in, They are the temperature errors e cab,i The absolute value of the temperature error change rate Δe cab,i The absolute value of the temperature control system power P cab,i and the temperature penalty term l cab,i The weight of the temperature penalty term l cab,i It is required that the actual cabin temperature of vehicle i cannot exceed the set upper and lower temperature limits; The power system energy consumption reward function R EMS,i Is the battery SOC indicator c SOC,i 、Fuel consumption indexc f,i And battery SOH index c SOH,i and battery SOC penalty term l e,i The weighted sum of is: in, They are the battery SOC index c SOC,i 、Fuel consumption indexc f,i And battery SOH index c SOH,i and battery SOC penalty term l e,i The weight of the battery SOC index c SOC,i The battery SOC change of vehicle i during battery discharge is required to be as small as possible, and the battery is required to be as stable as possible during battery charging. SOC,i =0; wherein the fuel consumption index c f,i is the fuel consumption rate of vehicle i; wherein the battery SOH index c SOH,i The capacity loss of the battery of vehicle i is required to be as small as possible; the battery SOC penalty term l e,i It is required that the battery SOC of vehicle i cannot exceed the set upper and lower limits of the battery SOC; S45: Construct a rule-based power allocation model: take the working mode output by the deep reinforcement learning agent as input, and use the set power allocation rule to complete the power allocation of the power system under the selected working mode; no power allocation rule is set for the parking mode (0) and the starting mode (1); S46: constructing a network structure of a deep Q network algorithm for generating Q values ​​of all possible actions in the current state, and a target network for generating a target Q value; S47: The experience replay pool of the deep Q network algorithm stores the experience data of the interaction between the intelligent agent and the energy management environment model. During model training, a certain amount of data is randomly extracted from the experience replay pool, and new data of the continuous interaction between the intelligent agent and the energy management environment model is also stored in the experience replay pool.

7. The hybrid electric vehicle platoon energy-saving driving hierarchical control method based on deep reinforcement learning according to claim 1 is characterized by: The training of the energy-saving driving hierarchical control model based on deep reinforcement learning in step S5 specifically includes the following steps: S51: Initialize the upper-layer collaborative adaptive cruise control model based on the double-delayed deep deterministic policy gradient algorithm and the lower-layer energy management model considering cabin temperature control based on the deep Q network, including the hyperparameters of each neural network and the experience replay pool; S52: Upper-layer collaborative adaptive cruise control model based on double-delayed deep deterministic policy gradient algorithm: Connect the environment model with the algorithm model, and generate interactive data during training (S i,t ,A i,t ,R i,t ,S i,t+1 ) is added to the experience replay pool D1; At each time step, the gradient descent algorithm is used to minimize the loss function L to update the evaluation networks Critic1 and Critic2: Among them, y is the target value; are the evaluation value function and the target evaluation value function, θ i ,θ i ′ are the parameters of the evaluation network and the target evaluation network respectively; S i,t ,A i,t ,R i,t S is the vehicle’s current state, actions taken, and rewards obtained; i,t+1 is the state of the vehicle at the next moment; γ is the discount factor; π φ′ is the target policy function; ∈ is the action noise; For state S i,t+1 The sum of the actions taken according to the target policy function and the action noise ∈; After evaluating the networks Critic1 and Critic2 for d steps, the policy network Actor is updated using deterministic policy gradients: Among them, φ is the parameter of the policy network; Update the two target evaluation networks and the target policy network separately: i i ′=τθ i +(1-τ)θ i ′,for i=1,2 φ′=τφ+(1-τ)φ′ Among them, τ is the soft update coefficient, φ′ is the parameter of the target policy network; S53: Lower-level energy management model considering cabin temperature control based on deep Q network: connect the environmental model and the algorithm model to generate interactive data during the training process Added to experience replay pool D2; At each time step, the loss function L is minimized using the gradient descent method E To update the evaluation network: in, are the evaluation value function and the target evaluation value function, θ E,i ,θ E,i ′ are the parameters of the evaluation network and the target network respectively; The vehicle’s current state, actions taken, rewards obtained, and state at the next moment; Status possible actions to be taken; The target network parameters are updated by hard updating, and the network parameters θ are evaluated every C steps. E,i Copy to the target network; S54: Iteratively update the energy-saving driving hierarchical control model based on deep reinforcement learning: After the energy-saving driving hierarchical control model based on deep reinforcement learning converges, test the model. If it does not meet expectations, optimize the model's hyperparameters and reward function, and return to step S51.

Images