Networked automobile ecological driving method based on deep reinforcement learning at non-signal-control intersection

Abstract

The invention relates to a non-signal-control intersection networked vehicle ecological driving method based on deep reinforcement learning. The method comprises the following steps that 1) a deep reinforcement learning algorithm is coupled with a microscopic traffic flow model to drive a strategy module to generate traffic flow features; the vehicle cooperative control collision avoidance strategy module adjusts the speed and path of the vehicle; the ecological driving strategy module calculates instantaneous energy consumption and carbon emission of the vehicle; 2) through input and feedback of the three modules, a deep reinforcement learning coupling model cooperative control module obtains three function modules of ecological energy saving, high efficiency and safety and driving comfort award; 3) forming a multi-stage multi-target reward function module; 4) the traffic scene module simulates a real traffic environment; and 5) calling data of the multi-stage multi-target reward function module and the traffic scene module by the main function module, and finally outputting a non-signal-control intersection automobile automatic driving strategy. Through the synergistic effect of all the modules, self-adaptive ecological driving can be realized in a complex dynamic traffic environment.

Description

[Technical Field]

[0001] This invention relates to an autonomous driving method, specifically a connected vehicle ecosystem driving method based on deep reinforcement learning at non-signal-controlled intersections, belonging to the field of autonomous driving technology. [Background Technology]

[0002] In current traffic simulation technologies, the control and simulation of intelligent connected vehicles have received widespread attention, especially for autonomous driving and cooperative control in complex traffic scenarios. Specifically, in complex traffic scenarios, guiding the eco-driving mode of intelligent connected vehicles through reward functions is of great significance for achieving energy conservation, emission reduction, and safe and efficient driving. However, current research largely focuses on optimizing traffic signal-controlled scenarios or applying traditional reinforcement learning methods; the design of reward functions for intersections without traffic signals is still in its early stages. Furthermore, existing eco-driving strategies still have shortcomings in comprehensively considering energy consumption, emissions, and driving efficiency, and cannot meet the multi-objective optimization needs of intelligent connected vehicles in dynamic traffic environments.

[0003] Chinese Invention Patent Application No. 202210845539.1 proposes a vehicle-road cooperative control system and method based on multi-agent federated reinforcement learning. This method achieves a vehicle-road cooperative control framework through the collaborative work of a roadside static processing module and a vehicle-side dynamic processing module. It utilizes the federated reinforcement learning algorithm FTD3 to connect the reinforcement learning module and the federated learning module to protect privacy and reduce communication overhead. Simultaneously, the RSU neural network participates in model aggregation but not in training, thereby improving the system's robustness and real-time performance. However, this method primarily focuses on information aggregation and privacy protection in vehicle-road cooperation, without deeply exploring energy consumption optimization for eco-driving and its adaptability at intersections without signal control. Therefore, its optimization capabilities for vehicle energy consumption and emissions are limited, making it difficult to achieve comprehensive eco-driving goals.

[0004] Chinese Invention Patent Application No. 202211334615.9 proposes an eco-driving method for new energy vehicles. This method is based on a multi-agent deep deterministic policy gradient algorithm (DDPG), combined with adaptive cruise control (ACC) and energy management strategy (EMS), to construct a multi-agent eco-driving framework. Through improved priority experience replay technology and heterogeneous agent decoupling, driving performance is optimized and energy efficiency is improved. However, this method lacks dynamic adjustment and optimization of the eco-driving reward function, and its adaptability to different types of vehicles in complex traffic scenarios is insufficient, especially in scenarios without signalized intersections, failing to effectively achieve adaptive trade-offs and optimization of multiple objectives.

[0005] Chinese Invention Patent Application No. 202310020281.6 proposes an eco-driving method for hybrid electric vehicles based on multi-objective deep reinforcement learning. This method utilizes the MODRL algorithm to construct an energy consumption optimization model for hybrid electric vehicles and achieves adaptive selection of multi-objective weights through conditional networks and reward weight sampling mechanisms, thus solving the problem of trade-offs in multi-objective optimization. However, the optimization scheme for hybrid electric vehicles proposed by this method has certain limitations in applications without signalized intersections, making it difficult to adapt to diverse traffic flow characteristics, especially in balancing driving efficiency and safety under different traffic conditions.

[0006] Chinese invention patent application No. 202310234762.7 relates to a path tracking control method based on deep reinforcement learning. This method solves the path tracking problem in an end-to-end manner by decoupling the vehicle dynamics model and combining it with the TD3 algorithm. It also designs a multi-objective reward function to enhance the model's generalization ability and adapt to different operating conditions. However, this method focuses solely on path tracking control and does not address the multi-objective optimization requirements of eco-driving, particularly in terms of comprehensive consideration of energy consumption, emissions, and traffic efficiency. Therefore, it cannot achieve eco-driving optimization for intelligent connected vehicles in complex environments.

[0007] Chinese invention patent application No. 202410078961.8 proposes a trajectory planning method for intelligent connected vehicles based on longitudinal and lateral movement. This method optimizes the vehicle's longitudinal energy consumption and lateral lane-changing comfort by using a pseudospectral method to transform the control problem into a nonlinear programming problem, ultimately generating an optimal eco-driving trajectory that comprehensively considers energy consumption and comfort. However, this method focuses primarily on vehicle trajectory planning, with limited consideration for the adaptive design of the reward function in eco-driving modes, and fails to fully consider the multi-objective optimization needs in complex environments such as intersections without signal control.

[0008] Chinese invention patent application No. 202410386663.5 discloses a traffic strategy for intelligent connected vehicles based on safety reinforcement learning. This method avoids violations of safety constraints during training through a safety layer error correction mechanism and transforms nonlinear traffic light constraints into time-varying linear state constraints, thereby effectively solving the reward sparsity problem and improving the safety and real-time performance of the strategy. However, this method is mainly optimized for signalized intersections and cannot handle complex traffic flow scenarios without signalized intersections. It is difficult to adapt to the changing characteristics of dynamic traffic flow and fails to effectively improve the ecological driving capabilities of vehicles in situations without signalized intersections.

[0009] Chinese invention patent application No. 202410443493.X proposes a carbon emission reduction method based on C-V2X vehicle-road cooperation. This method utilizes roadside units (RSUs) and vehicle information, combined with a deep reinforcement learning model to generate optimal driving actions, thereby optimizing carbon emission reduction and reducing CO2 emissions during vehicle operation while improving traffic efficiency. However, this method primarily focuses on carbon emission reduction targets and lacks comprehensive optimization for eco-driving modes at intersections without signal control, failing to achieve an effective balance between energy consumption, emissions, driving efficiency, and safety, thus limiting its applicability in complex traffic environments.

[0010] In summary, most of the published technical solutions are optimized for specific scenarios or functions, but there is a lack of technical solutions that combine ecological driving strategies with deep reinforcement learning in complex traffic scenarios without signalized intersections.

[0011] Therefore, in order to solve the above-mentioned technical problems, it is indeed necessary to provide an innovative connected vehicle ecological driving method based on deep reinforcement learning at non-signal-controlled intersections to overcome the defects in the existing technology. [Summary of the Invention]

[0012] The purpose of this invention is to provide a deep reinforcement learning-based ecological driving method for connected vehicles at non-signal-controlled intersections. This method combines an ecological driving mode designed based on a reward function with deep reinforcement learning and ecological driving strategies to solve the problem of balancing energy conservation, emission reduction, driving efficiency, and safety in complex traffic environments, and to provide a more comprehensive and efficient simulation method for intelligent connected vehicles.

[0013] To achieve the above objectives, the technical solution adopted by the present invention is as follows: a connected vehicle ecological driving method based on deep reinforcement learning for non-signal-controlled intersections, which adopts an intelligent connected vehicle ecological driving system. The system includes a deep reinforcement learning algorithm coupled with a micro-traffic flow model driving strategy module, a vehicle cooperative control collision avoidance strategy module, an ecological driving strategy module, a deep reinforcement learning coupled model cooperative control module, a list-based multi-objective reward value acquisition module, a multi-stage multi-objective reward function module, a traffic scenario module, and a main function module.

[0014] The driving method includes the following steps:

[0015] 1) The strategy module based on deep reinforcement learning algorithm coupled with micro traffic flow model generates accurate traffic flow features by simulating the dynamic interaction behavior between vehicles and combining speed difference model and headway; the vehicle cooperative control collision avoidance strategy module dynamically adjusts vehicle speed and path by using collision point prediction and collision time calculation; the eco-driving strategy module is based on VT-Micro model to calculate vehicle instantaneous energy consumption and carbon emissions and integrates them into the reward function of deep reinforcement learning algorithm to incentivize vehicles to optimize driving behavior and achieve energy conservation and emission reduction.

[0016] 2) Through the input and feedback of the above three modules, the deep reinforcement learning coupled model collaborative control module uses the DDPG algorithm to learn and dynamically adjust the vehicle's optimal decision-making strategy by continuously interacting with the environment, so as to obtain the ecological energy-saving reward function module, the high-efficiency safety reward function module, and the driving comfort reward function module.

[0017] 3) The multi-objective reward value acquisition module based on the list calls the ecological energy-saving reward function module, the high-efficiency safety reward function module, and the driving comfort reward function module to form a multi-stage multi-objective reward function module, which is used to comprehensively optimize the needs of energy consumption, safety and driving comfort;

[0018] 4) The traffic scenario module provides a simulation environment for a two-way six-lane intersection without signalized intersections, supports the simulation process of multiple driving modes, and fully simulates the real traffic environment;

[0019] 5) The main function module retrieves data from the multi-stage multi-objective reward function module and the traffic scenario module, and completes the system operation through two modes: Train and Test. The Train mode performs real-time training and deep reinforcement learning model optimization, while the Test mode verifies the optimization results and evaluates their effectiveness, ultimately outputting the autonomous driving strategy for intelligent connected vehicles at non-signal-controlled intersections.

[0020] The non-signal-controlled intersection connected vehicle eco-driving method based on deep reinforcement learning of the present invention is further described as follows: The specific method for constructing the eco-driving strategy by the eco-driving strategy module is as follows:

[0021] 1-1-1), Traffic environment information acquisition based on end-to-end technology: Intelligent connected vehicles perceive their own status and relative information with surrounding vehicles in real time;

[0022] 1-1-2), Energy consumption and emission assessment based on the VT-Micro model, wherein the VT-Micro model is as follows:

[0023]

[0024] In the formula: N is the total number of vehicles in the simulation scenario; This represents the average fuel consumption of each vehicle in the scenario, calculated based on the VT-Micro model; e i,fuel Let be the instantaneous fuel consumption (L·s) of vehicle i at time t. -1 ), Let t be the vehicle's CO2 emission rate (kg·s) at time t. -1 ), where δ1 and δ2 are the correlation coefficients between speed and fuel consumption, respectively, and are 3.5 × 10⁻⁶. -8 kg·h·km -1 ·s -1 and 2.39 kg·L -1 v and a represent the vehicle speed (m·s) respectively. -1 ) and acceleration (m·s -2 ), m, n represent the polynomial order; the ecological driving strategy parameters K are obtained by fitting the parameters of the VT-Micro model through real vehicle simulation. m,n .

[0025] The non-signal-controlled intersection connected vehicle ecological driving method based on deep reinforcement learning of the present invention further includes: introducing relative collision time (RCT) and vehicle distance (S) into the vehicle cooperative control collision avoidance strategy module. d As a key indicator, the core calculation function of the vehicle cooperative control collision avoidance strategy is as follows:

[0026]

[0027] In the formula: L(·) is the function for calculating the distance to the nearest collision point, x i For vehicles entering or entering the intersection; x′ i For vehicles on the path corresponding to the nearest collision point that did not pass through the collision point; v i ,v i ' represents the corresponding vehicle speed; S d x represents i With x i The Euclidean distance between the vehicles is used to determine the risk of a vehicle collision by calculating the relative collision time and the distance between the vehicles.

[0028] The non-signal-controlled intersection connected vehicle eco-driving method based on deep reinforcement learning of the present invention is further described as follows: The specific calculation method of the deep reinforcement learning coupled model collaborative control module is as follows:

[0029] 2-1), Initialize algorithm parameters: Initialize the actor network parameters θ in the deep reinforcement learning algorithm. μ and the network parameters θ of the critics Q Eco-driving strategy parameter K m,nThe optimal speed sensitivity coefficient 'a' and the speed difference sensitivity coefficient 'λ' between vehicles in the micro-traffic flow model are determined; at the same time, the parameters L(·) of the cooperative control collision avoidance strategy are set.

[0030] 2-2), Iterative Optimization: In each iteration of the algorithm, the current vehicle state information of the system (s) is first obtained. t ,a t ,r t ,s t+1 This includes the vehicle's speed, acceleration, current position, and relative information with other vehicles;

[0031] 2-3) Processing state information for each vehicle: For each vehicle, first extract the current state, action, reward, and next state information from the state information. The data is stored in the experience pool; simultaneously, the vehicle's current energy consumption and emissions status are calculated based on the VT-Micro model, and its energy consumption and emissions performance is analyzed; subsequently, the reward value of the vehicle is calculated by combining the eco-driving strategy and the cooperative control strategy.

[0032] 2-4), Generating actual driving acceleration: Acceleration generated through deep reinforcement learning Acceleration calculated by micro-traffic flow models The actual driving acceleration of the vehicle is generated based on the coupled drive strategy. And store it in the experience pool for later optimization;

[0033] 2-5), Update vehicle status information: based on actual acceleration Calculate and update vehicle speed Location and the state information at the next moment At the same time, the state data is stored in the experience pool to support subsequent training;

[0034] 2-6) Update deep reinforcement learning parameters: After each iteration, the parameters of the deep reinforcement learning algorithm are updated using the stored data, thereby improving the model's adaptability and optimization ability.

[0035] 2-7), Evaluate model performance: Evaluate the vehicle in the simulation environment, analyze its energy consumption, carbon emissions, speed, acceleration and rate of change of acceleration, and record the collision rate and the number of vehicles colliding, so as to comprehensively measure the algorithm performance.

[0036] 2-8), Output optimization strategy: After the iteration process is completed, output the optimized vehicle driving strategy.

[0037] The non-signal-controlled intersection connected vehicle eco-driving method based on deep reinforcement learning of the present invention is further described as follows: In step 2), the reward function in the eco-energy-saving reward function module is as follows:

[0038]

[0039] In the formula: For the vehicle at its maximum speed v M and maximum acceleration a M Fuel consumption under the following conditions; N is the total number of vehicles in the environment; γ1 is the ecological energy saving coefficient.

[0040] The non-signal-controlled intersection connected vehicle ecological driving method based on deep reinforcement learning of the present invention is further described as follows: In step 2), the reward function in the efficient and safe reward function module is as follows:

[0041]

[0042] In the formula: T l and D l These are the spatial and temporal thresholds for safe and efficient vehicle driving, respectively. When the threshold is below T... l and D l When a vehicle is deemed to pose a collision risk, the reward function takes effect and penalizes the current behavior strategy; v ex The desired driving speed for the environment is less than the vehicle's maximum speed v. M The closer the speed is to the expected speed, the greater the reward function will give; β is the scaling factor; γ2, γ3, and γ4 are the efficiency and safety coefficients.

[0043] The non-signal-controlled intersection connected vehicle ecological driving method based on deep reinforcement learning of the present invention is further described as follows: In step 2), the reward function in the driving comfort reward function module is as follows:

[0044]

[0045] In the formula: jerk=da / dt≈Δa / Δt is the derivative of acceleration, a M and a m These represent the maximum and minimum accelerations under environmental constraints; γ5 is the driving comfort coefficient.

[0046] The non-signal-controlled intersection connected vehicle ecological driving method based on deep reinforcement learning of the present invention is further described in step 3):

[0047] 3-1) Iterate through each vehicle, process each vehicle in the scene one by one, analyze its driving path and state information, and prepare to calculate the corresponding reward value.

[0048] 3-2) Iterate through all lane information. For each vehicle, iterate through possible locations within the set lane range and check the vehicle's driving status on different paths.

[0049] 3-3) Analyze the lane status, further refine the lane information, and classify the different values ​​of the corresponding State so as to calculate the reward value in combination with the driving scenario;

[0050] 3-4) Determine the starting position of the lane and calculate the distance. If the lane where the vehicle is located is within the starting path range, extract its state at the starting position and calculate the feature value L(xi) of the lane. Then, determine the distance from the vehicle to the nearest vehicle at the intersection based on the relative position of the vehicle at the intersection, and calculate the collision risk and safety index of the path.

[0051] 3-5), Calculate the reward value: Based on the relative distance and speed difference between the vehicle and other vehicles in the intersection, calculate the multi-objective reward value R2 and store the result in the reward value list List;

[0052] 3-6) Determine the maximum reward value after traversal: After all traversals are completed, select the maximum value from the reward value list as the final reward value of the vehicle, which reflects the optimal reward strategy in its current state, i.e., R2 = max(List);

[0053] 3-7), Output reward value: The maximum reward value of each vehicle is used as the output to complete the acquisition of multi-objective reward value.

[0054] The non-signal-controlled intersection connected vehicle ecological driving method based on deep reinforcement learning of the present invention is further as follows: In step 3), when the vehicle is driving normally, the multi-stage multi-objective reward function module uses a list-based multi-objective reward function to obtain all vehicles that are closest to all collision points in the driving path of the current vehicle, calculates the corresponding reward function value for each vehicle and adds them to the list. After all calculations are completed, the maximum reward value is retrieved and the reward value calculation of the current vehicle is completed.

[0055] When a vehicle leaves an intersection or collides with another vehicle during its journey, a constant reward value is used for feedback. Ultimately, the reward functions for eco-friendly energy conservation, high efficiency and safety, and driving comfort are combined to obtain a multi-stage, multi-objective reward function:

[0056]

[0057] The non-signal-controlled intersection connected vehicle ecological driving method based on deep reinforcement learning of the present invention can also be: In step 4), the traffic scene module provides the following specific vehicle action control equation:

[0058]

[0059] In the formula: s t μ(s) represents the vehicle's state information at time t in the deep reinforcement learning algorithm. t |θ μ ) represents the strategy adopted by the actor network based on the state information within the intersection at time t regarding vehicle driving behavior; a t This indicates that the algorithm is in the current state s. t The policy acceleration under the following conditions; v t P represents the vehicle's speed at time t; t t represents the distance between the vehicle and the center of the intersection at time t; T represents the program control cycle.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] 1. The deep reinforcement learning-based connected vehicle ecological driving method for non-signal-controlled intersections of the present invention couples deep reinforcement learning based on reward function design with an ecological driving mode. Deep reinforcement learning learns the optimal behavior strategy through interaction with the environment, while the ecological driving strategy comprehensively optimizes multiple objectives such as energy consumption, emissions, traffic efficiency, and safety to ensure that the vehicle achieves adaptive driving behavior in complex traffic environments. By combining deep reinforcement learning with the ecological driving strategy, the driving efficiency and safety of intelligent connected vehicles at non-signal-controlled intersections are improved.

[0062] 2. This invention combines deep reinforcement learning with ecological driving strategies, making it particularly suitable for complex traffic scenarios without signalized intersections. Through the design of a multi-objective reward mechanism, it ensures that vehicles achieve a balance between energy consumption, emissions, safety, and traffic efficiency. This addresses the shortcomings of existing technologies that mainly focus on information aggregation and privacy protection in vehicle-road cooperation, without deeply considering the multi-objective optimization needs of ecological driving.

[0063] 3. This invention combines deep reinforcement learning with ecological driving strategies. Through the design of the reward function, it achieves efficient driving in signalless intersection scenarios, solving the shortcomings of existing technologies that mainly target energy management for new energy vehicles and lack multi-objective optimization and adaptability to different types of vehicles in complex signalless scenarios.

[0064] 4. This invention proposes a multi-objective reward mechanism for intersections without signal control, which comprehensively considers multiple objectives such as energy consumption, emissions, driving efficiency and safety. It solves the shortcomings of existing technologies, which, although they achieve multi-objective optimization of hybrid vehicles, mainly focus on car-following scenarios and are difficult to adapt to the complex traffic characteristics of intersections without signal control.

[0065] 5. This invention not only focuses on optimal decision-making for path tracking, but also optimizes eco-driving performance through a multi-objective reward mechanism, including energy consumption and emission targets, thus solving the technical problem that existing technologies only address path tracking and fail to effectively solve energy conservation and eco-driving.

[0066] 6. This invention proposes a method for optimizing vehicle trajectories at signalless intersections through deep reinforcement learning and a multi-objective reward mechanism. This method significantly improves the overall performance in terms of energy consumption, emissions, and comfort, and solves the shortcomings of existing technologies that mainly focus on lane-changing comfort and fail to achieve an effective trade-off between multiple objectives.

[0067] 7. This invention is particularly applicable to intersections without signal control. By dynamically optimizing vehicle driving parameters through a multi-objective reward function, it effectively solves the shortcomings of existing technologies in scenarios without signal control, especially in terms of adaptability and safety in complex dynamic traffic flow environments.

[0068] 8. This invention introduces a multi-objective reward mechanism, combining deep reinforcement learning with carbon emission reduction strategies, enabling vehicles to maintain efficient and environmentally friendly driving performance in complex traffic environments. This solves the technical problem that existing technologies mainly focus on carbon emission reduction targets and fail to effectively combine ecological driving strategies for overall optimization of multiple objectives.

[0069] 9. This invention achieves multi-objective optimization of eco-driving through a reward mechanism, which can adaptively adjust according to real-time traffic conditions. This includes dynamically weighting multiple objectives such as energy consumption, emissions, driving efficiency, and safety to ensure optimal decision-making in different scenarios. By sensing changes in the traffic environment in real time, the reward mechanism can flexibly adjust the priority of each objective, thereby optimizing the vehicle's eco-driving performance at intersections without signalized traffic control, achieving efficient, safe, and environmentally friendly driving results. [Attached Image Description]

[0070] Figure 1 This is a general framework diagram of the non-signal-controlled intersection connected vehicle ecological driving method based on deep reinforcement learning according to the present invention.

[0071] Figure 2 This is a decision-making flowchart for the coupled microscopic traffic flow model-driven strategy module of the present invention.

[0072] Figure 3 This is a schematic diagram of a signalless intersection collision according to the present invention.

[0073] Figure 4 This invention provides a simulation scenario diagram of a two-way six-lane intersection without signalized intersections for the traffic scenario module.

[0074] Figure 5 This is a scenario diagram of the vehicle-to-intersection traffic control system according to the present invention.

[0075] Figure 6 This is a screenshot of the simulation training results viewed using Tensorboard in Train mode.

[0076] Figure 7a and Figure 7b This is a Matplotlib visualization of the invention in Train mode.

[0077] Figure 8 In Test mode, this invention includes simulation images of each step of the process.

[0078] Figure 9 This is a screenshot of a simulation video showing each step of the process in Test mode.

[0079] Figure 10a This is a schematic diagram illustrating the calculation of instantaneous fuel consumption using the VT-Micro model and carbon emission formula according to the present invention.

[0080] Figure 10b This is a schematic diagram illustrating the calculation of carbon dioxide emission rate using the VT-Micro model and carbon emission formula according to the present invention.

[0081] Figure 11a This is a graph showing the change in vehicle collision rate with the number of training steps according to the present invention.

[0082] Figure 11b This is a graph showing the change in the number of training steps for the cumulative number of vehicles involved in collisions according to the present invention.

[0083] Figure 12a This is a graph showing the change in the average speed of the vehicle as a function of the number of training steps.

[0084] Figure 12b This is a graph showing the change rate of vehicle acceleration as a function of the number of training steps.

Detailed Implementation Methods

[0085] Please refer to the instruction manual appendix. Figure 1 As shown, this invention is a method for connected vehicle ecological driving based on deep reinforcement learning at non-signal-controlled intersections. It adopts an intelligent connected vehicle ecological driving system, which consists of multiple modules, including a deep reinforcement learning algorithm coupled with a micro-traffic flow model driving strategy module, a vehicle cooperative control collision avoidance strategy module, an ecological driving strategy module, a deep reinforcement learning coupled model cooperative control module, a list-based multi-objective reward value acquisition module, a multi-stage multi-objective reward function module, a traffic scenario module, and a main function module.

[0086] Specifically, the non-signal-controlled intersection connected vehicle ecological driving method based on deep reinforcement learning of the present invention includes the following steps:

[0087] Step 1) The deep reinforcement learning algorithm coupled with the micro-traffic flow model drives the strategy module to generate more accurate traffic flow features by simulating the dynamic interaction behavior between vehicles and combining the speed difference model and the headway. The vehicle cooperative control collision avoidance strategy module uses collision point prediction and collision time calculation to dynamically adjust the vehicle speed and path. The eco-driving strategy module is based on the VT-Micro model to calculate the instantaneous energy consumption and carbon emissions of the vehicle and integrates them into the reward function of the deep reinforcement learning algorithm to incentivize the vehicle to optimize driving behavior and achieve energy conservation and emission reduction.

[0088] Among these, the eco-driving strategy is not only core to reducing carbon emissions and energy consumption, but also provides a benchmark for energy consumption and emission optimization for subsequent modules. By using the VT-Micro model to evaluate vehicle energy consumption and carbon emissions in real time, the eco-driving strategy constitutes one of the core objectives of the entire system and directly influences the design and optimization of the reward function. Simultaneously, external traffic information acquired by the eco-driving strategy, such as speed, acceleration, and location data, serves as input to provide decision support for the micro-traffic flow model.

[0089] To address the complex traffic environments at signalless intersections, an eco-driving strategy is embedded into the system through a reward mechanism based on a deep reinforcement learning algorithm. This strategy not only independently optimizes energy consumption but also provides dynamic flow characteristics to the micro-traffic flow model, laying the foundation for accurate prediction of collaborative collision avoidance strategies.

[0090] In this embodiment, the specific method for constructing the eco-driving strategy by the eco-driving strategy module is as follows:

[0091] 1-1-1), Traffic environment information acquisition based on end-to-end technology: Intelligent connected vehicles perceive their own status (such as speed, acceleration, and position) and relative information with surrounding vehicles (such as headway and relative speed) in real time. This information is acquired through vehicle sensors or V2X communication technology, providing a basis for decision-making on eco-driving strategies.

[0092] 1-1-2), to quantify vehicle energy consumption and emissions, the VT-Micro model is used to evaluate each vehicle in a real-time traffic environment. The VT-Micro model is as follows:

[0093]

[0094] In the formula: N is the total number of vehicles in the simulation scenario; This represents the average fuel consumption of each vehicle in the scenario, calculated based on the VT-Micro model; e i,fuel Let be the instantaneous fuel consumption (L·s) of vehicle i at time t. -1 ), Let t be the vehicle's CO2 emission rate (kg·s) at time t.-1 ), where δ1 and δ2 are the correlation coefficients between speed and fuel consumption, respectively, and are 3.5 × 10⁻⁶. -8 kg·h·km -1 ·s -1 and 2.39 kg·L -1 v and a represent the vehicle speed (m·s) respectively. -1 ) and acceleration (m·s -2 ), m, n represent the polynomial order; the ecological driving strategy parameters K are obtained by fitting the parameters of the VT-Micro model through real vehicle simulation. m,n As shown in Table 1 below.

[0095] Table 1

[0096]

[0097] Based on the traffic flow information provided by the eco-driving strategy, a deep reinforcement learning algorithm coupled with a micro-traffic flow model further characterizes the interaction forces between vehicles, simulating dynamic behavior in complex traffic environments. Real-time energy consumption and emission assessment data from the eco-driving strategy, combined with the speed, spacing, and other characteristics of the micro-traffic flow model, not only improves the characterization of the traffic environment but also provides multi-dimensional constraints for the state input of the deep reinforcement learning algorithm. This coupling mechanism effectively overcomes the "black box" nature of deep reinforcement learning algorithms in traffic scenarios, making vehicle decisions closer to real-world driving conditions.

[0098] This invention uses two improved typical microscopic traffic flow models and deep reinforcement learning algorithms for comparative experiments:

[0099] (1) No speed difference model (optimized speed model)

[0100] The optimized speed model, proposed by Bando et al., adjusts the vehicle speed based on the difference between the vehicle's current speed and the speed of the vehicle in front to achieve the optimal driving speed. The control equation is as follows:

[0101]

[0102] (2) Model with velocity difference (full velocity difference model)

[0103] The full speed difference model, proposed by Jiang et al., considers the impact of the speed difference between the front and rear vehicles on vehicle behavior, particularly the vehicle response during emergency braking and acceleration. The control equations focus on the dynamic adjustment of the speed difference, and are as follows:

[0104]

[0105] Since the optimized speed model and the full speed difference model still produce decelerations that do not conform to reality, and cannot avoid vehicle collisions during emergency braking, this invention combines the improvements made to the optimized speed model by adding a speed difference term, and proposes a speed difference model with a distance difference reciprocal term and a speed difference term, the governing equation of which is as follows:

[0106]

[0107] In the formula: Δv n =v n+1 -v n ,Δv n The speed difference between the vehicles in front and behind; Δx n =x n+1 -x n ,Δx n V[Δx] represents the distance between the front and rear of the vehicles; n (t)]=V1+V2tanh[C1(Δx n -l c [-C2], V(·) is the speed optimization function identified by Helbing and Tilch through measured data; a is the optimal speed sensitivity coefficient; λ is the sensitivity coefficient for the speed difference between the front and rear vehicles; k is the speed difference term coefficient; V1, V2, C1, C2 are the constant terms of the speed difference model; l c Let be the length of the car body; tanh(·) is the hyperbolic tangent function. When the distance between the front and rear of the vehicles is close, the speed difference term has a significant effect, while when the distance is far, the speed difference term has a weak effect.

[0108] Furthermore, the specific decision-making process of the strategy-driven module based on deep reinforcement learning algorithm coupled with microscopic traffic flow model is as follows: Figure 2 As shown, it includes the following steps:

[0109] 1-2-1), Initialize parameters, perform initialization for each intelligent connected vehicle, including initializing the actual vehicle drive acceleration. Acceleration generated by deep learning algorithms Acceleration generated by micro-traffic flow model Minimum acceleration a m Maximum acceleration a M .

[0110] (1-2-2) Acquire vehicle status and interact with the environment. For each vehicle, acquire its current status information, including driving lane, speed, acceleration, and relevant information about vehicles in front and behind. Then, input this status information into the system and interact with the environment. Calculate the vehicle's acceleration 'a' using a deep learning algorithm. i DRL And at the same time, another acceleration value is generated using a micro traffic flow model.

[0111] (1-2-3) Select and constrain acceleration, compare the acceleration generated by the deep learning algorithm with the acceleration generated by the micro traffic flow model, and select the larger value as the candidate actual driving acceleration. Subsequently, the candidate accelerations were limited to the range between the minimum and maximum accelerations.

[0112] 1-2-4), output the actual driving acceleration And its output is used as the result of the algorithm.

[0113] Furthermore, based on the dynamic environmental information provided by the microscopic traffic flow model, the vehicle cooperative control collision avoidance strategy further optimizes vehicle path and speed adjustments. The performance of the eco-driving strategy is particularly important in this module: vehicle energy consumption assessment and speed optimization affect path planning, ensuring that while reducing collision risk, low fuel consumption and carbon emissions are maintained. This module not only relies on traffic flow model predictions but also provides real-time feedback to the reward mechanism. The reward function, by combining indicators such as collision distance and collision time, collaborates with an efficient and safe reward function to adjust the strategy in real time. This achieves a dynamic balance between energy consumption optimization and safety, while also providing optimization directions for subsequent deep reinforcement learning algorithms.

[0114] A diagram illustrating collisions at intersections without signal control is shown below. Figure 3 As shown, collision types can be categorized into TT (straight-on collision with straight-on), LL (left-turn collision with left-turn collision), and LT (left-turn collision with straight-on collision) based on the direction of travel. This invention introduces relative collision time (RCT) and vehicle separation distance (S). d As a key indicator, the core calculation function of the vehicle cooperative control collision avoidance strategy is as follows:

[0115] S d =||x i -x′ i ||

[0116] In the formula: L(·) is the function for calculating the distance to the nearest collision point, x i For vehicles entering or entering the intersection (using a follow-the-car driving strategy on straight roads outside the intersection); x′ i For vehicles on the path corresponding to the nearest collision point that did not pass through the collision point; v i ,v i ' represents the corresponding vehicle speed; S d x represents i With x i The Euclidean distance between them; the risk of vehicle collision is determined by calculating the relative collision time and the distance between the vehicles, RCT and Sd The smaller the size, the greater the likelihood of a collision between vehicles, requiring proactive strategies to avoid such collisions.

[0117] Step 2): Through the input and feedback of the above three modules, the deep reinforcement learning coupled model collaborative control module uses the DDPG algorithm (Deep Deterministic Policy Gradient, a deep reinforcement learning algorithm for solving continuous action space problems) to learn and dynamically adjust the vehicle's optimal decision-making strategy by continuously interacting with the environment, so as to obtain the ecological energy-saving reward function module, the high-efficiency safety reward function module, and the driving comfort reward function module.

[0118] Specifically, by coupling the inputs and feedbacks of the micro-traffic flow model-driven strategy module, the vehicle cooperative control collision avoidance strategy module, and the eco-driving strategy module, the deep reinforcement learning algorithm integrates the objective of the eco-driving strategy, the dynamic characteristics of the micro-traffic flow model, and the safety constraints of cooperative control. In each iteration, the algorithm obtains a reward value input from the eco-driving strategy and updates the strategy network based on vehicle status (such as energy consumption, emissions, and collision risk). The specific implementation steps are as follows:

[0119] 2-1), Initialize algorithm parameters: Initialize the actor network parameters θ in the deep reinforcement learning algorithm. μ and the network parameters θ of the critics Q Eco-driving strategy parameter K m,n The optimal speed sensitivity coefficient 'a' and the speed difference sensitivity coefficient 'λ' between vehicles in the micro-traffic flow model are determined; at the same time, the parameters L(·) of the cooperative control collision avoidance strategy are set.

[0120] 2-2), Iterative Optimization: In each iteration of the algorithm, the current vehicle state information of the system (s) is first obtained. t ,a t ,r t ,s t+1 This includes information such as the vehicle's speed, acceleration, current location, and relative information with other vehicles.

[0121] 2-3) Processing state information for each vehicle: For each vehicle, first extract the current state, action, reward, and next state information from the state information. The data is stored in the experience pool; simultaneously, the vehicle's current energy consumption and emissions status are calculated based on the VT-Micro model, and its energy consumption and emissions performance is analyzed; subsequently, the reward value of the vehicle is calculated by combining the eco-driving strategy and the cooperative control strategy.

[0122] 2-4), Generating actual driving acceleration: Acceleration generated through deep reinforcement learning Acceleration calculated by micro-traffic flow models The actual driving acceleration of the vehicle is generated based on the coupled drive strategy. And store it in the experience pool for later optimization.

[0123] 2-5), Update vehicle status information: based on actual acceleration Calculate and update vehicle speed Location and the state information at the next moment At the same time, the state data is stored in the experience pool to support subsequent training.

[0124] 2-6) Update deep reinforcement learning parameters: After each iteration, the parameters of the deep reinforcement learning algorithm are updated using the stored data, thereby improving the model's adaptability and optimization ability.

[0125] 2-7) Evaluate model performance: Evaluate the vehicle in the simulation environment, analyze its energy consumption, carbon emissions, speed, acceleration and rate of change of acceleration, and record the collision rate and the number of colliding vehicles to comprehensively measure the algorithm performance.

[0126] 2-8), Output optimization strategy: After the iteration process is completed, output the optimized vehicle driving strategy to ensure that it can achieve the goals of energy saving, safety and efficient driving in complex traffic environment.

[0127] Furthermore, to reflect the importance of eco-driving, vehicle energy consumption and emissions are considered, and energy conservation and emission reduction are achieved by reducing vehicle fuel consumption and carbon emissions. Specifically, fuel consumption and carbon emissions can be calculated using the VT-Micro model's eco-energy conservation reward function module, whose reward function is as follows:

[0128]

[0129] In the formula: For the vehicle at its maximum speed v M and maximum acceleration a M Fuel consumption under the following conditions; N is the total number of vehicles in the environment; γ1 is the ecological energy saving coefficient.

[0130] As can be seen, this invention, under the framework of deep reinforcement learning, designs a reward function to achieve an ecological driving mode, introduces energy consumption, emissions, driving efficiency and safety as multiple objectives into the reward mechanism, constructs an intelligent driving strategy suitable for intersections without signal control, and adjusts the reward value based on the VT-Micro model to ensure optimal vehicle behavior in complex traffic scenarios.

[0131] Safety is one of the most critical factors in vehicle operation. To assess vehicle safety, relative collision time (RCT) and vehicle separation distance (S) are introduced. d As a key indicator, RCT measures the time it takes for two vehicles to collide while maintaining their current speeds; S d This is used to measure the Euclidean distance between two vehicles; the greater the distance, the lower the risk of collision. Simultaneously, to improve vehicle traffic efficiency, relative velocity (RV) is introduced to increase vehicle speed. The reward function in the efficient safety reward function module is as follows:

[0132]

[0133] In the formula: T l and D l These are the spatial and temporal thresholds for safe and efficient vehicle driving, respectively. When the threshold is below T... l and D l When a vehicle is deemed to pose a collision risk, the reward function takes effect and penalizes the current behavior strategy; v ex The desired driving speed for the environment is less than the vehicle's maximum speed v. M The closer the speed is to the expected speed, the greater the reward function will give; β is the scaling factor; γ2, γ3, and γ4 are the efficiency and safety coefficients.

[0134] In the driving comfort reward function module, jerk is a physical quantity describing the rate of change of acceleration, which is closely related to driving comfort; excessively large acceleration changes will reduce driving comfort. The reward function of the driving comfort reward function module is as follows:

[0135]

[0136] In the formula: jerk=da / dt≈Δa / Δt is the derivative of acceleration, a M and a m These represent the maximum and minimum accelerations under environmental constraints; γ5 is the driving comfort coefficient.

[0137] Step 3) The multi-objective reward value acquisition module based on the list calls the ecological energy-saving reward function module, the high-efficiency safety reward function module, and the driving comfort reward function module to form a multi-stage multi-objective reward function module, which is used to comprehensively optimize the requirements of energy consumption, safety and driving comfort.

[0138] Specifically, for different vehicle collision points, this invention proposes a list-based reward value acquisition module to construct a multi-objective reward function. Specifically, based on the collision diagram in the "Vehicle Cooperative Control Collision Avoidance Strategy," vehicle collisions at intersections can be divided into TT (collision between straight-going paths), LL (collision between left-turning paths), and LT (collision between left-turning paths and straight-going paths), with more specific definitions provided for each collision situation.

[0139] ①TT (Collision between straight paths)

[0140] Based on lane division, vehicle information states can be defined, and the corresponding lane where a collision occurred can be searched. State = [0], Start = [1,4,7,10], End = [[10,1,4,7],[4,7,10,1]]. The specific formula for calculating the distance to the collision point can be defined as:

[0141]

[0142] Specifically, l represents the width of the lane, which is determined by iterating through the Start and End lists, i.e., if x i Lane = Start[j],x i =End[j].

[0143] ②LL (Collapse between left-turn paths)

[0144] Based on lane division, vehicle information states can be defined, and the corresponding lane where a collision occurred can be searched. State = [1], Start = [0,3,6,9], End = [[3,6,9,0],[9,0,3,6]]. The specific formula for calculating the distance to the collision point can be defined as:

[0145] if State = [1]

[0146]

[0147] Specifically, l represents the width of the lane, which is determined by iterating through the Start and End lists, i.e., if x i Lane = Start[j],x i =End[j].

[0148] ③LT (Collapse between left-turn path and straight-ahead path)

[0149] Based on lane division, vehicle information states can be defined, and the corresponding lane where a collision occurred can be searched. State = [2], Start = [0,3,6,9], End = [[10,1,4,7],[7,10,1,4]]. The specific formula for calculating the distance to the collision point can be defined as:

[0150] if State = [2]

[0151]

[0152] Specifically, l represents the width of the lane, which is determined by iterating through the Start and End lists, i.e., if x i Lane = Start[j],x i =End[j].

[0153] By traversing all feasible lanes, identifying the corresponding collision points of vehicles, and using the core calculation function of the vehicle cooperative control collision avoidance strategy:

[0154] S d =||x i -x′ i ||

[0155] The specific RCT and S were calculated. d Then, substitute it into the "efficient and safe reward function module" to get the specific R2.

[0156] Furthermore, the algorithm process of the multi-objective reward value acquisition module is as follows:

[0157] 3-1) Iterate through each vehicle, process each vehicle in the scene one by one, analyze its driving path and status information, and prepare to calculate the corresponding reward value.

[0158] 3-2) Iterate through all lane information. For each vehicle, iterate through possible locations within the set lane range and check the vehicle's driving status on different paths.

[0159] 3-3) Analyze the lane status and further refine the lane information. Classify and process different values ​​of the corresponding State (such as straight, left turn, right turn) in order to calculate the reward value in combination with the driving scenario.

[0160] 3-4), determine the starting position of the lane and calculate the distance. If the lane where the vehicle is located is within the starting path range, extract its state at the starting position and calculate the feature value L(x) of that lane. iThen, by determining the relative position of the vehicle at the intersection, the distance from the vehicle to the nearest vehicle at the intersection is determined, and the collision risk and safety indicators of the path are calculated.

[0161] 3-5) Calculate the reward value: Based on the relative distance and speed difference between the vehicle and other vehicles in the intersection, calculate the multi-objective reward value R2 and store the result in the reward value list List.

[0162] 3-6) Determine the maximum reward value after traversal: After all traversals are completed, select the maximum value from the reward value list as the final reward value of the vehicle, which reflects the optimal reward strategy in its current state, i.e., R2 = max(List).

[0163] 3-7), Output reward value: The maximum reward value of each vehicle is used as the output to complete the acquisition of multi-objective reward value.

[0164] Furthermore, the multi-stage, multi-objective reward function module runs throughout the entire system, integrating the goals of ecological energy conservation, high efficiency and safety, and driving comfort. In the cooperative control module, the reward function provides guidance for the optimization of the deep reinforcement learning model through real-time assessment of collision risk and traffic efficiency. Simultaneously, the ecological energy conservation reward function utilizes the energy consumption assessment results of the VT-Micro model to guide the vehicle to optimize driving behavior in the direction of energy conservation and emission reduction.

[0165] The list-based multi-objective reward value acquisition module enables effective adjustment of reward values. Taking the collision distance calculated in the cooperative control collision avoidance strategy as an example, if the vehicle approaches the collision point, the reward function will increase the weight of safety and reduce acceleration changes, thereby avoiding sudden braking and collisions. Simultaneously, the energy consumption assessment data of the eco-driving strategy ensures that energy consumption is minimized during hazard avoidance, demonstrating the close collaboration between the modules.

[0166] The multi-stage, multi-objective reward function module shown uses a list-based multi-objective reward function when the vehicle is driving normally. It obtains all vehicles that are closest to all collision points in the driving path, calculates the corresponding reward function value for each vehicle, and adds them to the list. After all calculations are completed, it retrieves the maximum reward value and completes the reward value calculation for the current vehicle.

[0167] When a vehicle leaves an intersection or collides with another vehicle during its journey, a constant reward value is used for feedback. Ultimately, the reward functions for eco-friendly energy conservation, high efficiency and safety, and driving comfort are combined to obtain a multi-stage, multi-objective reward function:

[0168]

[0169] In summary, this invention uses a list-based multi-objective reward value acquisition module to calculate and record the reward values ​​of all vehicles corresponding to collision points, and finally selects the one with the largest reward value as the final reward to optimize the decision-making strategy of the deep reinforcement learning model. This module ensures that the system can find the optimal control behavior in complex traffic environments by comprehensively comparing the reward values ​​of different vehicles, thereby improving the system's adaptability and robustness.

[0170] Step 4), the traffic scenario module provides a simulation environment for a two-way six-lane intersection without signalized intersections (e.g., Figure 4 As shown, the simulation process supports multiple driving modes and fully simulates the real traffic environment.

[0171] Specifically, this invention sets up the following traffic scenario: a two-way six-lane intersection without signal control. Vehicles in the intersection scenario (e.g.) Figure 5 The diagram shows three autonomous driving paths: left turn, straight ahead, and right turn. Considering simulation complexity, a non-lane-changing car-following driving mode is adopted, where the vehicle maintains a fixed path and follows the vehicle in front through the intersection. The specific vehicle movement control equations are as follows:

[0172]

[0173] In the formula: s t μ(s) represents the vehicle's state information at time t in the deep reinforcement learning algorithm. t |θ μ ) represents the strategy adopted by the actor network based on the state information within the intersection at time t regarding vehicle driving behavior; a t This indicates that the algorithm is in the current state s. t The policy acceleration under the following conditions; v t P represents the vehicle's speed at time t; t t represents the distance between the vehicle and the center of the intersection at time t; T represents the program control cycle.

[0174] Step 5) The main function module retrieves data from the multi-stage multi-objective reward function module and the traffic scenario module, and completes the system operation through two modes: Train and Test. The Train mode performs real-time training and deep reinforcement learning model optimization, while the Test mode verifies the optimization results and evaluates their effectiveness, ultimately outputting the autonomous driving strategy for intelligent connected vehicles at non-signal-controlled intersections.

[0175] Specifically, the main function module integrates the functions of each module. During the training phase (Train mode), the deep reinforcement learning model continuously interacts with the environment to comprehensively optimize the energy consumption target, environmental characteristics of micro-traffic flow, and safety constraints of collaborative control for the ecological driving strategy. During the testing phase (Test mode), the system verifies the applicability and stability of the optimization strategy and outputs a comprehensive intelligent driving strategy that considers multiple objectives.

[0176] To facilitate system operation and modification of relevant default parameters, the system provides a program interface for modifying these parameters, allowing researchers to make changes and better adapt to independent project modules.

[0177] The Train mode primarily provides researchers with virtual training, assisting in simulating the behavior of intelligent connected vehicles at unsignalized intersections. In this mode, the program displays real-time simulation results on the terminal, and outputs specific operational parameters every 100 steps.

[0178] Each epoch consists of 6000 steps. After each epoch, system runtime information is written to x.cptk.index, x.cptk.meta, and x.cptk.data (where x represents the current epoch number). As the simulation system runs, the deep reinforcement learning model continuously moves towards the preset "zero collision rate" target. Finally, when the researcher is satisfied with the simulation results, the experiment can be stopped. The system will automatically search for the simulation information of the current epoch and record it in the three files best.cptk.index, best.cptk.meta, and best.cptk.data, facilitating the use of the dataset in subsequent test training modes.

[0179] When Train mode is enabled, Tensorboard can be used to view simulation training results in real time, such as... Figure 6 As shown, viewing simulation results using TensorBoard is more convenient than reading data through a terminal. After the simulation ends, the system provides Matplotlib visualization code for viewing the downloaded experimental data, which can be used for comparative studies of multiple simulation experiments. A visualization is provided here, such as... Figure 7a and Figure 7b As shown, the specific visualization effects and the number of comparison groups can be modified according to your needs.

[0180] The Test mode primarily provides virtual testing, which verifies whether the simulation data obtained from the Train mode can meet the researcher's needs, i.e., whether the goal of "zero collision rate" has been achieved.

[0181] Running the program in this mode will pop up a simulation image on the screen (it pops up by default and can be turned off). The simulation effect is displayed on the terminal, which is the simulation data calculated every 50 steps. In Test mode, a total of 1000 steps are run, and the total simulation data will be output in the end.

[0182] During operation, the system automatically captures information from the pop-up simulation images and stores them in the result_imgs file by default. This file contains images of each step of the process (such as...). Figure 8 As shown in the image, there is a complete simulation video (from step 0 to step 999) (as shown in the image). The file test.avi is the simulation video, and test is the default name.

[0183] Training experiment results and analysis:

[0184] To verify the superiority of the deep reinforcement learning-based connected vehicle eco-driving method for non-signaled intersections of this invention, experiments were conducted from three aspects: "eco-saving and energy-efficient," "high-efficiency and safe," and "collision avoidance." A comparative analysis was performed on the models driven solely by the DDPG algorithm and those driven by a coupled microscopic traffic flow model. In the following analysis, "DDPG," "no speed difference," and "with speed difference" refer to the DDPG algorithm, the model without speed difference, and the model with speed difference, respectively. Compared to pure DDPG driving and the model without speed difference driving, the model with speed difference showed the best performance in intelligent connected vehicle simulations at non-signaled intersections.

[0185] In terms of "ecological energy conservation," the VT-Micro model and carbon emission formula are used to calculate instantaneous fuel consumption based on speed and acceleration (e.g., ...). Figure 10a (as shown) and carbon dioxide emission rate (e.g. Figure 10b (As shown in the image). The results show that the model without speed difference has the most severe energy consumption, but it can pass through intersections at relatively fast driving speeds; the DDPG algorithm has the best energy consumption, but the slowest driving speed. In contrast, the model with speed difference has moderate driving speed and energy consumption, which is more in line with the concept of ecological energy conservation.

[0186] In terms of "efficiency and safety," the model without speed difference suffers from weak cooperative control strategies between vehicles due to high-speed travel, leading to frequent unrealistic collisions. While the DDPG algorithm avoids vehicle collisions to some extent through lower speeds, the model with speed difference interacts with the environment through more proactive cooperative control strategies, thus better avoiding collisions. Figure 11aAs can be seen, the collision rate of the speed difference model drops rapidly at the beginning, which effectively reflects the effect of the strategy in the early stage of operation. However, as the traffic flow increases, the previous driving strategy is no longer applicable, and the model adopts an active strategy to adjust, so that the overall collision rate gradually drops to zero. Figure 11b The superiority of the speed difference model is further demonstrated by the cumulative number of vehicles involved in collisions, especially when the number of training steps exceeds 3×10. 5 Subsequently, the cumulative number of collisions in the speed difference model almost stopped increasing, while the DDPG algorithm and the model without speed difference consistently showed a linear growth trend, with the growth rate of the model without speed difference exceeding that of the DDPG algorithm. This demonstrates the significant advantages of the speed difference model in dynamic adjustment and coordinated control, effectively achieving the goal of "high efficiency and safety".

[0187] Regarding "driving comfort", Figure 12a The results show that the no-speed-difference model and the speed-difference model rely on the coupled driving strategy to adopt an active driving strategy, while the DDPG algorithm adopts a low-speed driving strategy. The DDPG algorithm and the no-speed-difference model tend to approach the boundary in terms of driving speed, while the speed-difference model is more realistic. It adopts an active dynamic adjustment driving strategy around the desired speed, which helps to flexibly cope with constantly changing traffic conditions. Figure 12b It effectively demonstrates the comfort under different driving strategies. Compared with the high comfort of the DDPG algorithm, the speed difference model improves vehicle traffic efficiency by sacrificing some comfort, effectively reflecting the urgency of vehicles passing through unsignalized intersections, but still ensuring the "driving comfort" of the vehicle overall.

[0188] Test results and analysis:

[0189] Three algorithms were used to conduct 1000 rounds of simulation tests on intelligent connected vehicles at intersections without signalized intersections. The "eco-friendly and energy-saving," "high-efficiency and safe," and "driving comfort" performance under different algorithms were compared. The 111th, 333rd, and 4th points of the simulation were extracted from the scenario.

[0190] A comparative analysis of detailed simulation data from rounds 555, 777, and 999 is presented in Table 2.

[0191] Table 2 Results of different indicators in the test experiment

[0192]

[0193] The test results are similar to the training results. Regarding "eco-saving and energy-efficient" performance, the speed difference model, relying on a flexible driving strategy, significantly reduces fuel consumption and carbon emissions, decreasing by 40.38% and 40.39% respectively compared to the DDPG algorithm, demonstrating significant energy-saving and emission-reduction effects. Simultaneously, the model without speed differences also shows a reduction compared to the DDPG algorithm, indicating that high-speed driving strategies are more effective at saving energy and reducing emissions than low-speed driving strategies. The speed difference model interacts with the environment for cooperative driving, sacrificing some driving comfort but greatly ensuring vehicle driving safety, while the other two models still have unrealistic collision issues.

[0194] A detailed comparison of the results in Table 3 further illustrates the specific performance of different algorithms under various conditions. The speed difference model maintained a zero collision rate in all rounds, demonstrating the model's driving safety. In contrast, the DDPG and speed difference-free models, under high traffic conditions, effectively controlled energy consumption and carbon emissions at high speeds for intelligent connected vehicle ecosystem driving devices and systems based on deep reinforcement learning at non-signal-controlled intersections. Although the DDPG model had the lowest energy consumption and carbon emission rates, its driving speed was significantly lower than the other two models, impacting overall traffic efficiency.

[0195] Table 3 Comparison of Test Experiment Results

[0196]

[0197] The above-described specific embodiments are merely preferred embodiments of this invention and are not intended to limit this invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the protection scope of this invention.

Claims

1. A connected vehicle ecological driving method based on deep reinforcement learning at non-signal-controlled intersections, characterized by: It adopts an intelligent connected vehicle ecological driving system, which includes a driving strategy module based on a deep reinforcement learning algorithm coupled with a microscopic traffic flow model, a vehicle cooperative control collision avoidance strategy module, an ecological driving strategy module, a deep reinforcement learning coupled model cooperative control module, a list-based multi-objective reward value acquisition module, a multi-stage multi-objective reward function module, a traffic scenario module, and a main function module; The driving method comprises the following steps: 1) Based on the deep reinforcement learning algorithm coupled with the microscopic traffic flow model driving strategy module, the dynamic interaction behavior between vehicles is simulated, combined with the speed difference model and the headway, to generate accurate traffic flow characteristics; The vehicle cooperative control collision avoidance strategy module uses collision point prediction and collision time calculation to dynamically adjust the vehicle's speed and path; the ecological driving strategy module is based on the VT-Micro model, calculates the vehicle's instantaneous energy consumption and carbon emissions, and integrates them into the reward function of the deep reinforcement learning algorithm to encourage the vehicle to optimize driving behavior and achieve energy conservation and emission reduction; 2) Through the input and feedback of the above three modules, the deep reinforcement learning coupling model collaborative control module uses the DDPG algorithm to learn and dynamically adjust the optimal decision-making strategy of the vehicle by continuously interacting with the environment to obtain the ecological energy-saving reward function module, the efficient and safe reward function module, and the driving comfort reward function module; 3) The list-based multi-objective reward value acquisition module calls the ecological energy-saving reward function module, the efficient safety reward function module and the driving comfort reward function module to form a multi-stage multi-objective reward function module to comprehensively optimize the requirements of energy consumption, safety and driving comfort; 4) The traffic scene module provides a simulation environment for a two-way six-lane intersection without signal control, supports the simulation process of multiple driving modes, and fully simulates the real traffic environment; 5) The main function module retrieves the data of the multi-stage multi-objective reward function module and the traffic scenario module, and completes the system operation through the Train and Test modes: the Train mode performs real-time training and deep reinforcement learning model optimization, and the Test mode verifies the optimization results and evaluates their effectiveness, and finally outputs the automatic driving strategy of the intelligent connected vehicle at the non-signal-controlled intersection.

2. The connected vehicle ecological driving method based on deep reinforcement learning at a non-signal-controlled intersection as claimed in claim 2, characterized in that: The specific method of constructing the eco-driving strategy by the eco-driving strategy module is as follows: 1-1-1), traffic environment information acquisition based on end-to-end technology: intelligent connected vehicles perceive their own status and relative information with surrounding vehicles in real time; 1-1-2), energy consumption and emission assessment based on VT-Micro model, where the VT-Micro model is as follows: Where: N is the total number of vehicles in the simulation scene; represents the average fuel consumption of each vehicle in the scene, calculated based on the VT-Micro model; e i,fuel is the instantaneous fuel consumption of the i-th vehicle at time t (L·s -1 ), is the CO2 emission rate of the vehicle at time t (kg·s -1 ), δ1 and δ2 are the correlation coefficients of speed and fuel consumption, which are 3.5×10 -8 kg·h·km -1 ·s -1 and 2.39 kg·L -1 , v and a represent the vehicle speed (m·s -1 ) and acceleration (m·s -2 ), m, n represent the polynomial order respectively; the VT-Micro model parameters are fitted through real vehicle simulation to obtain the ecological driving strategy parameter K m,n .

3. The connected vehicle ecological driving method based on deep reinforcement learning at a non-signal-controlled intersection as claimed in claim 1, characterized in that: The vehicle cooperative control collision avoidance strategy module introduces the relative collision time (RCT) and the vehicle separation distance (S d ) is used as a key indicator, and the core calculation function of the vehicle cooperative control collision avoidance strategy is as follows: Where: L(·) is the distance calculation function of the nearest collision point, x i For vehicles that have entered or are entering the intersection; x i is the vehicle on the path corresponding to the nearest collision point that has not passed the collision point; v i ,v′ i is the corresponding vehicle speed; S d Represents x i and x′ i The Euclidean distance between the two vehicles; the risk of vehicle collision is determined by calculating the relative collision time and the distance between the vehicles.

4. The connected vehicle ecological driving method based on deep reinforcement learning at a non-signal-controlled intersection as claimed in claim 1, characterized in that: The specific calculation method of the deep reinforcement learning coupling model collaborative control module is: 2-1), Initialize algorithm parameters: Initialize the actor network parameters θ in the deep reinforcement learning algorithm μ and the critic network parameters θ Q , Eco-driving strategy parameter K m,n , the optimal speed sensitivity coefficient a and the front and rear vehicle speed difference sensitivity coefficient λ in the microscopic traffic flow model; at the same time, the cooperative control collision avoidance strategy parameter L(·) is set; 2-2), loop iteration optimization: In each iteration of the algorithm, first obtain the current vehicle status information of the system (s t ,a t ,r t ,s t+1 ), including the vehicle’s speed, acceleration, current position and relative information to other vehicles; 2-3), Processing the state information of each vehicle: For each vehicle, first extract the current state, action, reward, and next state information from the state information And store it in the experience pool; at the same time, calculate the current energy consumption and emission status of the vehicle based on the VT-Micro model, and analyze its energy consumption and emission performance; Then, the reward value of the vehicle is calculated by combining the ecological driving strategy with the cooperative control strategy. 2-4), Generate actual driving acceleration: acceleration generated by deep reinforcement learning and the acceleration calculated by the microscopic traffic flow model Generate the actual driving acceleration of the vehicle based on the coupled driving strategy And store it in the experience pool for subsequent optimization; 2-5), Update vehicle status information: based on actual acceleration Calculate and update the vehicle's speed Location and the status information of the next moment At the same time, the state data is stored in the experience pool to provide support for subsequent training; 2-6), Update deep reinforcement learning parameters: After each round of iteration, use the stored data to update the parameters of the deep reinforcement learning algorithm to improve the adaptability and optimization ability of the model; 2-7), Evaluate model performance: Evaluate the vehicle in a simulation environment, analyze its energy consumption, carbon emissions, speed, acceleration and acceleration change rate, and record the collision rate and number of collision vehicles to comprehensively measure the algorithm performance; 2-8), output optimization strategy: when the iteration process is completed, output the optimized vehicle driving strategy.

5. The connected vehicle ecological driving method based on deep reinforcement learning at a non-signal-controlled intersection as claimed in claim 1, characterized in that: In step 2), the reward function in the ecological energy-saving reward function module is as follows: Where: The vehicle is at maximum speed v M and the maximum acceleration a M The fuel consumption under the condition; N is the total number of vehicles in the environment; γ1 is the ecological energy saving coefficient.

6. The connected vehicle ecological driving method based on deep reinforcement learning at a non-signal-controlled intersection as claimed in claim 1, characterized in that: In step 2), the reward function in the efficient and safe reward function module is as follows: Where: T l and D l are the spatial threshold and time threshold for considering vehicle safe driving and efficient driving respectively. When it is lower than T l and D l When , the vehicle is judged to have a collision risk, the reward function takes effect and punishes the current behavior strategy; v ex is the expected driving speed of the environment, which is less than the maximum speed of the vehicle v M , the closer the speed is to the expected speed, the more rewards the reward function gives; β is the scaling factor; γ2, γ3, γ4 are efficient safety factors.

7. The connected vehicle ecological driving method based on deep reinforcement learning at a non-signal-controlled intersection as claimed in claim 1, characterized in that: In step 2), the reward function in the driving comfort reward function module is as follows: Where: jerk = da / dt ≈ Δa / Δt is the derivative of acceleration, a M and a m are the maximum acceleration and minimum acceleration under environmental constraints respectively; γ5 is the driving comfort coefficient.

8. The connected vehicle ecological driving method based on deep reinforcement learning at a non-signal-controlled intersection as claimed in claim 1, characterized in that: The step 3) is specifically as follows: 3-1), traverse each vehicle, process each vehicle in the scene one by one, analyze its driving path and status information, and prepare to calculate the corresponding reward value; 3-2), traverse all lane information, for each vehicle, traverse the possible positions within the set lane range, and check the driving conditions of the vehicle on different paths; 3-3), analyze the lane status, further refine the lane information, and classify the different values ​​of the corresponding state so as to calculate the reward value in combination with the driving scenario; 3-4), determine the starting position of the lane and calculate the distance. If the lane where the vehicle is located belongs to the starting path range, extract its state in the starting position and calculate the characteristic value L(xi) of the lane; then, determine the distance from the vehicle to the nearest vehicle at the intersection through the relative position of the vehicle at the intersection, and calculate the collision risk and safety index of the path; 3-5), calculate the reward value: based on the relative distance and speed difference between the vehicle and other vehicles in the intersection, calculate the multi-target reward value R2, and store the result in the reward value list List; 3-6), determine the maximum reward value after the traversal is completed: after all traversals are completed, select the maximum value from the reward value list as the final reward value of the vehicle, reflecting the optimal reward strategy under its current state, that is, R2 = max(List); 3-7), output reward value: take the maximum reward value of each vehicle as the output to complete the acquisition of multi-target reward values.

9. The connected vehicle ecological driving method based on deep reinforcement learning at a non-signal-controlled intersection as claimed in claim 1, characterized in that: In the step 3), when the vehicle is driving normally, the multi-stage multi-objective reward function module uses a list-based multi-objective reward function to obtain all vehicles closest to all collision points in the driving path of the current vehicle, calculate the corresponding reward function values ​​respectively and add them to the list. After all calculations are completed, the maximum reward value is retrieved to complete the reward value calculation of the current vehicle; When the vehicle leaves the intersection or collides during driving, a constant is used to feedback the reward value. Finally, the reward functions of ecological energy saving, high efficiency and safety, and driving comfort are summarized to obtain a multi-stage multi-objective reward function:

10. The connected vehicle ecological driving method based on deep reinforcement learning at a non-signal-controlled intersection as claimed in claim 1, characterized in that: In step 4), the traffic scene module controls the specific vehicle action equation as follows: Where: s t represents the state information of the vehicle at time t in the deep reinforcement learning algorithm; μ(s t |θ μ ) represents the strategy made by the actor network for vehicle driving behavior based on the state information in the intersection at time t; a t Indicates that the algorithm is in the current state s t Strategy acceleration under v t represents the vehicle speed at time t; P t It represents the distance between the vehicle and the center of the intersection at time t; T represents the program control cycle.

Images