Intelligent connected vehicle trajectory control method based on security reinforcement learning

Abstract

The application discloses an intelligent networked vehicle trajectory control method based on safety reinforcement learning, models a signal intersection scene as an undirected graph, takes a vehicle as a node and a communication relationship as an edge. A multi-agent control framework based on DDQN is constructed, a node dynamic spatial feature is extracted through a graph attention network as a state input, a CAV horizontal and longitudinal movement is taken as a joint action space, and a multi-objective reward function is designed. A main Q network outputs Q values of each action, combines a safety check mechanism to select an optimal action and implement control. The system stores a state, an action, a reward and a next state into an experience replay pool. A target Q network periodically samples data in the pool, optimizes training of main network parameters and improves stability and safety of a control strategy. The method realizes collaborative control of CAV horizontal and longitudinal trajectories, considers traffic efficiency and driving safety and is suitable for different networked penetration rate environments.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, more particularly to a method for trajectory control of intelligent connected vehicles based on safety reinforcement learning. BACKGROUND

[0002] With the development of advanced technologies such as artificial intelligence, 5G communication, and autonomous driving, intelligent connected vehicles (CAV) have shown great potential in alleviating traffic congestion, energy saving and emission reduction, and reducing the incidence of traffic accidents, and have become a research hotspot in the field of next-generation intelligent transportation systems (ITS). However, for a long time to come, intelligent connected mixed traffic flow composed of CAV and human-driven vehicles (HV) will exist for a long time. In intelligent connected mixed traffic flow, due to the complexity of space-time interaction, the uncertainty of human driving, and other factors, the trajectory control of CAV faces major challenges. Existing control algorithms mainly fall into two categories: one is based on optimization, and the other is based on learning. The optimization-based method mainly adopts mathematical modeling, which converts the trajectory decision-making task of CAV into an optimal solution problem under multiple constraints. However, this method faces the problems of high computational demand, difficulty in solving, and inability to adapt to low-latency dynamic decision-making. In contrast, the learning-based method represented by deep reinforcement learning (DRL) does not require complex modeling, learns human-like decision-making methods through repeated interaction with the environment, and has generalization and scalability.

[0003] The CAV trajectory control algorithm based on DRL, as shown in Figure 1 Generally follows the following research paradigm: 1) build a DRL simulation environment containing vehicles, traffic lights, and other traffic elements; 2) set a reasonable state space, control action, and reward function for the CAV trajectory control DRL agent; 3) train the CAV trajectory control DRL agent using gradient descent algorithm, and use the trained DRL agent to control the driving trajectory of CAV to improve traffic efficiency.

[0004] However, there are still many problems in the above prior art:

[0005] (1) The existing CAV trajectory control algorithm based on reinforcement learning mostly uses single-agent reinforcement learning method, lacking research on multi-CAV cooperative control in mixed traffic flow. Due to the lack of information exchange between agents, the single-agent method will make CAV focus too much on local optimal goal and ignore the impact of its decision on environment vehicles, reducing the traffic efficiency and safety of traffic flow.

[0006] (2) The existing trajectory control algorithm relies on the local information such as the state of the adjacent vehicle collected by V2V communication to make CAV driving decision, ignores the global traffic information provided by the infrastructure, and leads to limited communication range, incomplete perception and control conflict.

[0007] (3) The existing CAV trajectory control algorithm under the intelligent network environment only optimizes the longitudinal trajectory or the lateral trajectory of the vehicle, and is difficult to cope with the complex dynamic actual traffic environment.

[0008] Therefore, there is an urgent need for an intelligent network vehicle trajectory control method capable of improving the modeling expression ability and decision accuracy of the CAV trajectory control agent in a complex traffic scenario. SUMMARY

[0009] Therefore, the purpose of the present application is to provide an intelligent network vehicle trajectory control method based on safety reinforcement learning, which improves the modeling expression ability and decision accuracy of the CAV trajectory control agent in a complex traffic scenario, and builds an end-to-end solution from state perception, experience learning to real-time control.

[0010] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0011] The present application first provides an intelligent network vehicle trajectory control method based on safety reinforcement learning, comprising the following steps:

[0012] S1: modeling the intelligent network signal intersection scene including CAV vehicles and HV vehicles as an undirected graph, the CAV vehicles and the HV vehicles being nodes of the undirected graph, and the communication relationship between the vehicles being edges of the undirected graph;

[0013] S2: constructing a CAV trajectory control agent based on a DDQN network, including a main Q network and a target Q network with the same network architecture; extracting the dynamic spatial features between each node of the CAV and the HV in the undirected graph to obtain a node feature vector as the agent state space; taking the lateral / longitudinal driving decision actions of the CAV as the joint action space of the agent; and constructing a multi-objective reward function according to the global traffic information under the intelligent network signal intersection scene;

[0014] S3: the main Q network extracts the agent state space data at the current time, outputs the Q value of all lateral / longitudinal driving decision actions of the CAV under the current state, and selects the optimal joint action combined with the safety check mechanism, which is used for synchronous control of the CAV lateral / longitudinal decision actions;

[0015] S4: Obtain the reward of the CAV horizontal / longitudinal decision action at the current time according to the multi-objective reward function, and combine the agent state space data at the current time, the optimal joint action, and the next time agent state space data fed back by the intelligent connected signal intersection scene as sample data sets to be stored in the shared experience replay pool together;

[0016] S5: The target Q network periodically collects sample data sets from the shared experience replay pool to train and optimize the main Q network parameters. The optimized main Q network is used for real-time execution of S3 to realize real-time synchronous control of the CAV horizontal / longitudinal decision action.

[0017] Preferably, the undirected graph in S1 comprises a node feature matrix , an adjacency matrix , and a mask matrix ; wherein

[0018] At time t, the node feature matrix is represented as:

[0019] ;

[0020] Wherein, n represents the total number of vehicles of the CAV and the HV at the current time; represents the node feature matrix of the ith vehicle at time t. For the CAV vehicle, includes:

[0021] ;

[0022] For the HV vehicle, includes:

[0023] ;

[0024] Wherein, represents the ratio of the longitudinal speed of the vehicle to the maximum speed limit of the road; represents the ratio of the driving distance of the vehicle to the length of the road; represents the encoding of the lane where the vehicle is located; represents the state of the front traffic signal at the current time; represents the remaining green light time when the front traffic signal is green, or the waiting time required for the next green light phase to arrive when the front traffic signal is red or yellow; represents the ratio of the distance of the vehicle from the front signal intersection to the length of the lane; represents the ratio of the queuing length of the signal intersection to the total number of vehicles in the scene;

[0025] The adjacency matrix The information exchange relationship between vehicles is represented by different encoding, wherein the adjacency matrix of any two CAV vehicles The communication link relationship coding is given under the condition of establishing at any time, and the adjacency matrix of the CAV vehicle and the HV vehicle The communication link relationship coding is given under the condition of establishing under the condition constraint.

[0026] Mask matrix The HV vehicle lateral / longitudinal driving decision action output is used to filter the HV vehicle lateral / longitudinal driving decision action output in all vehicle lateral / longitudinal driving decision actions generated by the CAV trajectory control agent.

[0027] Preferably, the agent joint action space includes: CAV longitudinal acceleration action space and CAV lateral lane changing action space.

[0028] Preferably, the multi-objective reward function includes:

[0029] The speed reward function, the safety check mechanism reward function, the queue length reward function and the travel time reward function.

[0030] Preferably, the network architecture of the main Q network and the target Q network includes:

[0031] The node feature matrix of the ith vehicle at time t Through two fully connected layers, the first layer feature vector is obtained by adopting the RELU activation function for nonlinear mapping And the second layer feature vector ;

[0032] The second layer feature vector is sent into the graph attention network GAT, and the spatial interaction relationship between the vehicle i and the adjacent vehicles is extracted by adopting the multi-head attention mechanism, and the node feature vector of the ith vehicle at time t is obtained by adopting the residual connection and layer normalization , and the action strategy of the ith vehicle is generated ;

[0033] If the ith vehicle is a CAV vehicle, the mask matrix is adopted Output all actions of the CAV vehicle at time t, that is, the Q value corresponding to the action; if the ith vehicle is a HV vehicle, the mask matrix is adopted The action output corresponding to the HV vehicle is shielded.

[0034] Preferably, the CAV trajectory control agent includes the step of training and optimizing the parameters of the main Q network by the target Q network, and in the step:

[0035] The greedy strategy is adopted to select all lateral / longitudinal driving decision actions of the CAV under the current state, so that The random action is selected with a probability of ​the action with the maximum probability selection Q value is selected as the trajectory control strategy of the CAV:

[0036] ;

[0037] wherein, is the action policy of the i-th vehicle generated by the main Q network, is the action policy of the i-th vehicle generated by the main Q network,

[0038] Preferably, the CAV trajectory control agent comprises the step of training and optimizing the main Q network parameters using the target Q network, wherein in the step:

[0039] The optimization objective function of the Q network parameters is as follows:

[0040] ;

[0041] wherein, θ is the main Q network parameter, is the state of the agent state space at the current time , and is the estimated Q value of the action at the next time, and

[0042] ;

[0043] wherein, is the reward of the CAV horizontal / longitudinal decision action at the current time, is the discount factor, is the parameter of the target Q network, is the state of the corresponding CAV vehicle in the agent state space at the next time, is the action of the corresponding CAV vehicle at the next time, which is calculated by the main Q network at the state at the next time, i.e. , and then the action is input into the target Q network to obtain as the Q value estimation of the target Q network for the action at the state .

[0044] Preferably, the safety checking mechanism comprises:

[0045] a longitudinal action checking mechanism for checking whether the CAV vehicle in the current lane meets the safety acceleration requirement, and if not, calculating the safety acceleration and replacing the corresponding longitudinal driving decision action;

[0046] A lateral action checking mechanism is configured to check whether the distance between the CAV vehicle and all vehicles in the target lane meets the safety distance requirement when the CAV vehicle drives from the current lane to the target lane, and to shield corresponding lateral driving decision actions if not.

[0047] Compared with the prior art, the intelligent connected vehicle trajectory control method based on safety reinforcement learning provided by the application has the following beneficial effects:

[0048] The application adopts a centralized training and distributed execution mechanism, and in the training stage, the CAV agent is stimulated by a shared policy network and global state to learn how to maximize global traffic efficiency through collaborative decision-making.

[0049] Unlike the CAV trajectory control method that simply relies on V2V to obtain local state, the application comprehensively utilizes V2V and V2I to obtain global information of intelligent connected hybrid traffic, and fully integrates the state of neighboring vehicles, the global vehicle state, and the phase and remaining time of the signal intersection. The application models the intelligent connected hybrid traffic flow environment as an undirected graph, extracts the dynamic spatial correlation features between CAV and HV nodes by using a graph attention mechanism, improves the modeling expression ability and decision accuracy of the CAV trajectory control agent in a complex traffic scenario, and constructs an end-to-end solution from state perception, experience learning to real-time control.

[0050] The application proposes a CAV lateral and longitudinal trajectory joint control method. The existing method only studies CAV longitudinal acceleration and deceleration or lateral lane changing decision, and the improvement effect on intelligent connected hybrid traffic flow passing efficiency and energy saving and emission reduction is very limited. The CAV lateral and longitudinal trajectory joint control method proposed by the application can prevent traffic congestion in advance by planning the longitudinal driving speed and lateral lane changing behavior in advance.

[0051] The application proposes a safety checking mechanism for CAV trajectory decision. The existing CAV trajectory control method based on reinforcement learning mainly guides the agent to learn a safe strategy by punishing dangerous behavior, for example, giving a negative punishment when a vehicle collision occurs, which inevitably causes unsafe behavior of the agent during training and testing. Such unsafe CAV trajectory control strategy may damage hardware and even cause harm to humans. The safety checking mechanism checks the lateral action and longitudinal action of the vehicle before the CAV execution agent outputs the action, shields dangerous behavior and replaces it with a safe driving strategy, ensures the safety of CAV training and evaluation, and guides the vehicle to learn a safe driving strategy in the training stage through a safety reward function.

[0052] The present application provides a CAV trajectory control method suitable for signalized intersection scenarios. Existing control methods mainly solve the CAV trajectory control problem in traffic scenarios such as vehicle following, ramp merging and splitting at non-signalized intersections, and do not involve the influence of signal timing on CAV trajectory. The present application transmits the state and remaining time of the traffic signal to the CAV through V2I technology, so that the CAV can plan the driving speed and lane changing decision in advance according to the perceived signal timing information, so that the CAV can quickly and smoothly pass through the signalized intersection during the green light period, and can slowly reach the stop line to wait during the red light period, thereby reducing the propagation speed and influence range of traffic congestion.

[0053] The present application provides an eco-driving strategy suitable for different CAV penetration scenarios in an intelligent connected environment. The reward function is designed by comprehensively considering the traffic efficiency and ecological environmental protection, which can guide the CAV to pass through the signalized intersection in a more energy-saving and more emission-reducing driving behavior, and has good environmental adaptability and robustness. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0055] Figure 1 The principle diagram of the intelligent connected vehicle trajectory control method based on safety reinforcement learning provided by the present application;

[0056] Figure 2 The principle diagram of the CAV trajectory control agent based on graph attention provided by the present application;

[0057] Figure 3 The experimental scene diagram provided by the present application;

[0058] Figure 4 The reward change diagram of CAV lateral / longitudinal decision action provided by the present application;

[0059] Figure 5 is a greenhouse gas emission diagram provided by the present application under 50% CAV penetration rate;

[0060] Figure 6 is a mixed vehicle space-time trajectory diagram of different CAV penetration rates provided by the present application. DETAILED DESCRIPTION

[0061] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of the present application.

[0062] The embodiments of the present application disclose a smart connected vehicle trajectory control method based on safety reinforcement learning, comprising the following steps:

[0063] S1: modeling the smart connected signal intersection scene including CAV vehicles and HV vehicles as a global undirected graph, the CAV vehicles and the HV vehicles being nodes of the global undirected graph, and the communication relationship between the vehicles being edges of the undirected graph;

[0064] S2: constructing a CAV trajectory control agent based on a DDQN network, including a main Q network and a target Q network with the same network architecture; extracting dynamic spatial features between each node of the CAV and the HV in the global undirected graph to obtain a node feature vector as an agent state space; taking the lateral / longitudinal driving decision actions of the CAV as a joint action space of the agent; and constructing a multi-objective reward function according to the global traffic information under the smart connected signal intersection scene;

[0065] S3: the main Q network extracts the agent state space data at the current time, outputs the Q values of all lateral / longitudinal driving decision actions of the CAV under the current state, and selects the optimal joint action in combination with a safety check mechanism, which is used for synchronous control of the lateral / longitudinal decision actions of the CAV;

[0066] S4: obtaining the reward of the lateral / longitudinal decision actions of the CAV at the current time according to the multi-objective reward function, and storing the agent state space data at the current time, the optimal joint action, and the next time agent state space data fed back by the smart connected signal intersection scene as sample data sets to a shared experience replay pool;

[0067] S5: the target Q network periodically collects sample data sets from the shared experience replay pool to train and optimize the main Q network parameters, trains the CAV trajectory control agent through an Adam algorithm, and uses the optimized main Q network to execute S3 in real time, so as to realize real-time synchronous control of the lateral / longitudinal decision actions of the CAV.

[0068] Figure 1A schematic diagram of a smart connected vehicle trajectory control method based on security reinforcement learning is shown. The embodiment adopts a centralized training and distributed execution architecture. In the training phase, a unified shared policy network is used to realize cooperative control between CAVs. In the execution phase, each CAV makes decisions based on its own local state, which saves computing overhead, improves training efficiency, and improves the traffic efficiency and safety of mixed traffic flow.

[0069] In some embodiments, the global undirected graph in S1 includes: a node feature matrix , an adjacency matrix , a mask matrix ; wherein,

[0070] At time t, the node feature matrix is expressed as:

[0071] ;

[0072] Wherein, n represents the total number of CAVs and HVs at the current time; represents the node feature matrix of the ith vehicle at time t, and for CAV vehicles, includes:

[0073] ;

[0074] For HV vehicles, since HV vehicles do not have V2I communication function, includes:

[0075] ;

[0076] Wherein, represents the ratio of the longitudinal speed of the vehicle to the maximum speed limit of the road; represents the ratio of the vehicle travel distance to the road length; represents the encoding of the lane where the vehicle is located, represents that the vehicle is located in the left lane, represents that the vehicle is located in the right lane; represents the state of the front traffic signal at the current time, represents that the signal light is red or yellow, and the vehicle cannot pass through, represents that the signal light is green, and the vehicle can pass through; represents the remaining green light time when the front traffic signal is green, or the waiting time required for the next green light phase to arrive when the front traffic signal is red or yellow; represents the ratio of the distance between the vehicle and the front signal intersection to the length of the lane; represents the ratio of the queuing length of the signal intersection to the total number of vehicles in the scene;

[0077] In the node feature matrix of the CAV vehicle, the traffic signal phase and timing information obtained by using V2I communication is used to guide the CAV to plan the speed in advance, to reasonably select the target lane, to realize the space-time coordination of the vehicle trajectory and the traffic signal state, to slow down the traffic congestion, and to reduce the traffic delay.

[0078] Adjacent matrix The information exchange relationship between vehicles is represented by different encodings, wherein the adjacent matrix of any two CAV vehicles The communication link relationship encoding is given at any time, and the adjacent matrix of the CAV vehicle and the HV vehicle The communication link relationship encoding is given under the condition constraint. In one specific embodiment, the adjacent matrix is defined as follows:

[0079] ;

[0080] wherein, indicates that the i-th vehicle and the j-th vehicle can exchange information at the current time; indicates that the i-th vehicle and the j-th vehicle cannot exchange information at the current time. Any two CAVs can establish a communication link at any time, and CAV and HV can establish a communication channel only when the distance is 50 meters.

[0081] Mask matrix is used to filter the HV vehicle lateral / longitudinal driving decision action output in all vehicle lateral / longitudinal driving decision actions generated by the CAV trajectory control agent. In one specific embodiment, the mask matrix is defined as follows:

[0082] ;

[0083] wherein, indicates that the corresponding vehicle type is CAV, indicates that the corresponding vehicle type is HV.

[0084] In some embodiments, the agent joint action space includes: a CAV longitudinal acceleration action space and a CAV lateral lane changing action space. By constructing the joint action space, the flexibility of CAV action is enhanced, and the problem of CAV integrated longitudinal and lateral decision-making in the intelligent networked mixed traffic scene is solved.

[0085] In one specific embodiment, the longitudinal acceleration of the CAV is discretely processed to obtain the action space, which is defined as follows:

[0086] ;

[0087] wherein, indicates the maximum deceleration; This indicates the maximum acceleration.

[0088] The lateral lane change of CAV is also discretized to obtain the motion space, which is defined as follows:

[0089] ;

[0090] Where -1 indicates that the CAV changes lanes to the left; 1 indicates that the CAV changes lanes to the right; and 0 indicates that the CAV stays in the current lane.

[0091] In some embodiments, the multi-objective reward function includes:

[0092] Speed ​​reward function, security check mechanism reward function, queue length reward function, and passage time reward function.

[0093] In one specific embodiment, to encourage CAVs to pass through the signalized intersection as quickly as possible, the speed reward function is set as follows:

[0094] ;

[0095] in, Indicates the minimum speed of the CAV; This indicates the road speed limit.

[0096] In one specific embodiment, to encourage the CAV agent to learn safe driving strategies during training, the reward function is set as follows, in conjunction with a safety check mechanism:

[0097] ;

[0098] Specifically, if the vertical action output by the agent is deemed unsafe by the safety check mechanism, If the agent's lateral movement is deemed unsafe by the safety check mechanism, .

[0099] In one specific embodiment, to encourage CAVs to minimize queue lengths in intelligent connected mixed traffic flows through coordination with signal timing, the reward function is defined as follows:

[0100] ;

[0101] in, This represents the average queue length at the signalized intersection.

[0102] In one specific embodiment, in order to reduce the travel time of CAVs at signalized intersections, the reward function is defined as follows:

[0103] ;

[0104] wherein, represents the travel time of the ith CAV from entering the signalized intersection segment to the current time.

[0105] In one specific embodiment, in combination with the above specific embodiments, the total reward is:

[0106] ;

[0107] wherein, is the weight of each sub-reward function.

[0108] By incorporating the traffic efficiency index and the energy saving and environmental protection index into the design of the reward function, the CAV learns a smoother and energy-saving driving strategy, meeting the dual optimization demand of fast traffic and energy saving and environmental protection under different penetration rates.

[0109] In some embodiments, the network architecture of the main Q network and the target Q network adopts a graph attention-based CAV trajectory control end-to-end decision network, as shown in Figure 2 , including:

[0110] The original node feature matrix of the ith vehicle at time t is . In the feature extraction process, the node feature matrix is first mapped to the first layer feature vector and the second layer feature vector through two fully connected layers with a RELU activation function for nonlinear mapping, and the calculation process is as follows:

[0111] ;

[0112] ;

[0113] wherein, , represents the weight matrix; and represent the bias.

[0114] The original node feature of the ith vehicle after nonlinear transformation is sent to the graph attention network (GAT). First, a linear transformation projection is performed to obtain , and a multi-head attention mechanism is further used to extract the spatial interaction relationship between vehicle i and neighboring vehicles. The attention score between the ith vehicle and its neighboring jth vehicle, the normalized attention score , and the node feature vector of the ith vehicle at time t are represented as follows:

[0115] ;

[0116] ;

[0117] ;

[0118] ;

[0119] where k represents the number of attention heads; and represent the learnable weights; LeakyReLU and ELU represent the activation functions of nonlinear modeling; represents vector splicing; represents the set of neighboring vehicles of vehicle i at time t.

[0120] In order to improve the stability of the agent training, the final output of the ith vehicle at time t is obtained by using residual connection and layer normalization as follows:

[0121] ;

[0122] After feature extraction, the node feature vector of each vehicle is obtained , and the action strategy of all vehicles is generated , and the mask matrix is used to screen out the action output corresponding to the CAV, which is represented as follows:

[0123] ;

[0124] When the element in the mask matrix , the action corresponding to the CAV vehicle is output.

[0125] In this embodiment, the graph attention mechanism is used to extract the spatio-temporal features as the state input of the reinforcement learning algorithm, which improves the modeling ability and decision accuracy of the CAV agent in complex dynamic traffic environment.

[0126] In some embodiments, the CAV trajectory control agent includes the step of training and optimizing the main Q network parameters using the target Q network. The target Q network is responsible for evaluating the value of the action corresponding to the next state, stabilizing the target value calculation and alleviating the overestimation problem of Q value. The parameters are synchronized from the main Q network to the target Q network every fixed number of steps. In the step:

[0127] The greedy strategy is used to select all horizontal / longitudinal driving decision actions of the CAV under the current state, and a random action is selected with a probability of , and the action with the maximum Q value is selected as the trajectory control strategy of the CAV with a probability of :

[0128] ;

[0129] wherein, the action policy of the ith vehicle generated by the main Q-network, the main Q-network outputs all the lateral / longitudinal driving decision actions of the CAV in the current state, and θ is the network parameter of the main Q-network.

[0130] In some embodiments, the CAV trajectory control agent comprises a step of training and optimizing the main Q-network parameter by using the target Q-network, in which step:

[0131] The optimization objective function of the Q-network parameter is as follows:

[0132] ;

[0133] wherein, θ is the main Q-network parameter, is the state of the agent in the state space at the current time the estimated Q-value of the action at the next time, the target Q-value is:

[0134] ;

[0135] wherein, is the reward of the lateral / longitudinal decision action of the CAV at the current time, is the discount factor, is the parameter of the target Q-network, is the state of the corresponding CAV vehicle in the agent state space at the next time, is the action performed by the corresponding CAV vehicle at the next time, which is calculated by the main Q-network at the next time state , that is, and then inputting the action into the target Q-network to obtain as the Q-value estimation of the target Q-network for the state action at the next time.

[0136] In some embodiments, for the intelligent network connected signal intersection scene, in order to avoid the occurrence of unsafe behaviors such as collision and red light running of the CAV in the training and execution process, a safety check mechanism is adopted to constrain the driving behavior of the CAV, so that it can be in a safe state in the training and execution phase. This mechanism is responsible for determining whether the action of each CAV output by the main Q-network is safe, and if the action leads to unsafe behavior of the CAV vehicle, the safe action is replaced or directly shielded. The safety check mechanism includes:

[0137] The longitudinal action checking mechanism, if the action is taken to make the vehicle distance to the front vehicle less than the minimum safety distance at the next time, the action is considered as unsafe action, the safe acceleration is used to replace and the negative value is punished in the safety reward function;

[0138] The lateral action checking mechanism, checking the CAV lateral action output by the agent, if the action leads to the vehicle distance to the front and rear vehicles of the target lane less than the minimum safety distance, the action is considered as unsafe behavior, the action is shielded and the negative value is punished in the safety reward function.

[0139] The embodiment filters the dangerous action by checking the lateral and longitudinal action strategy output by the CAV agent and guides the CAV agent to learn the safe and reliable driving strategy through the safety reward function.

[0140] In a specific embodiment, the longitudinal action checking step comprises:

[0141] The distance of the CAV from the signal lamp after the acceleration output by the agent is performed The definitions are as follows:

[0142] ;

[0143] Wherein, represents the distance of the CAV from the front signal lamp; represents the longitudinal speed of the CAV at the next time; represents the simulation step; represents the preset safety distance.

[0144] The minimum safety distance The minimum value of the emergency braking safety distance, the headway safety distance and the collision safety distance is represented as follows:

[0145] ;

[0146] Wherein, the emergency braking safety distance The calculation formula is represented as follows:

[0147] ;

[0148] Wherein, represents the driver reaction delay time; represents the speed of the CAV front vehicle; represents the safety buffer distance; represents the maximum deceleration of the CAV.

[0149] The headway safety distance The calculation formula is represented as follows:

[0150] ;

[0151] wherein, denotes the critical headway; denotes the vehicle length.

[0152] Collision safety distance The calculation formula is shown as follows:

[0153] ;

[0154] wherein, denotes the collision time threshold.

[0155] If the current time signal light is red and the distance of the CAV to the signal light at the next time is less than the minimum safety distance , the longitudinal acceleration of the CAV is replaced by the maximum deceleration . If the current signal light is green, it is necessary to determine whether there is a front vehicle, and the distance of the CAV to the front vehicle at the next time is calculated and compared with the minimum safety distance. The calculation formula is shown as follows:

[0156] ;

[0157] wherein, denotes the distance between the CAV and the front vehicle at the current time;

[0158] If , the safety acceleration required for the CAV to maintain the minimum safety distance with the front vehicle is calculated and the acceleration output by the agent is replaced, which is shown as follows:

[0159] .

[0160] In one specific embodiment, the lateral action checking step comprises:

[0161] When the CAV agent outputs a lateral action to control the CAV to drive from the current lane to the target lane, the safety checking mechanism calculates the distance between all vehicles on the target lane and the CAV at the next time, and if it is less than the minimum safety distance, it is considered as an unsafe action and is shielded, and the calculation formula is as follows:

[0162] ;

[0163] wherein, denote the position, speed of the CAV at the current time, position, speed, vehicle length of the vehicle on the target lane, respectively; denotes the time step; denotes the preset minimum safety distance of the vehicle.

[0164] To verify the effectiveness of the algorithm proposed in the present application, SUMO simulation software is used to build an intelligent networked signal intersection scene for experiment. As shown in Figure 3 , the experimental scene is a one-way two-lane signal intersection, each lane is 200 m long, the signal timing is green for 30 s, red for 30 s, and yellow for 4 s. Each CAV in the scene is regarded as a DRL control agent, and multiple CAVs constitute a multi-agent DRL control. CAVs realize the function of vehicle-to-vehicle (V2V) communication through the on-board communication unit (OBU) at the front end of the vehicle for communication and information exchange with other vehicles. The road side unit (RSU) is responsible for transmitting the signal light phase and timing information to the CAV, realizing the function of vehicle-to-infrastructure (V2I) communication. The longitudinal motion trajectory of the HV in the experimental scene is controlled by the intelligent driver model (IDM), the lateral motion trajectory is controlled by the LC2013 lane changing model, and the lateral and longitudinal motion trajectories of the CAV are controlled by the safety reinforcement learning method proposed in the present application.

[0165] The present application is based on the following assumptions:

[0166] 1) In the experimental scene, CAV and CAV, CAV and HV can normally realize the V2V communication function, and CAV and RSU can normally realize the V2I communication function, without considering communication delay and data packet loss.

[0167] 2) CAV can use V2V and V2I technology to receive real-time information of CAV and HV within its communication range, such as position, speed, signal phase and timing, etc., while there is no information exchange between HVs.

[0168] 3) CAV is completely controlled by the safety reinforcement learning algorithm proposed in the present application without human intervention.

[0169] The CAV trajectory control process based on safety reinforcement learning is shown in Figure 4 .

[0170] Step 1.1, set the total number of training rounds G, the number of simulation steps per round T = 1000. The number of small batch training samples is B = 64. The initial value of the greedy search is 0.4, and the decay factor is decreased to the minimum value every time the training is performed. The warm-up step number R = 75000.

[0171] Step 1.2, initialize the Q network parameter θ and the target Q network parameter , global simulation step g and shared experience replay pool M.

[0172] Step 1.3, at simulation time t, obtain the initial global state of the current intelligent connected hybrid traffic flow environment , including the node feature matrix , the adjacency matrix , and the mask matrix .

[0173] Step 1.4, in the warm-up phase (g < R), the agent selects CAV lateral lane changing actions and longitudinal acceleration / deceleration actions using an e-greedy strategy and executes them without training the agent, only interacting with the environment to collect experiences to fill the experience replay pool.

[0174] Step 1.5, when g > R, the Q network of the intelligent agent estimates the Q values of all CAV lateral lane changing actions and longitudinal acceleration / deceleration actions based on the state collected in step 1.3, and selects the optimal joint action through a greedy strategy to achieve synchronous control of CAV lateral and longitudinal trajectories.

[0175] Step 1.6, the action is filtered through a safety check mechanism to obtain a safe action .

[0176] Step 1.7, the CAV executes the lateral and longitudinal safe actions and obtains the reward and the next state from the intelligent connected signalized intersection environment, and stores the experience in the form of four-tuple data samples in the experience replay pool.

[0177] Step 1.8, every fixed number of steps, the agent randomly collects B small batches of data samples from the experience pool, and updates the learning parameters θ of the Q network using the Adam algorithm. Every fixed number of steps, the target Q network parameters are synchronized once from the Q network.

[0178] Step 1.9, repeat steps 1.3 to 1.8 until the total number of training rounds reaches G, the training is complete, and the agent is saved. Use the trained agent to evaluate the actual effect of CAV trajectory control in the test environment.

[0179] To verify the advantages of the CAV longitudinal and lateral trajectory joint control method based on safety reinforcement learning proposed in the application, five CAV penetration rates (0%, 30%, 50%, 80%, and 100%) scenarios are selected, and the CAV longitudinal and lateral trajectory control effect is evaluated by repeating experiments multiple times for each penetration rate. The evaluation indexes take into account traffic efficiency and environmental protection, including fuel consumption of all CAVs and HVs, CO2 emissions of all CAVs and HVs, average queue length, and average delay. The baseline model selects the IDM model to control the vehicle longitudinal trajectory and the LC2013 model to control the vehicle lateral trajectory. Figure 4 The reward change graph when the CAV penetration rate is 50% is shown, and the shaded part is the standard deviation. In the preheating stage at the beginning of training, the CAV adopts a greedy strategy to select random actions and interacts with the environment, and the reward fluctuates constantly. After 70 rounds, as the training steps increase, the reward value of the CAV agent gradually increases, and the fluctuation trend of the reward value gradually slows down, indicating that the CAV agent gradually learns the strategy of the approximate optimal solution and finally converges.

[0180] The efficiency and environmental protection index changes under different penetration rates are shown in Table 1. Under the control of the safety reinforcement learning algorithm, as the CAV penetration rate continuously increases, the improvement degree of each index gradually increases. At a 100% CAV penetration rate, the fuel consumption and CO2 emissions of the mixed traffic flow are significantly improved, with a 15.4% reduction in fuel consumption and a 15.2% reduction in CO2 emissions compared to the completely HV traffic flow (MPR=0%), indicating that the CAV agent has learned an energy-saving driving strategy. The average delay and average queue length also continuously decrease as the CAV penetration rate increases, and when the CAV reaches a 100% penetration rate, the average queue length of the mixed traffic flow is reduced by 78.7% compared to the completely HV traffic flow (MPR=0%). This indicates that under the control of the safety reinforcement learning algorithm proposed in the application, the CAV agent guides the mixed vehicle fleet to pass through the signal intersection more smoothly by reducing unnecessary braking and starting, thereby improving the traffic capacity of the signal intersection.

[0181] Table 1 Comparison of traffic efficiency and environmental protection indexes under different penetration rates

[0182]

[0183] To further verify the advantage of the CAV trajectory control algorithm based on the safety reinforcement learning in ecological driving, Fig. 5 shows the cumulative emissions of various greenhouse gases with the simulation time under the CAV penetration rate of 50%, wherein Fig. (a) is the cumulative CO2 emission change, Fig. (b) is the cumulative CO emission change, Fig. (c) is the cumulative HC emission change, and Fig. (d) is the cumulative NOx emission change. Under the control of the safety reinforcement learning algorithm, the emission indicators of the mixed traffic flow are all reduced compared with the control of the existing model, which shows the advantage of the CAV trajectory control algorithm based on the safety reinforcement learning in ecological driving.

[0184] The vehicle space-time trajectories under the CAV penetration rates of 0%, 30%, 50%, 80% and 100% are shown in Figs. (a), (b), (c), (d) and (e) of Fig. 6, wherein the solid line and the dashed line respectively represent the trajectories in different lanes. When the CAV penetration rate is 0%, that is, all the vehicles in the traffic flow are controlled by the benchmark model, the HV has an obvious deceleration action when reaching the signal intersection, and the generated traffic wave gradually propagates backward to form a queue. As the CAV penetration rate increases, the deceleration and queuing of the mixed traffic flow are improved under the control of the safety reinforcement learning algorithm proposed in the application. Especially when the CAV penetration rate is 100%, the queuing length of the mixed traffic flow is greatly reduced, and the trajectory is smoother. This shows that the CAV can plan the speed in advance through the accurate perception of the signal phase and the remaining time, avoid the driving behavior of sudden braking and sudden starting before the signal intersection, guide the surrounding vehicles to smoothly pass through the signal intersection, improve the queuing phenomenon of the mixed traffic flow, and improve the driving stability of the mixed traffic flow.

[0185] The simulation results verify the advantage of the CAV trajectory control method based on the safety reinforcement learning in improving the passing efficiency and energy saving and emission reduction of the mixed traffic flow. As the CAV penetration rate increases, the passing efficiency and the ecological environmental protection indicators of the mixed traffic flow are continuously improved, which shows that the safety reinforcement learning method proposed in the application can flexibly cope with the traffic situations under different CAV penetration rates in real traffic scenes, and has good universality.

[0186] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0187] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A trajectory control method for intelligent connected vehicles based on safety reinforcement learning, characterized in that, Includes the following steps: S1: Model the intelligent connected signal intersection scenario, which includes CAV vehicles and HV vehicles, as an undirected graph. CAV vehicles and HV vehicles are nodes of the undirected graph, and the communication relationships between vehicles are edges of the undirected graph. S2: Construct a CAV trajectory control agent based on the DDQN network, including a master Q network and a target Q network with the same network architecture; extract the dynamic spatial features between each node of the CAV and HV in the undirected graph to obtain node feature vectors, which serve as the agent's state space; and use the CAV's lateral / longitudinal driving decision actions as the agent's joint action space. A multi-objective reward function is constructed based on global traffic information in the intelligent connected signalized intersection scenario; the multi-objective reward function includes: speed reward function, safety inspection mechanism reward function, queue length reward function, and passage time reward function. The CAV trajectory control agent includes a step of training and optimizing the parameters of the main Q-network using a target Q-network, wherein: A greedy strategy is used to select all lateral / longitudinal driving decision actions of the CAV in the current state, in order to The probability of choosing a random action is 1. The probability of selecting the action with the largest Q value is used as the trajectory control strategy for the CAV. ； in, The first generation generated by the main Q network i Vehicle movement strategy The main Q network outputs all lateral and longitudinal driving decision actions of the CAV in the current state. θ Network parameters of the main Q network; The objective function for optimizing the Q network parameters is as follows: ； in, Main Q network parameters, The state in the agent's state space at the current moment Next action The estimated Q value, For the target Q value: ； in, The reward for the CAV's horizontal / vertical decision-making actions at the current moment. As a discount factor, The parameters of the target Q-network, This represents the state of the corresponding CAV vehicle in the agent's state space at the next moment. To determine the CAV vehicle's action in the next moment, the main Q network determines the state in the next moment. The following calculation yields, i.e. Then, the action is input into the target Q-network to obtain... As the target Q-network, the state Next action Q-value estimation; S3: The main Q-network extracts the agent's state space data at the current moment, outputs the Q-values ​​of all lateral / longitudinal driving decision actions of the CAV in the current state, and selects the optimal joint action in conjunction with a safety check mechanism for synchronous control of the CAV's lateral / longitudinal decision actions; the safety check mechanism includes: The longitudinal action check mechanism is used to check whether the CAV vehicle meets the safety acceleration requirements in the current lane. If not, it calculates the safety acceleration and replaces the corresponding longitudinal driving decision action. The lateral movement check mechanism is used to detect whether the distance between the CAV vehicle and all vehicles in the target lane meets the safety distance requirements when the CAV vehicle moves from the current lane to the target lane. If not, the corresponding lateral driving decision action is blocked. S4: Obtain the reward of the CAV horizontal / vertical decision action at the current moment according to the multi-objective reward function, and combine it with the agent state space data at the current moment, the optimal joint action, and the agent state space data at the next moment fed back from the intelligent connected signal intersection scenario, and store them together as a sample data group in the shared experience playback pool. S5: The target Q network periodically collects sample data sets from the shared experience replay pool to train and optimize the parameters of the main Q network. The optimized main Q network is used to execute S3 in real time to achieve real-time synchronous control of CAV horizontal / vertical decision-making actions.

2. The intelligent connected vehicle trajectory control method based on safety reinforcement learning according to claim 1, characterized in that, The undirected graph in S1 includes: a node feature matrix. Adjacency matrix Mask matrix ;in, At any moment t Node feature matrix Represented as: ； in, n This represents the total number of vehicles in CAV and HV at the current moment; Indicates the first i The car at any time t The node feature matrix, for CAV vehicles, is as follows: include: ； For HV vehicles, include: ； in, This represents the ratio of a vehicle's longitudinal speed to the road's maximum speed limit. This represents the ratio of the distance traveled by the vehicle to the length of the road. The code indicating the lane the vehicle is in; Indicates the current status of the traffic light ahead; This indicates the remaining green light time when the traffic light ahead is green, or the waiting time required for the next green light phase to arrive when the traffic light ahead is red or yellow. This represents the ratio of the distance between the vehicle and the signalized intersection ahead to the length of the lane. This represents the ratio of the queue length at the signalized intersection to the total number of vehicles in the scene. Adjacency Matrix The information exchange relationship between vehicles is represented by different codes, where the adjacency matrix of any two CAV vehicles is... Assigning encoding to establish communication link relationships at any time, and the adjacency matrix of CAV vehicles and HV vehicles. Establish communication link relationship coding under given conditional constraints; Mask matrix Used to filter the HV vehicle lateral / longitudinal driving decision action outputs from all vehicle lateral / longitudinal driving decision actions generated by the CAV trajectory control agent.

3. The intelligent connected vehicle trajectory control method based on safety reinforcement learning according to claim 1, characterized in that, The agent joint action space includes: CAV longitudinal acceleration action space and CAV lateral lane-changing action space.

4. The intelligent connected vehicle trajectory control method based on safety reinforcement learning according to claim 2, characterized in that, The network architecture of the primary Q network and the target Q network includes: No. i The car is t Time node feature matrix The first layer feature vector is obtained by using two fully connected layers and a ReLU activation function for nonlinear mapping. Second layer feature vector ; The second layer feature vector The data is fed into the Graph Attention Network (GAT) and a multi-head attention mechanism is used to extract vehicle data. i The spatial interaction relationship with neighboring vehicles is obtained by using residual connections and layer normalization to obtain the first... i The car is t Node feature vector at time step And thus generate the first i Vehicle movement strategy ; If the first i If the vehicle is a CAV (Continuous Access Vehicle), then a mask matrix is ​​used. Output at t At time 1, all actions of the CAV vehicle are represented, i.e., the Q value of the corresponding action; if the... i If the vehicle is an HV vehicle, then a mask matrix is ​​used. Disable the action output corresponding to the HV vehicle.

Images