Mixed traffic flow cooperative control method based on double-layer parameterization deep reinforcement learning

Abstract

In order to solve the defects in the prior art, the invention discloses a mixed traffic flow cooperative control method based on double-layer parameterization deep reinforcement learning. According to the method, traffic high-dimensional spatial features are extracted from an intersection environment through a convolutional neural network, a first-layer model CNN-MPDQN is constructed in combination with a multi-channel parameterization depth Q network algorithm, and the main task of the method is to select a passing priority lane and determine passing duration. And then, on the basis of the first-layer algorithm, a second-layer DQN algorithm is adopted to optimize and coordinate the passing sequence of the vehicles on the lane, and the problem of vehicle track conflict is solved. Through continuous loop training and iteration of a two-layer deep reinforcement learning algorithm model, model parameters are optimized step by step, and finally an optimal convergence model is obtained; the model can effectively cope with various traffic conditions and vehicle behaviors, improves the traffic efficiency and safety of the intersection, and is especially suitable for a mixed flow intersection without signal control.

Description

Technical Field

[0001] The present invention relates to the field of cooperative control of autonomous driving and human driving, and in particular to a hybrid traffic flow cooperative control method based on double-layer parameterized deep reinforcement learning. Background Art

[0002] With the acceleration of urbanization, traffic congestion has become an important problem faced by many cities, especially the traffic efficiency of key traffic nodes such as intersections, which directly affects the smoothness and safety of the entire traffic network. In recent years, autonomous driving technology has developed rapidly, and connected autonomous vehicles (CAVs) have been gradually applied to traffic systems, bringing new challenges to traditional traffic control methods. In this context, how to coordinate the passage of CAVs and human-driven vehicles (HVs) at mixed-flow intersections, especially intersections without signal control, has become a key issue in traffic system optimization.

[0003] Traditional traffic light control methods usually adjust the traffic priority of intersections based on fixed-cycle signal switching or preset rules based on traffic flow. These methods are suitable for fixed traffic patterns, but when faced with mixed traffic (CAV and HV coexisting), traditional methods lack flexibility and intelligence and cannot make optimal decisions based on real-time traffic conditions, resulting in uneven traffic flow and low traffic efficiency.

[0004] In recent years, deep reinforcement learning (DRL), as an emerging artificial intelligence technology, has achieved remarkable success in many fields, especially in traffic control and autonomous driving. DRL continuously obtains feedback from the environment and learns how to maximize a certain goal (such as traffic flow or safety) through intelligent decision-making. However, although DRL has made some progress in traffic control, existing research still has limitations: (1) For mixed-flow intersections, existing DRL methods often do not fully consider the synergy between the two, resulting in mutual interference and incoordination between CAVs and HVs, reducing the traffic efficiency of intersections; (2) Existing DRL methods usually optimize for a single task, such as the order of vehicle passage, and it is difficult to comprehensively consider multiple levels of decision-making. In actual traffic scenarios, it is necessary not only to determine the priority of lanes, but also to consider the dynamic adjustment of coordinated and conflicting lanes, and to optimize the duration of passage at the same time. These factors require a multi-level reinforcement learning mechanism to coordinate. (3) Traffic intersections usually have highly complex environmental characteristics (such as traffic density, traffic status, vehicle speed, etc.), which are key factors affecting traffic efficiency and safety. However, existing reinforcement learning algorithms usually lack effective feature extraction capabilities when faced with these high-dimensional and complex traffic characteristics, resulting in unsatisfactory decision-making results. Summary of the invention

[0005] In order to address the shortcomings of the prior art, the present invention proposes a hybrid traffic flow collaborative control method based on two-layer parameterized deep reinforcement learning, combines convolutional neural network (CNN) for high-dimensional spatial feature extraction, and realizes dynamic adjustment of traffic priority, traffic duration and traffic order through a two-layer deep reinforcement learning model, thereby improving the intelligence level and traffic efficiency of the traffic control system.

[0006] The technical solution adopted by the present invention to solve the above-mentioned tasks is:

[0007] A hybrid traffic flow cooperative control method based on double-layer parameterized deep reinforcement learning includes the following steps: Construct a road model for an intersection with mixed traffic of manual and automated driving without signals, and divide the intersection area into functional areas; Based on the original multi-channel parameterized deep Q network (MPDQN) algorithm, the first-layer deep reinforcement learning algorithm model (CNN-MPDQN) was constructed in combination with CNN. CNN was used to effectively extract the high-dimensional spatial characteristics of traffic at intersections. The core task of the CNN-MPDQN algorithm is to select a lane with the highest priority from multiple lanes and determine the duration of traffic in the lane. Based on the lane with the highest priority selected by the first layer of reinforcement learning, the coordination lane and the conflicting lane are determined; during the duration of the passage, vehicles on the coordination lane can coordinate to pass through the intersection, while vehicles in the conflicting lane need to stop and wait for the next round of optimization; According to the traffic volume of each coordinated lane, the proportion of travel duration is allocated respectively to maximize the use of travel duration to optimize traffic flow efficiency; Construct a second-layer deep reinforcement learning algorithm model, using the deep Q network (DQN) algorithm. During the first-layer passage duration, reinforcement learning is used to determine the passage order of vehicles on the coordination lane, aiming to solve the trajectory conflict problem of vehicles on the coordination lane. Under the high-level decision-making of traffic order based on two-layer reinforcement learning, priority vehicles pass through the intersection at the maximum safe speed, while conflicting vehicles need to slow down and stop and wait in front of the conflicting position; Through continuous cyclic training and iteration of the two-layer deep reinforcement learning algorithm model, the model parameters are gradually optimized, and finally the optimal convergence model is obtained; this model can effectively respond to various traffic conditions and vehicle behaviors, and improve the traffic efficiency and safety of intersections.

[0008] Furthermore, the constructed road model is a four-way three-lane intersection, with left-turn, straight and right-turn lanes in each direction, and the right-turn lane is uncontrolled; the intersection area is divided into a detection zone and a conflict zone; the road section between the maximum communication range and the stop line is the detection zone, and the conflict zone is an area where a vehicle may conflict with vehicles in other directions after crossing the stop line; CAVs and HVs coexist on the road, and vehicles that enter the detection zone and communicate with the intersection central processor are identified as CAVs, while vehicles that can be detected by the central processor but do not communicate are identified as HVs; all vehicles travel according to pre-planned trajectories.

[0009] Furthermore, the driving behavior of the HV is mainly divided into two types: following the preceding vehicle and free driving without a preceding vehicle; when two free HVs from conflicting directions arrive at the intersection at the same time, they only need to follow the right-hand rule to pass through the intersection; if there is a vehicle in front of the HV, the HV needs to maintain a safe distance to avoid collision. The target speed at the moment should meet the following conditions: in, is the maximum speed limit of the road, is the maximum acceleration, is the simulation update step size, For safe following speed, The formula is: in, is the speed of the preceding vehicle at time t, is the inter-vehicle distance at time t, for the reaction time of human drivers; In addition, the introduction of random factors , represents the uncertainty of human driving behavior, and the final following speed is: .

[0010] Furthermore, the first-layer deep reinforcement learning algorithm model adopts the CNN-MPDQN algorithm, and the specific state space, action space, reward function, network architecture and loss function are designed as follows: 4.1 State Space Design In order to intuitively present the current traffic conditions at the intersection, the gridding method is used to discretize the intersection, in which the side length of the grid is set to the lane width; the state space is represented by a matrix of the position, speed and waiting time of the vehicle on the lane; specifically, the position matrix is ​​used to indicate whether each grid is occupied by a vehicle, and 1 or 2 is used to indicate the presence of a CAV or HV in the position matrix, respectively. When a vehicle occupies more than half of the grid area, the corresponding grid is marked as 1 or 2, otherwise it is marked as 0; the speed matrix records the speed value of the vehicle when there is a vehicle in the grid; the waiting time matrix records the waiting time value of the vehicle in the grid; this method can effectively reflect the traffic flow and vehicle status of the intersection; 4.2 Action Space Design The parameterized action space consists of a set of discrete actions, , where each discrete action corresponds to a continuous parameter, ; Specifically, the discrete action is represented by all candidate lanes. After selecting one of the lanes as the highest priority lane, the travel duration of the lane is defined as its corresponding continuous parameter; the action space can be expressed as: in, Yes all A continuous set of parameters; 4.3 Reward Function Design Design a multi-objective reward function that comprehensively considers efficiency, fairness, and HV , which is defined as: in, It indicates the waiting time of vehicles that are forced to stop and wait due to conflict with vehicles in the highest priority lane; Designed to increase the chances of vehicles becoming coordinated vehicles, allowing them to coordinate through the intersection rather than becoming conflicting vehicles and being forced to wait; Aims to increase the chances of more HVs becoming priority vehicles, rather than becoming conflicting or coordination vehicles that interfere with the safe passage of HVs and the coordination of second-tier CAV vehicles; in, Indicates Vehicle waiting time, Indicates the waiting time threshold, represents the number of conflicting vehicles, represents the number of coordinated vehicles, represents the ratio of CAVs in the highest priority lane, , , represents the weight coefficient; 4.4 Network Architecture Design The CNN-MPDQN algorithm model structure includes a continuous parameter strategy network, a discrete action Q network and a target network; CNN is used to effectively extract the high-dimensional spatial characteristics of traffic at intersections; specifically, the continuous parameter strategy network takes the current state as input, and after processing by the convolution layer, the fully connected layer and the activation function, outputs the optimal vector of all continuous parameters in the action space; then the vector is separated so that each discrete action only inputs the parameters related to it, thereby reducing the influence of other continuous parameters on the Q value estimation; in order to keep the length of the continuous action parameter vector consistent, the irrelevant continuous parameters are set to 0, and then the current state vector processed by the convolution layer and the separated continuous action are spliced ​​to form The vector is input into the discrete action Q network, and after passing through the fully connected layer and activation function, the output Matrix, and extract the diagonal Q value to form a vector, and finally get the optimal Q value of the discrete action Q network; 4.5 Loss Function Design The loss function consists of two parts: the discrete action Q network and the continuous parameter policy network. Specifically, for the discrete action Q network loss function, it is constructed in the form of mean square error, aiming to minimize the target loss: in, is the n-step objective function, expressed as: in, is the discount factor, and They are the discrete action target Q network parameters and the continuous parameter strategy target network parameters respectively; For the continuous parameter policy network loss function, the goal is to maximize the Q-value estimate, defined as: .

[0011] Furthermore, the coordination lane is a lane combination that has no trajectory conflict with the highest priority lane, and the conflict lane is a lane combination that has a trajectory conflict with the highest priority lane.

[0012] Furthermore, the proportion of the travel duration is allocated according to the traffic volume of each coordinated lane. The specific calculation formula for the weight of the coordinated lanes is: in, To coordinate the number of lanes, and Respectively Article and The number of waiting vehicles in the detection zone on the lane, and Respectively Article and The future predicted traffic flow for each lane is calculated based on the arrival time of vehicles, which follows a Poisson distribution.

[0013] Furthermore, the second-layer deep reinforcement learning algorithm model adopts the DQN algorithm. During the first-layer passage duration, the reinforcement learning algorithm determines the passage order of vehicles on the coordinated lane. The specific state space, action space, and reward function of the model are designed as follows: 7.1 State Space Design The gridding method is used to discretize the intersection. In the second-layer deep reinforcement learning algorithm model, the focus is on the traffic order of vehicles on the coordination lane. Therefore, the state space only represents the state information of the vehicles on the coordination lane, and is described by the matrices of the position, expected position, speed and waiting time of the vehicles on the coordination lane. Specifically, the position matrix is ​​used to indicate whether each grid is occupied by a vehicle, and the expected position matrix represents the grid that the vehicle will occupy in the future, which is used to reflect the conflict relationship between vehicles. When there is a vehicle in the grid, the speed matrix records the speed value of the vehicle. The waiting time matrix records the waiting time value of the vehicle in the grid. This design can accurately represent the vehicle state of the coordination lane and provide a basis for optimizing the traffic order. 7.2 Action Space Design The response action is designed to coordinate the traffic sequence of vehicles on the lane, expressed as: Total candidate action combinations; based on this, the reinforcement learning algorithm identifies the optimal pass sequence combination; 7.3 Reward Function Design The reward function of the second layer is defined to minimize the waiting time and passing time of the coordinated vehicles, expressed as: in, Indicates penalty coordination vehicle waiting time, It means to punish the passing time of coordinated vehicles, aiming to make coordinated vehicles pass the intersection as quickly as possible; Indicates Coordinate the passing time of vehicles, Indicates the passing time threshold, , represents the weight coefficient; 7.4 Network Architecture Design The DQN algorithm model structure includes a Q-value network and a target network. Consistent with the first-layer algorithm model, CNN is used as a feature extractor. Specifically, the Q-value network takes the current state as input, and after processing by the convolution layer, the fully connected layer and the activation function, outputs the optimal Q value of the action. The target network is used to prevent the Q value from updating too quickly, thereby avoiding training instability. Specifically, a soft update method is adopted, and the parameters of the target network are synchronized with the Q-value network once every N steps, that is: in, is the target network parameter, is the Q value network parameter; 7.5 Loss Function Design The loss function is constructed in the form of mean squared error, which aims to reduce the loss with the n-step objective function: .

[0014] Furthermore, under the high-level decision of traffic order of two-layer reinforcement learning, the priority vehicle passes through the intersection at the maximum safe speed, while the conflicting vehicle needs to slow down and stop and wait in front of the conflicting position; specifically, the CAV vehicle with priority in the coordination lane may encounter the HV traveling freely in other coordination lanes during driving. To deal with this situation, the method adopts an event-triggered re-decision mechanism.

[0015] Furthermore, the method adopts an event-triggered re-decision mechanism. Specifically, when it is retrieved that the priority CAV vehicle and the HV vehicle may occupy the intersection conflict point at the same time, the system will readjust the maximum safe speed of vehicles passing in the coordination lane according to the estimated time for the priority CAV and HV to arrive at the intersection conflict point. The maximum safe speed can ensure that the priority CAV vehicle and the HV vehicle do not collide and pass through the intersection conflict area safely. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the collaborative control method of the present invention;

[0017] Figure 2 This is a schematic diagram of the intersection scene;

[0018] Figure 3 A schematic diagram for constructing a state matrix;

[0019] Figure 4 This is a schematic diagram of the CNN-MPDQN algorithm model structure of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be described in further detail below in conjunction with the accompanying drawings.

[0021] like Figure 1 As shown, the collaborative control method based on double-layer parameterized deep reinforcement learning described in the present invention is applied to mixed flow intersections, and the specific steps are as follows:

[0022] Step 1: Construct a road model for an intersection with mixed flow of manual and automated driving without signals.

[0023] like Figure 2 As shown in the figure, the constructed road model is a four-way three-lane intersection, with left-turn, straight-ahead and right-turn lanes in each direction, and the right-turn lane is uncontrolled; the intersection area is divided into a detection zone and a conflict zone; the section between the maximum communication range and the stop line is the detection zone, and the conflict zone is the area where vehicles may conflict with vehicles in other directions after crossing the stop line; CAVs and HVs coexist on the road, and vehicles that enter the detection zone and communicate with the intersection central processor are identified as CAVs, while vehicles that can be detected by the central processor but do not communicate are identified as HVs; all vehicles drive according to pre-planned trajectories. Inside the intersection, the point where two driving trajectories intersect is defined as the conflict point position.

[0024] The collaborative method described in the present invention fully considers the different driving behaviors of HVs, and realizes effective coordination with CAVs without interfering with the normal driving of HVs as much as possible. The driving behaviors of HVs considered are mainly divided into two types: following the preceding vehicle and free driving without a preceding vehicle; when two free HVs from conflicting directions arrive at an intersection at the same time, they only need to follow the right-hand rule to pass through the intersection; if there is a vehicle in front of the HV, the HV needs to maintain a safe distance to avoid collision, so the HV The target speed at the moment should meet the following conditions: in, is the maximum speed limit of the road, is the maximum acceleration, is the simulation update step size, For safe following speed, The formula is: in, is the speed of the preceding vehicle at time t, is the inter-vehicle distance at time t, for the reaction time of human drivers; In addition, the introduction of random factors , represents the uncertainty of human driving behavior, and the final following speed is: .

[0025] Step 2: Use the CNN-MPDQN algorithm to build the first-layer deep reinforcement learning algorithm model. Its core task is to select a lane with the highest priority from multiple lanes and determine the duration of the lane. The specific state space, action space, reward function, network architecture and loss function are designed as follows: Step 2.1: State Space Design In order to intuitively present the current traffic conditions at the intersection, the gridding method is used to discretize the intersection, where the side length of the grid is set to the lane width; Figure 3 As shown, Figure 1 The intersection scene shown is discretized into grids to construct a state space matrix. It should be noted that the state matrix does not include information about right-turning vehicles or vehicles that have passed through the conflict area. The purpose of this is to focus on vehicles that are currently inside the intersection or in front of the conflict area to simplify the state space and improve the computational efficiency of the algorithm. The state space is represented by a matrix of the position, speed, and waiting time of the vehicle on the lane; specifically, the position matrix is ​​used to indicate whether each grid is occupied by a vehicle, and 1 or 2 is used to indicate the presence of a CAV or HV in the position matrix, respectively. When a vehicle occupies more than half of the grid area, the corresponding grid is marked as 1 or 2, otherwise it is marked as 0; the speed matrix records the speed value of the vehicle when there is a vehicle in the grid; the waiting time matrix records the waiting time value of the vehicle in the grid; this method can effectively reflect the traffic flow and vehicle status of the intersection; Step 2.2: Action Space Design The parameterized action space consists of a set of discrete actions, , where each discrete action corresponds to a continuous parameter, ; Specifically, the discrete action is represented by all candidate lanes. After selecting one of the lanes as the highest priority lane, the travel duration of the lane is defined as its corresponding continuous parameter; the action space can be expressed as: in, Yes all A continuous set of parameters; Step 2.3: Reward function design Design a multi-objective reward function that comprehensively considers efficiency, fairness, and HV , which is defined as: in, It indicates the waiting time of vehicles that are forced to stop and wait due to conflict with vehicles in the highest priority lane; Designed to increase the chances of vehicles becoming coordinated vehicles, allowing them to coordinate through the intersection rather than becoming conflicting vehicles and being forced to wait; Aims to increase the chances of more HVs becoming priority vehicles, rather than becoming conflicting or coordination vehicles that interfere with the safe passage of HVs and the coordination of second-tier CAV vehicles; in, Indicates Vehicle waiting time, Indicates the waiting time threshold, represents the number of conflicting vehicles, represents the number of coordinated vehicles, represents the ratio of CAVs in the highest priority lane, , , represents the weight coefficient; Step 2.4: Network architecture design like Figure 4 As shown in the figure, the CNN-MPDQN algorithm model structure includes a continuous parameter strategy network, a discrete action Q network and a target network; CNN is used to effectively extract the high-dimensional spatial characteristics of the traffic at the intersection; specifically, the continuous parameter strategy network takes the current state as input, and after processing by the convolution layer, the fully connected layer and the activation function, outputs the optimal vector of all continuous parameters in the action space; then the vector is separated so that each discrete action only inputs the parameters related to it, thereby reducing the influence of other continuous parameters on the Q value estimation; in order to keep the length of the continuous action parameter vector consistent, the irrelevant continuous parameters are set to 0, and then the current state vector processed by the convolution layer and the separated continuous action are spliced ​​to form The vector is input into the discrete action Q network, and after passing through the fully connected layer and activation function, the output Matrix, and extract the diagonal Q value to form a vector, and finally get the optimal Q value of the discrete action Q network; Step 2.5: Loss function design The loss function consists of two parts: the discrete action Q network and the continuous parameter policy network. Specifically, for the discrete action Q network loss function, it is constructed in the form of mean square error, aiming to minimize the target loss: in, is the n-step objective function, expressed as: in, is the discount factor, and They are the discrete action target Q network parameters and the continuous parameter strategy target network parameters respectively; For the continuous parameter policy network loss function, the goal is to maximize the Q-value estimate, defined as: .

[0026] Step 3: According to the lane with the highest priority selected by the first layer of reinforcement learning, determine the coordination lane and the conflict lane; the coordination lane is a lane combination that does not conflict with the highest priority lane, and the conflict lane is a lane combination that conflicts with the highest priority lane. Figure 2 As shown in the figure, assuming that the SN (South to North) lane is the highest priority lane, the NS (North to South), EN (East to North) and SE (South to East) lanes are coordination lanes, while other left-turn and through lanes are conflict lanes; during the duration of the passage, vehicles on the coordination lanes can coordinate to pass through the intersection, while vehicles in the conflict lanes need to stop and wait for the next round of optimization;

[0027] Step 4: According to the traffic volume of each coordinated lane, the proportion of travel duration is allocated respectively to maximize the use of travel duration to optimize traffic flow efficiency; The specific calculation formula for the weight of the coordinated lanes is: in, To coordinate the number of lanes, and Respectively Article and The number of waiting vehicles in the detection zone on the lane, and Respectively Article and The future predicted traffic flow for each lane is calculated based on the arrival time of vehicles, which follows a Poisson distribution.

[0028] Step 5: Construct the second-layer deep reinforcement learning algorithm model, using the DQN algorithm. During the first-layer passage duration, reinforcement learning is used to determine the passage order of vehicles on the coordinated lane, aiming to solve the trajectory conflict problem of vehicles on the coordinated lane. The specific state space, action space, reward function, network architecture and loss function are designed as follows: Step 5.1: State Space Design The gridding method is used to discretize the intersection. In the second-layer deep reinforcement learning algorithm model, the focus is on the traffic order of vehicles on the coordination lane. Therefore, the state space only represents the state information of the vehicles on the coordination lane, and is described by the matrices of the position, expected position, speed and waiting time of the vehicles on the coordination lane. Specifically, the position matrix is ​​used to indicate whether each grid is occupied by a vehicle, and the expected position matrix represents the grid that the vehicle will occupy in the future, which is used to reflect the conflict relationship between vehicles. When there is a vehicle in the grid, the speed matrix records the speed value of the vehicle. The waiting time matrix records the waiting time value of the vehicle in the grid. This design can accurately represent the vehicle state of the coordination lane and provide a basis for optimizing the traffic order. Step 5.2: Action Space Design The response action is designed to coordinate the traffic sequence of vehicles on the lane, expressed as: Total candidate action combinations; based on this, the reinforcement learning algorithm identifies the optimal pass sequence combination; Step 5.3: Reward function design The reward function of the second layer is defined to minimize the waiting time and passing time of the coordinated vehicles, expressed as: in, Indicates penalty coordination vehicle waiting time, It means to punish the passing time of coordinated vehicles, aiming to make coordinated vehicles pass the intersection as quickly as possible; Indicates Coordinate the passing time of vehicles, Indicates the passing time threshold, , represents the weight coefficient; Step 5.4: Network architecture design The DQN algorithm model structure includes a Q-value network and a target network. Consistent with the first-layer algorithm model, CNN is used as a feature extractor. Specifically, the Q-value network takes the current state as input, and after processing by the convolution layer, the fully connected layer and the activation function, outputs the optimal Q value of the action. The target network is used to prevent the Q value from updating too quickly, thereby avoiding training instability. Specifically, a soft update method is adopted, and the parameters of the target network are synchronized with the Q-value network once every N steps, that is: in, is the target network parameter, is the Q value network parameter; Step 5.5: Loss function design The loss function is constructed in the form of mean squared error, which aims to reduce the loss with the n-step objective function: .

[0029] Step 6: Under the high-level decision-making of the traffic order based on two-layer reinforcement learning, priority vehicles pass through the intersection at the maximum safe speed, while conflicting vehicles need to slow down and stop and wait in front of the conflicting position.

[0030] In this step, the CAV vehicle with priority in the coordination lane may encounter the HV driving freely in other coordination lanes during driving. To deal with this situation, the present invention adopts an event-triggered re-decision mechanism: when it is retrieved that the priority CAV vehicle and the HV vehicle may occupy the intersection conflict point at the same time, the system will readjust the maximum safe speed of the vehicles passing in the coordination lane according to the estimated time for the priority CAV and HV to reach the intersection conflict point. This maximum safe speed can ensure that the priority CAV vehicle and the HV vehicle do not collide and pass the intersection conflict area safely.

[0031] Step 7: Through continuous cyclic training and iteration of the two-layer deep reinforcement learning algorithm model, the model parameters are gradually optimized, and finally the optimal convergence model is obtained; this model can effectively respond to various traffic conditions and vehicle behaviors, and improve the traffic efficiency and safety of intersections.

Claims

1. A hybrid traffic flow collaborative control method based on double-layer parameterized deep reinforcement learning, characterized in that: The following steps are involved: Construct a road model for mixed-flow intersections without signals and divide the intersection area into functional areas; Based on the original multi-channel parameterized deep Q network (MPDQN) algorithm, the first-layer deep reinforcement learning algorithm model (CNN-MPDQN) is constructed in combination with the convolutional neural network (CNN). The high-dimensional spatial characteristics of traffic at the intersection are effectively extracted through CNN. The core task of the CNN-MPDQN algorithm is to select a lane with the highest priority from multiple lanes and determine the duration of the lane. Based on the lane with the highest priority selected by the first layer of reinforcement learning, the coordination lane and the conflicting lane are determined; during the duration of the passage, vehicles on the coordination lane can coordinate to pass through the intersection, while vehicles in the conflicting lane need to stop and wait for the next round of optimization; According to the traffic volume of each coordinated lane, the proportion of travel duration is allocated respectively to maximize the use of travel duration to optimize traffic flow efficiency; Construct a second-layer deep reinforcement learning algorithm model, using the deep Q network (DQN) algorithm. During the first-layer passage duration, reinforcement learning is used to determine the passage order of vehicles on the coordination lane, aiming to solve the trajectory conflict problem of vehicles on the coordination lane. Under the high-level decision-making of traffic order based on two-layer reinforcement learning, priority vehicles pass through the intersection at the maximum safe speed, while conflicting vehicles need to slow down and stop and wait in front of the conflicting position; Through continuous cyclic training and iteration of the two-layer deep reinforcement learning algorithm model, the model parameters are gradually optimized, and finally the optimal convergence model is obtained; this model can effectively respond to various traffic conditions and vehicle behaviors, and improve the traffic efficiency and safety of intersections.

2. The hybrid traffic flow collaborative control method based on double-layer parameterized deep reinforcement learning according to claim 1 is characterized in that: The road model constructed is a four-way three-lane intersection, with left-turn, straight and right-turn lanes in each direction, and the right-turn lane is uncontrolled; the intersection area is divided into a detection zone and a conflict zone; the section between the maximum communication range and the stop line is the detection zone, and the conflict zone is an area where a vehicle may conflict with vehicles in other directions after crossing the stop line; autonomous driving vehicles (CAV) and human-driven vehicles (HV) coexist on the road, and vehicles that enter the detection zone and communicate with the intersection central processor are identified as CAVs, while vehicles that can be detected by the central processor but do not communicate are identified as HVs; all vehicles travel according to pre-planned trajectories.

3. The hybrid traffic flow collaborative control method based on double-layer parameterized deep reinforcement learning according to claim 2 is characterized by: The driving behaviors of HVs are mainly divided into two types: following the vehicle in front and free driving without a vehicle in front. When two free HVs from conflicting directions arrive at the intersection at the same time, they only need to follow the right-hand rule to pass the intersection. If there is a vehicle in front of the HV, the HV needs to maintain a safe distance to avoid collision. Therefore, the HV The target speed at the moment should meet the following conditions: in, is the maximum speed limit of the road, is the maximum acceleration, is the simulation update step size, For safe following speed, The formula is: in, is the speed of the preceding vehicle at time t, is the distance between vehicles at time t, for the reaction time of human drivers; In addition, the introduction of random factors , represents the uncertainty of human driving behavior, and the final following speed is: 。 4. The hybrid traffic flow collaborative control method based on double-layer parameterized deep reinforcement learning according to claim 1 is characterized in that: The first-layer deep reinforcement learning algorithm model adopts the CNN-MPDQN algorithm. The specific state space, action space, reward function, network architecture and loss function are designed as follows: 4.1 State Space Design In order to intuitively present the current traffic conditions at the intersection, the gridding method is used to discretize the intersection, where the side length of the grid is set to the lane width; the state space is represented by a matrix of the position, speed and waiting time of the vehicle on the lane; specifically, the position matrix is ​​used to indicate whether each grid is occupied by a vehicle, and 1 or 2 is used to indicate the presence of a CAV or HV in the position matrix, respectively. When a vehicle occupies more than half of the grid area, the corresponding grid is marked as 1 or 2, otherwise it is marked as 0; the speed matrix records the speed value of the vehicle when there is a vehicle in the grid; the waiting time matrix records the waiting time value of the vehicle in the grid; This method can effectively reflect the traffic flow and vehicle status at the intersection; 4.2 Action Space Design The parameterized action space consists of a set of discrete actions, , where each discrete action corresponds to a continuous parameter, ; Specifically, the discrete action is represented by all candidate lanes. After selecting one of the lanes as the highest priority lane, the travel duration of the lane is defined as its corresponding continuous parameter; the action space can be expressed as: in, Yes all A continuous set of parameters; 4.3 Reward Function Design Design a multi-objective reward function that comprehensively considers efficiency, fairness, and HV , which is defined as: in, It indicates the waiting time of vehicles that are forced to stop and wait due to conflict with vehicles in the highest priority lane; Designed to increase the chances of vehicles becoming coordinated vehicles, allowing them to coordinate through the intersection rather than becoming conflicting vehicles and being forced to wait; Aims to increase the chances of more HVs becoming priority vehicles, rather than becoming conflicting or coordination vehicles that interfere with the safe passage of HVs and the coordination of second-tier CAV vehicles; in, Indicates Vehicle waiting time, Indicates the waiting time threshold, represents the number of conflicting vehicles, represents the number of coordinated vehicles, represents the ratio of CAVs in the highest priority lane, , , represents the weight coefficient; 4.4 Network Architecture Design The CNN-MPDQN algorithm model structure includes a continuous parameter strategy network, a discrete action Q network and a target network; CNN is used to effectively extract the high-dimensional spatial characteristics of traffic at intersections; specifically, the continuous parameter strategy network takes the current state as input, and after processing by the convolution layer, the fully connected layer and the activation function, outputs the optimal vector of all continuous parameters in the action space; then the vector is separated so that each discrete action only inputs the parameters related to it, thereby reducing the influence of other continuous parameters on the Q value estimation; in order to keep the length of the continuous action parameter vector consistent, the irrelevant continuous parameters are set to 0, and then the current state vector processed by the convolution layer and the separated continuous action are spliced ​​to form The vector is input into the discrete action Q network, and after passing through the fully connected layer and activation function, the output Matrix, and extract the diagonal Q value to form a vector, and finally get the optimal Q value of the discrete action Q network; 4.5 Loss Function Design The loss function consists of two parts: the discrete action Q network and the continuous parameter policy network. Specifically, for the discrete action Q network loss function, it is constructed in the form of mean square error, aiming to minimize the target loss: in, is the n-step objective function, expressed as: in, is the discount factor, and They are the discrete action target Q network parameters and the continuous parameter strategy target network parameters respectively; For the continuous parameter policy network loss function, the goal is to maximize the Q-value estimate, defined as: 。 5. The hybrid traffic flow collaborative control method based on double-layer parameterized deep reinforcement learning according to claim 1 is characterized in that: The coordination lanes are a lane combination that has no trajectory conflict with the highest priority lane, and the conflicting lanes are a lane combination that has a trajectory conflict with the highest priority lane.

6. The hybrid traffic flow collaborative control method based on double-layer parameterized deep reinforcement learning according to claim 1 is characterized by: The proportion of the travel duration is allocated according to the traffic volume of each coordinated lane. The specific calculation formula for the weight of the coordinated lanes is: in, To coordinate the number of lanes, and Respectively Article and The number of waiting vehicles in the detection zone on the lane, and Respectively Article and The future predicted traffic flow for each lane is calculated based on the arrival time of the vehicles, which follows a Poisson distribution.

7. The hybrid traffic flow collaborative control method based on double-layer parameterized deep reinforcement learning according to claim 1 is characterized by: The second-layer deep reinforcement learning algorithm model adopts the DQN algorithm. During the first-layer passage duration, the reinforcement learning algorithm determines the passage order of vehicles on the coordinated lanes. The specific state space, action space, and reward function of the model are designed as follows: 7.1 State Space Design The gridding method is used to discretize the intersection. In the second-layer deep reinforcement learning algorithm model, the focus is on the traffic order of vehicles on the coordination lane. Therefore, the state space only represents the state information of the vehicles on the coordination lane, and is described by the matrices of the position, expected position, speed and waiting time of the vehicles on the coordination lane. Specifically, the position matrix is ​​used to indicate whether each grid is occupied by a vehicle, and the expected position matrix represents the grid that the vehicle will occupy in the future, which is used to reflect the conflict relationship between vehicles. When there is a vehicle in the grid, the speed matrix records the speed value of the vehicle. The waiting time matrix records the waiting time value of the vehicle in the grid. This design can accurately represent the vehicle state of the coordination lane and provide a basis for optimizing the traffic order. 7.2 Action Space Design The response action is designed to coordinate the traffic sequence of vehicles on the lane, expressed as: Total candidate action combinations; based on this, the reinforcement learning algorithm identifies the optimal pass sequence combination; 7.3 Reward Function Design The reward function of the second layer is defined to minimize the waiting time and passing time of the coordinated vehicles, expressed as: in, Indicates penalty coordination vehicle waiting time, It means to punish the passing time of coordinated vehicles, aiming to make coordinated vehicles pass the intersection as quickly as possible; Indicates Coordinate the passing time of vehicles, Indicates the passing time threshold, , represents the weight coefficient; 7.4 Network Architecture Design The DQN algorithm model structure includes a Q-value network and a target network. Consistent with the first-layer algorithm model, CNN is used as a feature extractor. Specifically, the Q-value network takes the current state as input, and after processing by the convolution layer, the fully connected layer and the activation function, outputs the optimal Q value of the action. The target network is used to prevent the Q value from updating too quickly, thereby avoiding training instability. Specifically, a soft update method is adopted, and the parameters of the target network are synchronized with the Q-value network once every N steps, that is: in, is the target network parameter, is the Q value network parameter; 7.5 Loss Function Design The loss function is constructed in the form of mean squared error, aiming to reduce the loss with the n-step objective function: 。 8. The hybrid traffic flow collaborative control method based on double-layer parameterized deep reinforcement learning according to claim 1 is characterized by: Under the high-level decision of traffic order based on two-layer reinforcement learning, vehicles with priority pass through the intersection at the maximum safe speed, while conflicting vehicles need to slow down and stop and wait in front of the conflicting position; specifically, CAV vehicles with priority in the coordination lane may encounter HVs traveling freely in other coordination lanes during driving; to deal with this situation, the method adopts an event-triggered re-decision mechanism.

9. The hybrid traffic flow collaborative control method based on double-layer parameterized deep reinforcement learning according to claim 8 is characterized by: The method adopts an event-triggered re-decision mechanism. Specifically, when it is retrieved that a priority CAV vehicle and an HV vehicle may occupy an intersection conflict point at the same time, the system will readjust the maximum safe speed of vehicles passing in the coordination lane according to the estimated time for the priority CAV and HV to arrive at the intersection conflict point. The maximum safe speed can ensure that the priority CAV vehicle and the HV vehicle do not collide and pass through the intersection conflict area safely.

Images