A hierarchical regional coordinated signal control method based on reinforcement learning

Abstract

A hierarchical regional coordinated signal control method based on reinforcement learning, the present invention relates to the field of intelligent transportation technology, the control method comprises the following steps: determining the collection and calculation cycle of the intersection terminal, the intersection terminal collects the vehicle data and signal state l at a distance of X meters from the intersection at time t1 through a camera, and calculates the mixed state s of the lane lane and the mixed state of the intersection intersection , where the vehicle data includes the vehicle position x i and vehicle speed v i The advantages of the present invention are: reasonable state representation of single-vehicle data collected by cameras is used as the basis for decision-making at the intersection, which improves the scientific nature of the decision. The global traffic state is predicted by using GCN at the regional level, and the target is sent to the regional end, further enriching the information content of intersection decision-making.

Description

Technical Field

[0001] The invention relates to a layered regional coordinated signal control method based on reinforcement learning, and relates to the technical field of intelligent transportation. Background Art

[0002] In the field of modern urban traffic management, traffic congestion, traffic accidents and emergency rescue are becoming increasingly complex, and innovative methods are needed to solve these problems. Traditional traffic signal control and vehicle coordination methods are often difficult to adapt to the ever-changing traffic conditions, especially when random traffic accidents and irrational driving behaviors occur, the problem becomes more complicated.

[0003] Reinforcement learning algorithms are a very promising approach to addressing these challenges. Reinforcement learning algorithms enable traffic systems to continuously learn and optimize decisions in real-time environments, and control traffic lights to make accurate phase changes. Reinforcement learning algorithms can be trained in virtual environments and deployed in real environments, effectively reducing training costs and time. Reinforcement learning algorithms can simultaneously control the coordinated movement of multiple agents, effectively solving the problem of coordinated control of complex vehicle conditions at intersections. However, existing reinforcement learning intersection control methods have low accuracy, poor real-time control effects, and poor coordination between intersections. To this end, we propose a hierarchical regional coordinated signal control based on reinforcement learning.

[0004] The comparison with the existing technology is as follows:

[0005] Compared with the patent CN 113299079 B "A regional intersection signal control method based on PPO and graph convolutional neural network"

[0006] 1. In patent CN 113299079 B, the layered concept involved reflects that the control model has an upper and lower two-layer structure. The lower layer is a single-point intersection control model based on the PPO reinforcement learning algorithm, and the upper layer is a regional coordinated control model based on a graph convolutional neural network. However, the upper layer output is the predicted number of vehicles as an action. The time step of this method is 5s, but the action space will increase with the time step and the number of intersections. It depends on an extremely high frequency of information interaction, which restricts its use in areas with poor information interaction. The layering of this patent is mainly reflected in the state of the upper region after the state prediction output time length. The lower intersection end makes decisions based on the predicted state and the current state. The state change of the regional layer is obtained from the environment. The training difficulty does not change significantly with the spatial and temporal dimensions, and can adapt to lower information interaction situations.

[0007] 2. In patent CN 113299079 B, the state of a single intersection is represented by the number of vehicles in the lane and the phase of the intersection, which has strong operability, but it is difficult for a single traffic state to fully express the real state of the intersection. To address this problem, the existing equipment, the intersection camera, uses the YOLO algorithm to obtain single-vehicle-level data and count data, namely vehicle speed, position, number of vehicles on the road and number of parking vehicles. The real state of the intersection is described in detail through single-vehicle-level data, and the vehicle position is squared to punish queuing.

[0008] 3. In patent CN 113299079 B, the graph convolution neural network used in the upper layer mainly increases the extraction of spatial information by increasing the number of graph convolution layers, but no specific design is made for the graph convolution itself. This patent considers the surrounding state of the intersection in combination with the operation of the intersection, and uses the collected number of road vehicles and parking numbers in the graph convolution to calculate the intersection efficiency to correct the self-adjacency matrix, so that when using graph convolution for spatial feature extraction, the impact of intersection efficiency on the state is further considered.

[0009] In technical comparison with patent CN 114627657 A "An adaptive traffic signal control method based on graph deep reinforcement learning"

[0010] 1. Patent CN 114627657 A uses a two-layer graph convolutional neural network to extract global information for intersection decision-making, which improves the effectiveness of decision-making information, but does not make a specific design for the graph convolution itself. This patent considers the surrounding state of the intersection in combination with the operation of the intersection. In the graph convolution, the number of collected road vehicles and parking numbers are used to calculate the intersection efficiency to correct the self-adjacency matrix, so that when using graph convolution for spatial feature extraction, the impact of intersection efficiency on the state is further considered.

[0011] 2. In patent CN 114627657 A, the action output by the intersection is a fixed-frequency phase switching, which can effectively cope with emergencies at intersections. However, in reality, the continuous realization of the current phase is also a factor that affects the driver's driving behavior. To address this problem, this patent allows the region to output a predicted value of the duration of the current phase based on the global traffic trend at the regional layer. This value will not affect the actual phase execution. The training of the predicted value can be used to update the prediction model through the difference between the predicted value and the actual duration.

[0012] 3. In patent CN 114627657 A, single-vehicle-level data is used for state expression. The traffic state of the intersection is expressed by queue length, number of vehicles and average speed, which can more comprehensively reflect the real situation of the intersection. However, the existing data is not fully applied. To address this problem, the state expression function is constructed using vehicle speed and position to make full use of the collected single-vehicle-level data. At the same time, the vehicle position is squared to punish the queue. Summary of the invention

[0013] In response to the above problems, the present invention proposes a hierarchical regional coordinated signal control method based on reinforcement learning. The vehicle-level data collected by the camera is used to reasonably represent the state as the basis for decision-making at the intersection, which improves the scientific nature of the decision. The regional layer uses GCN to predict the global traffic state, and sends targets to the regional end, further enriching the decision-making information content at the intersection.

[0014] To achieve the above object, the technical solution adopted by the present invention is:

[0015] A hierarchical regional coordination signal control method based on reinforcement learning, the control method comprising the following steps:

[0016] Step 1: Determine the collection and calculation cycle of the intersection terminal, select one of the time periods of 5s, 10s, and 20s. The intersection terminal collects vehicle data, counting information, and signal status l at a distance of X meters from the intersection at time t1 through a camera, and calculates the mixed state s of the lanes. lane and the mixed state of the intersection intersection , where the vehicle data includes the vehicle position x i and vehicle speed v i , the counting information includes the total number of vehicles n car and the total number of parking lots n stop , the mixed state s of the intersection intersection Mixed state of lanes including 8 import lanes lane ;

[0017] Step 2: The regional terminal generates the current traffic graph G1(A,H) of the regional traffic by receiving the mixed state of the intersection terminal at time t1, where A is the adjacency matrix of the intersection, H is the mixed traffic state matrix containing the mixed state of each intersection, and the self-adjacency matrix is ​​calculated in combination with the road network state. And predict the mixed state after 1T time to generate the predicted traffic graph G2 and the signal duration L of this intersection;

[0018] Step 3: The intersection terminal receives the mixed state s of the predicted intersection in the predicted traffic graph G2 at time t1 intersection_predict ,Use the reinforcement learning controller at the intersection to generate control strategy data and store the interaction strategy in the experience pool;

[0019] Step 4: Calculate the regional terminal for predicting the loss value, including the difference between the actual traffic map G3 and the predicted traffic map G2 after 1T, and the difference between the duration L of the predicted signal and the cumulative duration of the intersection breaking phase remaining unchanged;

[0020] Step 5: Use the DQN algorithm for intersection-side reinforcement learning, and set the reward to maintain high consistency with the mixed state at the intersection;

[0021] Step 6: Based on the signal duration L at the intersection, reasonably guide the vehicle speed.

[0022] As a further improvement of the present invention, the lane mixing state S of the intersection in the first step lane Including the current speed v of the lane vehicle i and distance x i , the image data collected in real time by the monitoring terminal at the intersection is obtained through the YOLO algorithm, and its operating formula is:

[0023]

[0024] where x i represents the vehicle flow, v i represents the vehicle speed, and the actual x i Value and v i The value is entered into the formula for calculation.

[0025] As a further improvement of the present invention, in the second step, the self-adjacency matrix Considering that the counting information in the first step includes the total number of vehicles n car and the total number of parking lots n stop , its operating formula is as follows:

[0026]

[0027]

[0028] in is the adjusted identity matrix, I is the identity matrix, and the subscript ii represents the element in the i-th row and i-th column of I.

[0029] As a further improvement of the present invention, the DQN algorithm mentioned in step 6 is a hierarchical regional coordinated signal control based on reinforcement learning, and the training process of DQN includes the following steps:

[0030] S1, experience replay, stores the agent's experience in the environment in the experience replay buffer, randomly extracts these experiences for training, and reduces the correlation between data;

[0031] S2. Target network. DQN uses two neural networks: one is the evaluation network Q-network, which is used to select actions and calculate Q values; the other is the target network, which is used to calculate the target Q value;

[0032] S3, ε-greedy strategy, the agent will explore with a certain probability ε and choose non-optimal actions in order to explore more state space.

[0033] As a further improvement of the present invention, the speed in step 1 is expressed in meters per second (m / s), and the distance is the distance from the vehicle to the traffic light (m).

[0034] As a further improvement of the present invention, in order to grasp the changing trend of the entire road network flow, including the prediction time, the regional terminal in step 2 can predict the problems of the adjacent intersections affecting the traffic conditions at the intersection.

[0035] As a further improvement of the present invention, the current traffic map G1 and the predicted traffic map G2 generated by the regional terminal in the second step and the third step, the duration L of the intersection does not conflict with the current signal state l, and the duration L is to help the driver better adjust the current vehicle speed, and is not a specific execution time.

[0036] By adopting the above technical solution, compared with the prior art, the beneficial effects of the present invention are:

[0037] 1. Determine the collection and calculation cycle of the intersection terminal, select one of the time periods of 5s, 10s, and 20s. The intersection terminal collects vehicle data, counting information, and signal status l at a distance of X meters from the intersection at time t1 through a camera, and calculates the mixed state s of the lane lane and the mixed state of the intersection intersetion , where the vehicle data includes the vehicle position x i and vehicle speed v i , the counting information includes the total number of vehicles n car and the total number of parking lots n stop , the mixed state s of the intersection intersetion Mixed state of lanes including 8 import lanes lane ;

[0038] 2. The data collected by the intersection end is transmitted to the regional terminal in real time. The regional terminal generates the current traffic map G1 of the regional traffic and calculates the self-adjacency matrix based on the road network status. The mixed state after 1T time is predicted to generate the predicted traffic map G2 and the signal duration L of this intersection, which is beneficial for the intersection to grasp the traffic change trend from the road network level and generate phase actions suitable for this road. At the same time, the regional end will generate a phase duration prediction to facilitate drivers to make appropriate driving behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a flow chart in an embodiment of the present invention;

[0040] Figure 2 Schematic diagram of the method steps in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The specific embodiments of the present invention are further described below in conjunction with the accompanying drawings. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0042] Example 1, please refer to the attached Figure 1-2 The present invention provides a technical solution: a hierarchical regional coordinated signal control based on reinforcement learning: the control method comprises the following steps:

[0043] Step 1: Determine the collection and calculation cycle of the intersection terminal, select one of the time periods of 5s, 10s, and 20s. The intersection terminal collects vehicle data, counting information, and signal status l at a distance of X meters from the intersection at time t1 through a camera, and calculates the mixed state s of the lanes. lane and the mixed state of the intersection intersection , where the vehicle data includes the vehicle position x i and vehicle speed v i , the counting information includes the total number of vehicles n car and the total number of parking lots n stop , the mixed state s of the intersection intersection Mixed state of lanes including 8 import lanes lane ;

[0044] Step 2: The regional terminal generates the current traffic graph G1(A,H) of the regional traffic by receiving the mixed state of the intersection terminal at time t1, where A is the adjacency matrix of the intersection, H is the mixed traffic state matrix containing the mixed state of each intersection, and the self-adjacency matrix is ​​calculated in combination with the road network state. And predict the mixed state after 1T time to generate the predicted traffic graph G2 and the signal duration L of this intersection;

[0045] Step 3: The intersection terminal receives the mixed state s of the predicted intersection in the predicted traffic graph G2 at time t1 intersection_predict ,Use the reinforcement learning controller at the intersection to generate control strategy data and store the interaction strategy in the experience pool;

[0046] Step 4: The measurement area terminal is used to predict the loss value, which includes the gap between the actual traffic map G3 and the predicted traffic map G2 after 1T time, and the gap between the duration L of the predicted signal and the cumulative duration when the intersection disconnection phase remains unchanged.

[0047] Please refer to the attached Figure 2 , the mixed lane state s of the intersection in the first step lane Including the current speed v of the lane vehicle i and distance x i , the image data collected in real time by the monitoring terminal at the intersection is obtained through the YOLO algorithm, and its operating formula is:

[0048]

[0049] where x i It can represent the vehicle flow, v i Can represent the vehicle speed, the actual x i Value and v i The value is entered into the formula for calculation;

[0050] Please refer to the attached Figure 1-2 , in the second step, the self-adjacency matrix Considering that the counting information in the first step includes the total number of vehicles n car and the total number of parking lots n stop , its operating formula is as follows:

[0051]

[0052]

[0053] in is the adjusted identity matrix, I is the identity matrix, and the subscript ii represents the element in the i-th row and i-th column of I.

[0054] The control method further comprises the following steps:

[0055] Step 5: Use the DQN algorithm for intersection-side reinforcement learning, and set the reward to maintain high consistency with the mixed state at the intersection;

[0056] Step 6: Based on the signal duration L at the intersection, reasonably guide the vehicle speed;

[0057] The DQN algorithm mentioned in step 6 is an algorithm based on deep reinforcement learning. The training process of DQN includes the following steps:

[0058] S1, experience replay, stores the agent's experience in the environment in the experience replay buffer, randomly extracts these experiences for training, and reduces the correlation between data;

[0059] S2. Target network. DQN uses two neural networks: one is the evaluation network (Q-network), which is used to select actions and calculate Q values; the other is the target network, which is used to calculate the target Q value;

[0060] S3, ε-greedy strategy, the agent will explore with a certain probability ε and choose non-optimal actions in order to explore more state space.

[0061] In this embodiment, the phase selection can set a 4-phase intersection, where:

[0062] At the intersection, information is collected at a frequency of 10 seconds, including the vehicle position information of 8 lanes x i , vehicle speed information v i , the phase information of this intersection l k , the traffic mixed state is calculated at the intersection, with the jth lane s lane_j :

[0063]

[0064] Among them, x i is the position of car i in lane j, v i is the speed of vehicle i in lane j, L is the detection length of the intersection or the length of the road, vmax is the maximum speed allowed by the road, and the data format of the kth intersection segment after processing is as follows:

[0065] s intersection =[s lane_1 ,s lane_2 ,...,s lane_8 ,l k ]

[0066] The data is uploaded to the regional end, which receives the data from the intersection and generates the timing information Light_time to generate the traffic map G1. The related data adjacency matrix A1 graph node feature H1 is input into the GCN-based road network predictor to generate the predicted traffic map G2, and the data s of each intersection in the predicted node feature H2 is obtained. intersectin_predict_j Return to each intersection.

[0067] Take a 2*2 traffic road network as an example, the upper left corner is 1, the upper right corner is 2, the lower left corner is 3, the lower right corner is 4, the adjacency matrix A, shape (2,2), the graph node feature H1, shape (4,9);

[0068] H1=[s intersection_1 ,s intersection_2 ,...,s intersection_9 ]

[0069] Supplementary explanation for output H2 H2[k][9] is the remaining time of the current phase at the k-th intersection at time t. k_t = = l k_t-1 , that is, the phase remains unchanged, then H 2_t [k][9]=H 2_t-1 [k][9]-10s, when l k_t ! = l k_t-1 , that is, the phase changes, at time t H 2_t [k][9] will be recalculated by GCN.

[0070] When in use, by determining the collection and calculation cycle of the intersection terminal, select one of the time periods of 5s, 10s and 20s. At this time, the intersection terminal collects the vehicle data and signal state l at a distance of X meters from the intersection at time t1 through the camera, and calculates the mixed state s of the lane. lane and the mixed state of the intersection intersection , where the vehicle data includes the vehicle position x i and vehicle speed v i , the mixed state s of the intersection intersection Mixed state of lanes including 8 import lanes lane , which can better describe the dynamic conditions of the intersection, reduce the state dimensions required for intersection control, and facilitate the use of signal control over a wider range.

[0071] Example 2, please refer to the attached Figure 1-2 The present invention provides a technical solution: a hierarchical regional coordinated signal control based on reinforcement learning, the control method comprising the following steps:

[0072] Step 1: Determine the collection and calculation cycle of the intersection terminal, select one of the time periods of 5s, 10s, and 20s. The intersection terminal collects vehicle data, counting information, and signal status l at a distance of X meters from the intersection at time t1 through a camera, and calculates the mixed state s of the lanes. lane and the mixed state of the intersection intersection , where the vehicle data includes the vehicle position x i and vehicle speed v i , the counting information includes the total number of vehicles n car and the total number of parking lots n stop , the mixed state s of the intersection intersection Mixed state of lanes including 8 import lanes lane ;

[0073] Step 2: The regional terminal generates the current traffic graph G1(A,H) of the regional traffic by receiving the mixed state of the intersection terminal at time t1, where A is the adjacency matrix of the intersection, H is the mixed traffic state matrix containing the mixed state of each intersection, and the self-adjacency matrix is ​​calculated in combination with the road network state. And predict the mixed state after 1T time to generate the predicted traffic graph G2 and the signal duration L of this intersection;

[0074] Step 3: The intersection terminal receives the mixed state s of the predicted intersection in the predicted traffic graph G2 at time t1 intersection_predict ,Use the reinforcement learning controller at the intersection to generate control strategy data and store the interaction strategy in the experience pool;

[0075] Step 4: The measurement area terminal is used to predict the loss value, which includes the gap between the actual traffic map G3 and the predicted traffic map G2 after 1T time, and the gap between the duration L of the predicted signal and the cumulative duration when the intersection disconnection phase remains unchanged.

[0076] Please refer to the attached Figure 1 -Attached Figure 2 , the speed in step one is expressed in meters per second (m / s), the distance is the distance from the vehicle to the traffic light in meters (m), in order to grasp the changing trend of the traffic flow of the entire road network, including the prediction time, the regional terminal can predict the problems of nearby intersections affecting the traffic conditions of this intersection, the current traffic map G1 and the predicted traffic map G2 generated by the regional terminal in the second and third steps, the duration L of the intersection does not conflict with the current signal state l, the duration L is to help the driver better adjust the current speed, and it is not a specific execution time.

[0077] In this embodiment, GCN is set to 2 or 3 layers, and the training loss is, H true The actual state of the road network is obtained by the next perception of the regional end, and H2 is predicted by the GCN predictor based on H1 at the regional end:

[0078]

[0079]

[0080] Among them, when l k_t = = l k_t-1 When the phase remains unchanged, the result is H2(9)_K 2 , when l k_t ! = l k_t-1 , that is, the phase changes and the result is 0.

[0081] The regional layer prediction formula is as follows:

[0082]

[0083] Adjacency matrix A, shape (2,2):

[0084] 0 1 1 0 1 0 0 1 1 0 0 1 0 1 1 0

[0085] Where A: adjacency matrix of graph structure;

[0086] There is a self-connected adjacency matrix;

[0087]

[0088] Adjusted identity matrix;

[0089]

[0090] I: identity matrix, subscript ii represents the element in row i and column i in I;

[0091] The degree matrix of the adjacency matrix with self-connection;

[0092]

[0093] H: characteristics of graph nodes;

[0094] l: number of neural network layers;

[0095] H true [1:8] refers to the first eight parameters related to the intersection status, H true [9] is the predicted remaining time. The intersection receives the s returned by the area end. intersection_predict_k , and s lane_k Contact us to get k After that, the input is based on the DQN intersection-side controller to generate a suitable action. The action is set to four phases: north-south straight, north-south left turn, east-west straight and east-west left turn, and the north-south straight, north-south left turn, east-west straight and east-west left turn are marked as a1, a2, a3 and a4 respectively.

[0096] The regional layer predicts the H value in the feature matrix of G1, which is the main update, and the regional layer then outputs the corresponding s in G2 intersection_predict Feedback to each intersection.

[0097] Vehicle speed guidance: Based on the predicted phase length returned by the regional end, corresponding speed guidance is issued for vehicles within the phase.

[0098] Specifically, the data collected at the intersection is transmitted to the regional terminal in real time. The regional terminal generates the current traffic map G1 of the regional traffic, and predicts the mixed state after 1T time to generate the predicted traffic map G2 and the signal duration L of this intersection. This is beneficial for the intersection to grasp the traffic change trend from a macro perspective at the road network level, and helps the intersection to generate phase actions suitable for this road. At the same time, the regional end will generate a phase duration prediction to facilitate the driver to make appropriate driving behavior.

[0099] Working principle:

[0100] Step 1: Determine the collection and calculation cycle of the intersection terminal, select one of the time periods of 5s, 10s, and 20s. The intersection terminal collects vehicle data, counting information, and signal status l at a distance of X meters from the intersection at time t1 through a camera, and calculates the mixed state s of the lanes. lane and the mixed state of the intersection intersection , where the vehicle data includes the vehicle position x i and vehicle speed v i , the counting information includes the total number of vehicles n car and the total number of parking lots n stop , the mixed state s of the intersection intersection Mixed state of lanes including 8 import lanes lane ;

[0101] Step 2: The regional terminal generates the current traffic graph G1(A,H) of the regional traffic by receiving the mixed state of the intersection terminal at time t1, where A is the adjacency matrix of the intersection, H is the mixed traffic state matrix containing the mixed state of each intersection, and the self-adjacency matrix is ​​calculated in combination with the road network state. And predict the mixed state after 1T time to generate the predicted traffic graph G2 and the signal duration L of this intersection;

[0102] Step 3: The intersection terminal receives the mixed state s of the predicted intersection in the predicted traffic graph G2 at time t1 intersection_predict ,Use the reinforcement learning controller at the intersection to generate control strategy data and store the interaction strategy in the experience pool;

[0103] Step 4: Calculate the regional terminal for predicting the loss value, including the difference between the actual traffic map G3 and the predicted traffic map G2 after 1T, and the difference between the duration L of the predicted signal and the cumulative duration of the intersection breaking phase remaining unchanged;

[0104] Step 5: Use the DQN algorithm for intersection-side reinforcement learning, and set the reward to maintain high consistency with the mixed state at the intersection;

[0105] Step 6: Based on the signal duration L at the intersection, reasonably guide the vehicle speed.

[0106] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A hierarchical regional coordination signal control method based on reinforcement learning, characterized in that: The control method comprises the following steps: Step 1: Determine the collection and calculation cycle of the intersection terminal, select one of the time periods of 5s, 10s, and 20s. The intersection terminal collects vehicle data, counting information, and signal status l at a distance of X meters from the intersection at time t1 through a camera, and calculates the mixed state s of the lanes. lane and the mixed state of the intersection intersection , where the vehicle data includes the vehicle position x i and vehicle speed v i , the counting information includes the total number of vehicles n car and the total number of parking lots n stop , the mixed state s of the intersection intersection Mixed state of lanes including 8 import lanes lane ; The mixed state of lanes at the intersection in step 1 s lane Including the current speed v of the lane vehicle i and distance x i , the image data collected in real time by the monitoring terminal at the intersection is obtained through the YOLO algorithm, and its operating formula is: where x i represents the vehicle flow, v i represents the vehicle speed, and the actual x i Value and v i The value is entered into the formula for calculation; Step 2: The regional terminal generates the current traffic graph G1(A,H) of the regional traffic by receiving the mixed state of the intersection terminal at time t1, where A is the adjacency matrix of the intersection, H is the mixed traffic state matrix containing the mixed state of each intersection, and the self-adjacency matrix is ​​calculated in combination with the road network state. And predict the mixed state after 1T time to generate the predicted traffic graph G2 and the signal duration L of this intersection; Step 3: The intersection terminal receives the mixed state s of the predicted intersection in the predicted traffic graph G2 at time t1 intersection_predict ,Use the reinforcement learning controller at the intersection to generate control strategy data and store the interaction strategy in the experience pool; Step 4: Calculate the regional terminal for predicting the loss value, including the difference between the actual traffic map G3 and the predicted traffic map G2 after 1T, and the difference between the duration L of the predicted signal and the cumulative duration of the intersection breaking phase remaining unchanged; Step 5: Use the DQN algorithm for intersection-side reinforcement learning, and set the reward to maintain high consistency with the mixed state at the intersection; Step 6: Based on the signal duration L at the intersection, reasonably guide the vehicle speed.

2. According to claim 1, a hierarchical regional coordinated signal control method based on reinforcement learning is characterized in that: The self-adjacency matrix in step 2 Considering the counting information in step 1, including the total number of vehicles n car and the total number of parking lots n stop , its operating formula is as follows: in is the adjusted identity matrix, I is the identity matrix, and the subscript ii represents the element in the i-th row and i-th column of I.

3. The hierarchical regional coordinated signal control method based on reinforcement learning according to claim 1, characterized in that: The DQN algorithm mentioned in step 6 is a hierarchical regional coordinated signal control based on reinforcement learning. The training process of DQN includes the following steps: S1, experience replay, stores the agent's experience in the environment in the experience replay buffer, randomly extracts these experiences for training, and reduces the correlation between data; S2. Target network. DQN uses two neural networks: one is the evaluation network Q-network, which is used to select actions and calculate Q values; the other is the target network, which is used to calculate the target Q value; S3, ε-greedy strategy, the agent will explore with a certain probability ε and choose non-optimal actions in order to explore more state space.

4. The hierarchical regional coordinated signal control method based on reinforcement learning according to claim 1, characterized in that: The speed in step 1 is expressed in meters per second (m / s), and the distance is the distance from the vehicle to the traffic light (m).

5. The hierarchical regional coordinated signal control method based on reinforcement learning according to claim 1, characterized in that: In the step 2, the regional terminal can predict the problem of the adjacent intersection affecting the traffic condition of the intersection in order to grasp the changing trend of the entire road network flow, including the prediction time.

6. The hierarchical regional coordinated signal control method based on reinforcement learning according to claim 1, characterized in that: In the current traffic map G1 and the predicted traffic map G2 generated by the regional terminal in step 2 and step 3, the duration L of the intersection does not conflict with the current signal state l. The duration L is to help the driver better adjust the current vehicle speed and is not a specific execution time.

Images