Single-intersection signal lamp phase control method and system based on FR-DDQN

Through the FR-DDQN-based method, combined with the fair reward function and deep Q network, the short-sightedness problem of reward mechanism in the existing traffic light control method is solved, and the traffic efficiency of intersections and the stability of signal light control is improved. Multiple fairness factors achieve more reasonable traffic light phase control.

CN119007463AActive Publication Date: 2024-11-22HUAQIAO UNIVERSITY +1

Patent Information

Application Number
CN202411036564.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-22
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

The existing traffic light control method based on reinforcement learning has short-sightedness in the reward mechanism, and failed to effectively consider the impact of current decisions on subsequent decisions and overall signal light control effects, and ignored fairness factors such as the average queuing length at the intersection and the utilization of green light time. comprehensive factors such as rate.

Method used

Using a single-intersection signal light phase control method based on FR-DDQN, the short-term and long-term impact of actions on the environment is evaluated by constructing an agent, an estimation network and a target network, combined with a fair reward function, including the average queue length and green light time utilization rate , travel time and red light waiting time, etc., the network parameters are updated using the ε-greedy strategy and gradient descent method to build a fair reward mechanism.

Benefits of technology

It improves the overall traffic efficiency of the intersection, enhances the stability and performance of signal light phase control, and takes into account multiple fairness factors, effectively improving the long-term traffic efficiency of traffic lights.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007463A_ABST
    Figure CN119007463A_ABST
Patent Text Reader

Abstract

The invention provides a single-intersection signal lamp phase control method and system based on FR-DDQN, and relates to the technical field of signal lamp phase control. Historical traffic data of an intersection are obtained, and a simulation environment and an intelligent agent are constructed; then constructing two networks with the same structure and different parameters; calculating an estimated Q value corresponding to each action in the current intersection state according to the estimation network, and selecting the action by adopting an epsilon-greedy strategy; calculating a reward value of a reward function after selecting an action, and storing the reward value into an experience playback set; calculating a target value from a next intersection state to a termination state according to the target network; and updating network parameters by adopting a loss function and a gradient descent method until the maximum simulation frequency is reached, and obtaining a signal lamp phase control strategy of the intersection. According to the method, the short-term influence of the selected action on the environment can be evaluated, the long-term influence of the action selection strategy on the environment can be controlled through a fairness reward mechanism, and the overall passing efficiency of the intersection within a period of time is effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of phase control of intersection signal lights, and in particular to a single intersection signal light phase control method and system based on FR-DDQN. Background Art

[0002] The transportation system is one of the most important infrastructures in modern cities, supporting the daily commuting and travel of millions of people. The normal operation of society now depends largely on an efficient transportation system. In the modern urban traffic structure, intersections are responsible for connecting and regulating traffic flows between adjacent roads. Therefore, strengthening the management and control of intersections can effectively alleviate congestion. Through intelligent signal light control, we can respond to different traffic conditions and improve the traffic efficiency of intersections.

[0003] Traffic light control is a Markov decision problem, which requires choosing the green light of a certain phase and the duration of the green light according to different traffic conditions, so as to improve the traffic efficiency of the intersection. Since the reinforcement learning method does not rely on prior knowledge, it can learn action strategies through continuous interaction with the environment. In the reinforcement learning algorithm, the learning goal of the agent directly affects the action decision. The reward mechanism of the existing traffic light control method based on reinforcement learning can only evaluate the short-term impact of the selection strategy, but ignores the long-term impact of the action strategy on the future moment. For example, the existing patent documents CN113380054A "A traffic light control method based on reinforcement learning" and CN111243299B "A single intersection signal control method based on 3DQN_PSER algorithm" are both traffic light control methods based on reinforcement learning. In terms of reward mechanism, they can only evaluate the short-term impact of the selection strategy, but ignore the long-term impact of the action strategy on the future moment, and do not consider the impact of fairness factors in the reward function. The existing patent document CN114299732B "A traffic light control method that considers travel time and fairness at the same time" constructs a single travel time strategy Q network and fairness strategy Q network, which only considers travel time and fairness, and does not consider other fairness factors, such as the average queue length at the intersection at the current moment, the average queue length at the intersection at the previous moment, and the green light time utilization rate of the current green light phase corresponding to the traffic direction and other comprehensive factors.

[0004] Therefore, in view of the short-sightedness problem existing in the reward mechanism, it is of practical significance to establish a reward mechanism that can control the traffic lights in the long term. In view of this, the applicant has specially proposed this application after studying the existing technology. Summary of the invention

[0005] The present invention aims to provide a single intersection signal light phase control method and system based on FR-DDQN to solve the problem that the existing method does not consider the impact of the current decision on subsequent decisions and the overall signal light control effect and is short-sighted.

[0006] In order to solve the above technical problems, the present invention is implemented through the following technical solutions:

[0007] A single intersection signal light phase control method based on FR-DDQN, comprising:

[0008] S1, obtain historical traffic data of the intersection;

[0009] S2, constructing an intelligent agent, the intelligent agent includes a state space, an action space and a fairness reward function; wherein the state space is the vehicle position distribution and speed on the entrance lane; the action space is all non-conflicting phases of the intersection; the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of the phase, the average travel time, the green light duration penalty item, the red light waiting time penalty item and the traffic condition reward and penalty items of each direction;

[0010] S3, constructing two networks with the same structure but different parameters, namely an estimation network and a target network, based on the historical traffic data and the intelligent agent; the estimation network calculates the estimated Q value corresponding to each action in the current intersection state; the target network calculates the target value from the next intersection state to the terminal state after the action is executed;

[0011] S4, calculating the estimated Q value corresponding to each action in the current intersection state according to the estimation network, and selecting an action using the ε-greedy strategy; after selecting the action, calculating the reward value through the fairness reward function, and storing the reward value in the experience replay set; calculating the target value from the next intersection state to the terminal state after executing the action according to the target network;

[0012] S5, using the loss function MSE to calculate the error between the target value and the estimated Q value, and updating the parameters of the estimation network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, thereby constructing a single intersection signal light phase control algorithm based on a fairness reward mechanism.

[0013] The present invention also provides a single intersection signal light phase control system based on FR-DDQN, comprising:

[0014] A simulation data acquisition unit, used to acquire historical traffic data of the intersection;

[0015] An agent construction unit is used to construct an agent, wherein the agent includes a state space, an action space and a fairness reward function; wherein the state space is the vehicle position distribution and speed on the entrance lane; the action space is all non-conflicting phases of the intersection; the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of the phase, the average travel time, the green light duration penalty item, the red light waiting time penalty item and the traffic condition reward and penalty items in each direction;

[0016] An algorithm network construction unit is used to construct two networks with the same structure and different parameters, namely an estimation network and a target network, based on the historical traffic data and the intelligent agent; the estimation network calculates the estimated Q value corresponding to each action in the current intersection state; and the target network calculates the target value from the next intersection state to the terminal state after the action is executed;

[0017] A network calculation unit is used to calculate the estimated Q value corresponding to each action in the current intersection state according to the estimation network, and select an action using the ε-greedy strategy; after selecting the action, calculate the reward value through the fairness reward function, and store the reward value in the experience playback set; calculate the target value from the next intersection state to the terminal state after executing the action according to the target network;

[0018] The network updating unit is used to calculate the error between the target value and the estimated Q value by using the loss function MSE, and to update the parameters of the estimated network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, so as to construct a single intersection signal light phase control algorithm based on a fairness reward mechanism.

[0019] The present invention also provides a single-intersection signal light phase control device based on FR-DDQN, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a single-intersection signal light phase control method based on FR-DDQN as described above.

[0020] The present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, a single intersection signal light phase control method based on FR-DDQN as described above is implemented.

[0021] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention provides a single intersection signal light phase control method based on FR-DDQN (Fai rness Reward mechanism with DDQN Algor ithm), which builds a signal light phase control model based on reinforcement learning and fairness reward mechanism, and provides a more reasonable reward function for the signal light phase control model from the perspectives of traffic efficiency and fairness. The traffic efficiency index can evaluate the short-term impact of the selected action on the environment, while the fairness reward controls the long-term impact of the action selection strategy on the environment, effectively improving the overall traffic efficiency of the intersection over a period of time, and providing a reference for the design of actual traffic light phase control.

[0023] The dual deep Q network constructed by the present invention includes an estimation network and a target network. The network architecture helps to reduce the over-estimation bias caused by a single network and improve the stability and performance of the strategy. In addition, in terms of fairness factors, in addition to considering the travel time, multiple specific fairness factors are also considered, including the average queue length at the intersection at the current moment, the average queue length at the intersection at the previous moment, and the green light time utilization rate of the current green light phase corresponding to the traffic direction. These factors are more comprehensive when dealing with complex traffic scenarios.

[0024] The present invention utilizes a CNN network to fit the Q-value function of a state-action pair and updates the network by gradient descent, thereby solving the problem that the Q function corresponding to each state cannot be listed due to the continuous and large-scale states generated during the operation of traffic flow.

[0025] Since the intersection states at consecutive moments are highly correlated, the present invention uses an experience replay set to break this limitation, storing the state, action, reward and next state in the experience replay set, and using a small batch sampling strategy to perform random uniform sampling from the experience replay set during training. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A schematic diagram of a single intersection signal light phase control method based on FR-DDQN provided in Example 1;

[0028] Figure 2 A schematic diagram of the intersection structure provided in Example 1;

[0029] Figure 3 A schematic diagram of the action space of a four-fork intersection provided in Example 1;

[0030] Figure 4 A schematic diagram of the CNN network structure provided in Example 1;

[0031] Figure 5 A schematic diagram of the loss function update training process provided in Example 1;

[0032] Figure 6 A training flow chart of a single intersection signal light phase control method provided in Example 1;

[0033] Figure 7 A comparison chart of average travel time during peak hours at Quanzhou Fengze Tianan intersection provided in Example 1;

[0034] Figure 8 A comparison chart of queue lengths in each direction at the Fengze Tianan intersection during peak hours provided in Example 1;

[0035] Fig. 9 A schematic diagram of a single intersection signal light phase control system based on FR-DDQN provided in Example 2.

[0036] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] Embodiment 1

[0039] Embodiment 1 of the present invention provides a single-intersection signal light phase control method based on FR-DDQN, which can be implemented by a single-intersection signal light phase control device based on FR-DDQN (hereinafter referred to as the phase control device), and in particular, executed by one or more processors in the phase control device.

[0040] In this embodiment, the phase control device may be an electronic device equipped with a processor, the processor having a computer program of the single intersection signal light phase control method based on FR-DDQN and the computer program can be executed, such as a computer, a smart phone, a smart tablet, a workstation, a server and other computing devices with data calculation and processing capabilities, which are not specifically limited in the present invention.

[0041] like Figure 1 As shown, a single intersection signal light phase control method based on FR-DDQN includes steps S1 to S5.

[0042] S1, obtain the historical traffic data of the intersection.

[0043] In this embodiment, the historical traffic data set of the intersection can be obtained from various traffic platforms. The traffic data of a certain intersection includes the number of entrance lanes n of the intersection 3 , the historical traffic flow q at each entrance of the intersection, the maximum speed v restricted by the road max wait.

[0044] S2, construct an intelligent agent, which includes a state space, an action space and a fairness reward function; wherein the state space is the vehicle position distribution and speed on the entrance lane; the action space is all non-conflicting phases of the intersection; the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of the phase, the average travel time, the green light duration penalty item, the red light waiting time penalty item and the traffic condition reward and penalty items in each direction.

[0045] In this embodiment, the basic structure of the intelligent agent is constructed, including the state space s, the action space a and the reward function r. Figure 2 As shown in the figure, the vehicle position distribution and speed on the entrance lane are selected to describe the traffic flow state space. The entrance lane L meters away from the intersection stop line is divided into cells of length c. If there is a vehicle in the cell, the position matrix The corresponding value is 1, otherwise the value is 0; when there is a vehicle in the cell, the speed of the vehicle is compared with the maximum speed v of the road limit max Normalized, recorded as velocity matrix The value of entrance lane j is expressed as The velocity matrix is ​​expressed as The position matrix of the intersection is expressed as The velocity matrix is ​​expressed as The state of the intersection at time t is denoted as s t =(pos t ,v t );

[0046] Among them, n3 Indicates the number of entrance lanes at the intersection, j indicates the entrance lane number; n 6 represents the number of lane division cells, and m represents the cell number.

[0047] In this embodiment, all non-conflicting phases of the intersection are selected as the action space, and the right turn direction is defined as not controlled by the traffic light, such as Figure 3 As shown, a total of 12 phase action spaces are finally defined: "straight from north to south + turn left", "straight from east to west + turn left", "straight from south to north + turn left", "straight from west to east + turn left", "straight from north to south + turn left from east to west", "straight from east to west + turn left from south to north", "straight from south to north + turn left from west to east", "straight from west to east + turn left from north to south", "straight from south to north + straight from north to south", "straight from east to west + straight from west to east", "turn left from south to north + turn left from north to south", "turn left from east to west + turn left from west to east";

[0048] The number of types of phase action spaces may also be determined according to actual conditions, such as 8 types, 4 types, etc., which is not limited here.

[0049] When constructing the fairness reward function, we consider the traffic efficiency and fairness separately; in terms of traffic efficiency, we consider the impact of each decision period and a traffic light cycle; in terms of decision period, we select the change of queue length and the utilization rate of green light time of each phase to reflect the short-term impact of the current decision on the traffic efficiency per unit time; in a traffic light cycle, we select the average travel time to evaluate the impact of the selected action set on the traffic efficiency within a cycle; in terms of fairness, we establish constraints and set penalty functions according to the green light duration and red light waiting time of each phase and the traffic conditions of each traffic direction within a traffic light cycle; among them,

[0050] The average queue length is obtained by calculating the difference between the average queue lengths of each approaching direction between two adjacent decision points; if the difference between the average queue lengths is a positive number, it means that the action reduces the average queue length of the intersection; if it is a negative number, it means that the action increases the average queue length of the intersection;

[0051] The green light time utilization rate is obtained by calculating the green light time utilization rates of the two traffic directions in the phase; if the green light time utilization rate is larger, it means that the number of vehicles passing through the traffic direction corresponding to the phase in a unit time period is larger, and the contribution to improving the traffic efficiency of the intersection is greater; otherwise, the contribution is smaller. The formula of the green light time utilization rate u is:

[0052]

[0053] Among them, u direpresents the green light time utilization rate of the intersection traffic direction di; di represents a certain traffic direction of the intersection, q' di represents the traffic flow through the intersection in the di direction, cp di Indicates the road capacity in direction di;

[0054] In this embodiment, di={go straight from north to south, turn left from north to south, go straight from east to west, turn left from east to west, go straight from south to north, turn left from south to north, go straight from west to east, turn left from west to east}.

[0055] The green light duration penalty term e 1 The minimum green light duration is g min and maximum green light duration g max Establish a penalty item. If the green light duration is within the green light range, then e 1 If it does not meet the requirements, the closer to the green light range, the 1 The smaller it is, the bigger it is;

[0056] The red light waiting time penalty term e 2 It is based on the maximum red light waiting time rw max Establish a penalty item. If the waiting time for a red light in a certain direction of traffic exceeds rw max , then the penalty term e 2 for If it does not exist, the penalty term e 2 is 0;

[0057] The traffic conditions in each direction are rewarded or penalized 3 By determining each maximum signal light cycle cy max The traffic conditions of all traffic directions within the period cy are used to establish reward and penalty items; if the period cy max If all traffic directions in the area are clear, then e 3 For Reward If you don't meet the requirements, you will be punished. Negative number

[0058] The penalty function is expressed as follows:

[0059]

[0060] in, represents the reward and penalty scale, g represents the duration of the green light in the current phase, rw represents the duration of the phase waiting for the red light, and g min is the minimum green light duration, g max is the maximum green light duration, cy max is the maximum semaphore cycle, rw max is the maximum waiting time for a red light;

[0061] The formula of the fairness reward function after weighting is:

[0062]

[0063] Among them, r t is the fairness reward function at the current time t; queue t is the average queue length at the intersection at the current time t, queue t-1 is the average queue length at the intersection at the previous moment; k 1 , k 2 , k 3 , k 4 , k 5 , k 6 is the weight coefficient; Indicates the traffic direction di corresponding to the current green light phase 1 、di 2 The green light time utilization rate; travel_time is the average travel time. If travel_time is not 0, the smaller the travel_time is, the faster the vehicles pass through the intersection and the better the traffic condition is. The larger the travel_time is, the worse the intersection condition is. If travel_time is 0, it is necessary to judge whether there are no vehicles entering the corresponding section or whether there is traffic congestion that prevents vehicles from passing through the corresponding section based on the queue length. If there are no vehicles in the queue, there are no vehicles entering the corresponding section. If there are vehicles in the queue, there is traffic congestion.

[0064] S3, based on the historical traffic data and the intelligent agent, construct two networks with the same structure but different parameters, namely the estimation network and the target network; the estimation network calculates the estimated Q value corresponding to each action in the current intersection state; the target network calculates the target value from the next intersection state to the terminal state after executing the action.

[0065] In this embodiment, two networks with the same structure but different parameters are constructed to update the Q value. The estimation network is responsible for calculating the current state s t The Q value corresponding to each action is Q t (s t ,a;θ); the target network is responsible for calculating the intersection state s after executing the action t+1 The cumulative value to the terminal state, that is, the target value y t (s t+1 ,a';θ'). Calculate the target value y t (s t+1 ,a';θ') in state s t+1 When the corresponding maximum estimated Q value is reached, the state s is first calculated by estimating the network t+1The action a' corresponding to the maximum estimated Q value is then used to calculate the state s using the target network t+1 The target Q value corresponding to action a' is calculated as follows:

[0066]

[0067] Among them, y t (s t+1 ,a';θ') is the target value at time t; s t 、s t+1 Respectively represent the execution of action a t Traffic flow status before and after; a t represents the action selected by the agent at time t; r t Indicates execution of action a t Reward after Q t (s t+1 ,a';θ) is the state s t+1 The estimated Q value corresponding to the next action a'; Indicates that in state s t+1 The action a' corresponding to the maximum Q value in the estimated network selected below, γ represents the discount coefficient, θ represents the parameters of the estimated network, and θ' represents the parameters of the target network.

[0068] S4, calculating the estimated Q value corresponding to each action in the current intersection state according to the estimation network, and selecting the action using the ε-greedy strategy; after selecting the action, calculating the reward value through the fairness reward function, and storing the reward value in the experience replay set; calculating the target value from the next intersection state to the terminal state after executing the action according to the target network.

[0069] In this embodiment, when selecting actions, the intelligent agent adopts the ε-greedy strategy to select actions, and realizes exploration and utilization in reinforcement learning by adjusting the exploration rate ε; wherein ε represents the probability of randomly selecting an action, 1-ε represents the probability of the corresponding action when the maximum estimated Q value is selected, and ε decreases continuously with the increase of the number of iterative training, realizing the process of transitioning from exploration to utilization.

[0070] In this embodiment, since the intersection states at consecutive moments have a strong correlation, the experience playback set is used to break this limitation and the current state s t , selected action a t , the state after executing the action s t+1 and the reward value r t The experience replay set is stored, and a small batch sampling strategy is used to randomly and uniformly sample from the experience replay set when training the algorithm network.

[0071] Then, according to the target network in the constructed network structure, the intersection state s after the action is executed is calculated t+1 To the target value y of the terminal state t (s t+1 ,a';θ').

[0072] S5, using the loss function MSE to calculate the error between the target value and the estimated Q value, and updating the parameters of the estimation network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, thereby constructing a single intersection signal light phase control algorithm based on a fairness reward mechanism.

[0073] In this embodiment, when updating the parameters of the algorithm network by the gradient descent method, the CNN network is used to fit the Q value function corresponding to the state action to solve the problem that when continuous large-scale states are generated during the operation of traffic flow, it is impossible to list the Q value function corresponding to each state. Figure 4 As shown in the figure, the CNN network is divided into 4 layers. The first layer consists of 32 convolution kernels with a step size of 2; the second layer consists of 64 convolution kernels with a step size of 1; the third layer is a fully connected layer consisting of 128 neurons; the fourth layer is a fully connected layer consisting of 64 neurons, and the fully connected layers all use the Relu activation function.

[0074] In this embodiment, if Figure 5 As shown, the loss function MSE is used to calculate the error between the target value and the estimated Q value. The formula of the loss function MSE is:

[0075]

[0076] Where m is the number of cells into which the entrance lane is divided, Q t (s t ,a t ; θ) is the current state s t For each action a t The corresponding estimated Q value.

[0077] The update formula of the estimation network is:

[0078] Q t+1 (s t ,a t ; θ) = Q t (s t ,a t ;θ)+α[y t (s t+1 ,a';θ')-Q t (s t ,a t ;θ)];

[0079] Among them, α represents the learning rate.

[0080] like Figure 6 The algorithm training flow chart shown in the figure shows that after the network model is built, the maximum number of simulations ST is set. max Each simulation process of ST is set with a simulation time T, and the calculation is iterated within T. If the current time t exceeds T, the number of simulations ST increases. If the number of simulations ST exceeds ST max Finally, the signal light phase control strategy of the intersection is output.

[0081] The details are as follows:

[0082] A1, initialize the network structure and parameters θ, θ', and the number of simulations ST = 0;

[0083] A2, reset the simulation environment, set t = 0;

[0084] A3, after selecting an action, randomly selects the probability of ε, and the probability of 1-ε is to select the optimal action;

[0085] A4, the traffic direction di corresponding to the selected action a turns on the green light for Δt seconds, and the new state s of the traffic flow is obtained after the execution is completed. t+1 ;

[0086] A5, calculate the reward value e t ;

[0087] A6, change the current state s t , selected action a t , the state after executing the action s t+1 and the reward value r t Save experience replay collection;

[0088] A7, by frequency f 1 、f 2 Update the estimated network and the target network respectively;

[0089] A8, let t = t + Δt;

[0090] A9, determines whether t is greater than or equal to T. If not, returns to A2; if so, proceeds to the next step;

[0091] A10, the simulation times ST increases, that is, ST = ST + 1;

[0092] A11, determine whether ε is greater than a threshold value of 0.1, if so, set ε = ε - 0.01; otherwise, proceed to the next step;

[0093] A12, determine whether ST exceeds the maximum number of simulation times ST max If not, continue iterating and return to A2; if so, end the iteration and obtain the signal light phase control solution.

[0094] In order to verify the effectiveness of the model and model solution proposed by the present invention, the present invention provides another preferred embodiment to illustrate the application of the present invention.

[0095] The Fengze Tian'an intersection in Quanzhou and the Ningxi Xingye intersection in Zhuhai were selected as experimental objects. The data set was selected from the road structure data provided by the Quanzhou Traffic Police Detachment, the traffic flow data during the morning peak period (7:30-8:45) and the off-peak period (10:00-12:00) on August 1, 2019, and the signal timing scheme for the corresponding period, and the intersection geometry data provided by the open data set of the intersection of Ningxi Road and Xingye Road in Zhuhai, the traffic flow data during the morning peak period (7:00-9:00) and the off-peak period (9:30-10:30) on September 15, 2015, and the signal timing scheme for the corresponding period. Then, simulation modeling was carried out based on the road structure data of the intersection.

[0096] In terms of evaluation methods and indicators, this embodiment uses traffic data during peak hours for training, and then applies the trained signal light phase control model to peak and off-peak hours for verification to evaluate the control effect of the signal light phase control model.

[0097] The average queue length queue and the average travel time travel_time of each entry direction and the traffic volume tr passing through the intersection are selected as evaluation indicators to compare the control effects of various methods. If the average queue length is large, the average travel time is long, or the traffic volume that can pass through the intersection in a fixed time period is small, it indicates that the control effect of the signal light phase control strategy is poor; if the average queue length is Smaller, average travel time The more traffic that can pass through the intersection in a shorter time or within a fixed time period, the better the control effect of the signal light phase control strategy. The specific calculation method is as follows:

[0098]

[0099] Among them, n 6 represents the number of exit lanes, q' o represents the outgoing traffic flow of lane o.

[0100] Select the length of the road section from the stop line L = 80 meters, the cell size c = 5 meters, and the maximum speed limit of the road is v max = 60km / h. The unit green light time of each phase of the traffic light is Δt = 5 seconds, and the minimum green light duration is limited to g min = 20 seconds, the maximum green light duration is limited to g max = 40 seconds, the maximum red light waiting time is rw max = 80 seconds, the maximum signal light cycle is cy max= 180 seconds. Due to the queue length difference Δqueue, phase green light time utilization u, green light duration penalty e 1 and the red light waiting time penalty term e 2 The calculation interval is the unit green light time Δt, while the average travel time travel_time and the traffic penalty term e 3 The calculation interval is the maximum signal light cycle cy max , and in the fairness reward and penalty item, press e 2 >e 1 Therefore, this embodiment sets the weight coefficient of each indicator according to the value, calculation interval and importance of each indicator. The parameter settings are shown in Table 1.

[0101] Table 1. Experimental parameter settings

[0102]

[0103] This embodiment adopts the original signal timing scheme (FTC), takes the change of average queue length as the reward mechanism (QD-DDQN), takes the change of vehicle delay, waiting time and average queue length as the reward mechanism (DWQ-DDQN), and takes the change of average queue length, average travel time and green light time utilization as the reward mechanism (QTU-DDQN) to compare the method of this chapter (FR-DDQN) with the method.

[0104] This example uses the traffic flow data during peak hours to train the single intersection signal light phase control model, and verifies it 10 times with different random seed numbers during peak hours and off-peak hours. Figure 7 As shown in Figure 2, the comparison of queue lengths in each direction at Fengze Tianan intersection during peak hours is as follows: Figure 8 Table 2 shows the comparison of simulation control results at Quanzhou Fengze Tianan intersection, taking the average value of 10 simulations with different random seed numbers, where the traffic volume is in units of hours, indicating the traffic volume passing through the intersection within one hour. Table 3 shows the comparison of simulation control results at Zhuhai Ningxi Xingye intersection during peak and off-peak hours, taking the average value of 10 simulation experiments with different random seeds.

[0105] Table 2. Simulation control results of Quanzhou Fengze Tianan intersection

[0106]

[0107] Table 3. Simulation control results of the Ningxi Xingye intersection in Zhuhai

[0108]

[0109]

[0110] According to the simulation control results in Table 2 and Table 3, the control effect of the method of the present invention (FR-DDQN) is better than that of the fixed timing control (FTC) and the method that only considers the traffic benefit reward (QD-DDQN, DWQ-DDQN, QTU-DDQN). The control effect of the signal light phase control model trained in the peak period is also good in the off-peak period, which shows that the method of the present invention can be applied to the traffic flow conditions in different periods and has universal applicability.

[0111] Peak hours: In the Quanzhou data set, the average travel time was reduced by 19.6769 seconds, the average queue length was reduced by 11.8641 meters, and the traffic volume increased by 1,289 vehicles per hour compared with the fixed timing control; in the Zhuhai data set, the average travel time was reduced by 42.3652 seconds, the average queue length was reduced by 45.9167 meters, and the traffic volume increased by 863 vehicles per hour. The average travel time was shortened by 52.31%, the average queue length was reduced by 61.72%, and the traffic volume passing through the intersection during the fixed time period increased by 48.40%.

[0112] Off-peak hours: In the Quanzhou data set, the average travel time was reduced by 17.8894 seconds compared with the fixed timing control, the average queue length was reduced by 7.662 meters, and the traffic volume increased by 1,521 vehicles per hour; in the Zhuhai data set, the average travel time was reduced by 40.2988 seconds compared with the fixed timing control, the average queue length was reduced by 40.3949 meters, and the traffic volume increased by 843 vehicles per hour. The average travel time was shortened by 56.37%, the average queue length was reduced by 85.24%, and the traffic volume passing through the intersection during the fixed time period increased by 59.13%.

[0113] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0114] The present invention provides a single intersection signal light phase control method based on FR-DDQN, which builds a signal light phase control model based on reinforcement learning and fairness reward mechanism, and provides a more reasonable reward function for the signal light phase control model from the perspectives of traffic efficiency and fairness. The traffic efficiency index can evaluate the short-term impact of the selected action on the environment, while the fairness reward controls the long-term impact of the action selection strategy on the environment, effectively improving the overall traffic efficiency of the intersection over a period of time, and providing a reference for the design of actual traffic light phase control.

[0115] Embodiment 2

[0116] like Fig. 9 As shown, the second embodiment of the present invention further provides a single intersection signal light phase control device based on FR-DDQN, comprising:

[0117] A simulation data acquisition unit, used to acquire historical traffic data of the intersection;

[0118] An agent construction unit is used to construct an agent, wherein the agent includes a state space, an action space and a fairness reward function; wherein the state space is the vehicle position distribution and speed on the entrance lane; the action space is all non-conflicting phases of the intersection; the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of the phase, the average travel time, the green light duration penalty item, the red light waiting time penalty item and the traffic condition reward and penalty items in each direction;

[0119] An algorithm network construction unit is used to construct two networks with the same structure and different parameters, namely an estimation network and a target network, based on the historical traffic data and the intelligent agent; the estimation network calculates the estimated Q value corresponding to each action in the current intersection state; and the target network calculates the target value from the next intersection state to the terminal state after the action is executed;

[0120] A network calculation unit is used to calculate the estimated Q value corresponding to each action in the current intersection state according to the estimation network, and select an action using the ε-greedy strategy; after selecting the action, calculate the reward value through the fairness reward function, and store the reward value in the experience playback set; calculate the target value from the next intersection state to the terminal state after executing the action according to the target network;

[0121] The network updating unit is used to calculate the error between the target value and the estimated Q value by using the loss function MSE, and to update the parameters of the estimated network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, so as to construct a single intersection signal light phase control algorithm based on a fairness reward mechanism.

[0122] Embodiment 3

[0123] The third embodiment of the present invention also provides a single-intersection signal light phase control device based on FR-DDQN, which includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the single-intersection signal light phase control method based on FR-DDQN as described above.

[0124] Embodiment 4

[0125] The fourth embodiment of the present invention further provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, the single-intersection signal light phase control method based on FR-DDQN as described above is implemented.

[0126] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0128] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or partly contributed to the prior art or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. It should be noted that, in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0129] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0130] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0131] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0132] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A single intersection signal light phase control method based on FR-DDQN, characterized in that: include: S1, obtain historical traffic data of the intersection; S2, constructing an intelligent agent, the intelligent agent includes a state space, an action space and a fairness reward function; wherein the state space is the vehicle position distribution and speed on the entrance lane; the action space is all non-conflicting phases of the intersection; the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of the phase, the average travel time, the green light duration penalty item, the red light waiting time penalty item and the traffic condition reward and penalty items of each direction; S3, constructing two networks with the same structure but different parameters, namely an estimation network and a target network, based on the historical traffic data and the intelligent agent; the estimation network calculates the estimated Q value corresponding to each action in the current intersection state; the target network calculates the target value from the next intersection state to the terminal state after the action is executed; S4, calculating the estimated Q value corresponding to each action in the current intersection state according to the estimation network, and selecting an action using the ε-greedy strategy; after selecting the action, calculating the reward value through the fairness reward function, and storing the reward value in the experience replay set; calculating the target value from the next intersection state to the terminal state after executing the action according to the target network; S5, using the loss function MSE to calculate the error between the target value and the estimated Q value, and updating the parameters of the estimation network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, thereby constructing a single intersection signal light phase control algorithm based on a fairness reward mechanism.

2. A single intersection signal light phase control method based on FR-DDQN according to claim 1, characterized in that ,The historical traffic data include: the number of entrance lanes of the intersection, the ,historical traffic volume of each entrance of the intersection and the ,maximum speed restricted by the road.

3. The single intersection signal light phase control method based on FR-DDQN according to claim 2, characterized in that: When constructing the fairness reward function, we consider traffic efficiency and fairness separately. In terms of traffic efficiency, we consider the impact of each decision period and a traffic light cycle. In the decision-making period, the change of queue length and the green light time utilization rate of each phase are selected to reflect the short-term impact of the current decision on the traffic efficiency within a unit time; in a traffic light cycle, the average travel time is selected to evaluate the impact of the selected action set on the traffic efficiency within a cycle; in terms of fairness, constraints are established and penalty functions are set according to the green light duration and red light waiting time of each phase and the traffic conditions of each traffic direction within a traffic light cycle; among them, The average queue length is obtained by calculating the difference between the average queue lengths of each approaching direction between two adjacent decision points; if the difference between the average queue lengths is a positive number, it means that the action reduces the average queue length of the intersection; if it is a negative number, it means that the action increases the average queue length of the intersection; The green light time utilization rate is obtained by calculating the green light time utilization rates of the two traffic directions in the phase; if the green light time utilization rate is larger, it means that the number of vehicles passing through the traffic direction corresponding to the phase in a unit time period is larger, and the contribution to improving the traffic efficiency of the intersection is greater; otherwise, the contribution is smaller. The formula of the green light time utilization rate u is: Among them, di represents a certain traffic direction at the intersection, q' di represents the traffic flow through the intersection in the di direction, cp di Indicates the road capacity in direction di; The green light duration penalty term e1 is the minimum green light duration g min and maximum green light duration g max A penalty term is established. If the duration of the green light is within the green light range, e1 is 0. If it is not, the closer to the green light range, the smaller e1 is, and vice versa. The red light waiting time penalty term e2 is based on the maximum red light waiting time rw max Establish a penalty item. If the waiting time for a red light in a certain direction of traffic exceeds rw max , then the penalty term e2 is If it does not exist, the penalty term e2 is 0; The traffic condition reward and penalty items e3 in each direction are determined by judging each maximum signal light cycle cy max The traffic conditions of all traffic directions within the period cy are used to establish reward and penalty items; if the period cy max If all traffic directions in the area are clear, then e3 is a bonus. If you don't meet the requirements, you will be punished. Negative number The penalty function is expressed as follows: in, represents the reward and punishment scale, g represents the duration of the green light in the current phase, and rw represents the duration of the phase waiting for the red light; Thus, the formula of the fairness reward function is obtained as follows: Among them, r t is the fairness reward function at the current time t; queue t is the average queue length at the intersection at the current time t, queue t-1 is the average queue length at the intersection at the previous moment; k1, k2, k3, k4, k5, k 65 is the weight coefficient; They respectively represent the green light time utilization rates of the traffic directions di1 and di2 corresponding to the current green light phase; travel_time is the average travel time. If travel_time is not 0, the smaller the travel_time is, the faster the vehicles pass through the intersection and the better the traffic condition is; the larger the travel_time is, the worse the intersection condition is; if travel_time is 0, it is necessary to judge whether there are no vehicles entering the corresponding section or whether there is traffic congestion that prevents vehicles from passing through the corresponding section based on the queue length. If there are no vehicles in the queue, then there are no vehicles entering the corresponding section; if there are vehicles in the queue, then there is traffic congestion.

4. A single intersection signal light phase control method based on FR-DDQN according to claim 3, characterized in that , the calculation formula of the target value is: Among them, y t (s t+1 ,a';θ') is the target value at time t; s t 、s t+1 Respectively represent the execution of action a t Traffic flow status before and after; a t represents the action selected by the agent at time t; r t Indicates execution of action a t Reward after Q t (s t+1 ,a';θ) is the state s t+1 The estimated Q value corresponding to the next action a'; Indicates that in state s t+1 The action a' corresponding to the maximum Q value in the estimated network selected below, γ represents the discount coefficient, θ represents the parameters of the estimated network, and θ' represents the parameters of the target network.

5. The single intersection signal light phase control method based on FR-DDQN according to claim 1 is characterized in that When the ε-greedy strategy is used to select actions, the exploration and utilization in reinforcement learning are realized by adjusting the exploration rate ε; where ε represents the probability of randomly selecting an action, 1-ε represents the probability of the corresponding action when the maximum estimated Q value is selected, and ε decreases as the number of iterative training increases.

6. The single intersection signal light phase control method based on FR-DDQN according to claim 1, characterized in that When updating the parameters of the algorithm network by the gradient descent method, the CNN network is used to fit the Q value function corresponding to the state action to solve the problem that the Q value function corresponding to each state cannot be listed when continuous large-scale states are generated during the operation of traffic flow; wherein, the CNN network is divided into 4 layers, the first layer consists of 32 convolution kernels with a step size of 2; the second layer consists of 64 convolution kernels with a step size of 1; the third layer is a fully connected layer composed of 128 neurons; the fourth layer is a fully connected layer composed of 64 neurons, and the fully connected layers all use the Relu activation function.

7. The single intersection signal light phase control method based on FR-DDQN according to claim 4 is characterized in that , the formula of the loss function MSE is: Where m is the number of cells into which the entrance lane is divided, Q t (s t ,a t ; θ) is the current state s t For each action a t The corresponding estimated Q value.

8. The single intersection signal light phase control method based on FR-DDQN according to claim 7 is characterized in that ,The update formula of the estimation network is: Q t+1 (s t ,a t ;θ)=Q t (s t ,a t ;θ)+α[y t (s t+1 ,a';θ')-Q t (s t ,a t ;i)]; Among them, α represents the learning rate.

9. A single intersection signal light phase control method based on FR-DDQN according to claim 8, characterized in that , the experience replay set includes state s t 、Action a t , Reward t and status t+1 , when training the algorithm network, a small batch sampling strategy is adopted to randomly and uniformly sample from the experience replay set.

10. A single intersection signal light phase control system based on FR-DDQN, characterized in that: include: A simulation data acquisition unit, used to acquire historical traffic data of the intersection; An agent construction unit is used to construct an agent, wherein the agent includes a state space, an action space and a fairness reward function; wherein the state space is the vehicle position distribution and speed on the entrance lane; the action space is all non-conflicting phases of the intersection; the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of the phase, the average travel time, the green light duration penalty item, the red light waiting time penalty item and the traffic condition reward and penalty items in each direction; An algorithm network construction unit is used to construct two networks with the same structure and different parameters, namely an estimation network and a target network, based on the historical traffic data and the intelligent agent; the estimation network calculates the estimated Q value corresponding to each action in the current intersection state; and the target network calculates the target value from the next intersection state to the terminal state after the action is executed; A network calculation unit is used to calculate the estimated Q value corresponding to each action in the current intersection state according to the estimation network, and select an action using the ε-greedy strategy; after selecting the action, calculate the reward value through the fairness reward function, and store the reward value in the experience playback set; calculate the target value from the next intersection state to the terminal state after executing the action according to the target network; The network updating unit is used to calculate the error between the target value and the estimated Q value by using the loss function MSE, and to update the parameters of the estimated network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, so as to construct a single intersection signal light phase control algorithm based on a fairness reward mechanism.

Citation Information

Patent Citations

  • Idle traffic light intelligent control method based on reinforcement learning

    CN110930734A

  • Signal intersection adaptive control method based on improved deep Q network

    CN117315960A

  • Bus priority traffic signal cooperative control method based on multi-agent deep reinforcement learning

    CN117746654A

  • Decision planning method for automatic driving vehicle in urban traffic scene based on reinforcement learning

    CN118396034A

Cited By

  • Safety management system and method for intelligent money box

    CN119940939A

  • Intersection traffic signal control method in intelligent network connection vehicle passing environment

    CN120319044A