A single intersection signal light phase control method and system based on FR-DDQN

Abstract

The present invention provides a single-intersection signal light phase control method and system based on FR-DDQN, which relates to the technical field of signal light phase control. The present invention obtains historical traffic data from the intersection to construct a simulation environment and intelligent agent; then constructs two networks with identical structures but different parameters; calculates the estimated Q value corresponding to each action in the current intersection state based on the estimation network, and selects the action using the ε-greedy strategy; after selecting the action, calculates the reward value of the reward function and stores it in the experience replay set; calculates the target value from the next intersection state to the terminal state based on the target network; and uses the loss function and gradient descent method to update the network parameters until the maximum number of simulations is reached, thereby obtaining the signal light phase control strategy for the intersection. The present invention can not only evaluate the short-term impact of the selected action on the environment, but also control the long-term impact of the action selection strategy on the environment through a fairness reward mechanism, effectively improving the overall traffic efficiency of the intersection over a period of time.

Description

Technical Field

[0001] The present invention relates to the technical field of phase control of intersection signal lights, and in particular to a single intersection signal light phase control method and system based on FR-DDQN. Background Art

[0002] Transportation systems are one of the most crucial infrastructures in modern cities, supporting the daily commuting and travel of millions of people. The normal functioning of modern society relies heavily on an efficient transportation system. In modern urban traffic structures, intersections connect and regulate traffic flows between adjacent roads. Therefore, strengthening intersection management and control can effectively alleviate congestion. Intelligent traffic light control can adapt to varying traffic conditions and improve intersection efficiency.

[0003] Traffic light control is a Markov decision problem, requiring the decision of which phase of a green light to open and how long to open it, based on varying traffic conditions, ultimately improving traffic efficiency at the intersection. Because reinforcement learning methods do not rely on prior knowledge and can learn action strategies through continuous interaction with the environment, the agent's learning goals directly influence its action decisions. Existing reinforcement learning-based traffic light control methods use reward mechanisms that only evaluate the short-term impact of the selected strategy, ignoring the long-term impact of the action strategy on future moments. For example, existing patents CN113380054A, "A Traffic Light Control Method Based on Reinforcement Learning," and CN111243299B, "A Single Intersection Signal Control Method Based on a 3DQN_PSER Algorithm," are both reinforcement learning-based traffic light control methods. Their reward mechanisms only evaluate the short-term impact of the selected strategy, ignoring the long-term impact of the action strategy on future moments, and fail to consider the impact of fairness factors in their reward functions. Existing patent document CN114299732B, "A traffic light control method that simultaneously considers travel time and fairness," constructs a single travel time strategy Q network and a fairness strategy Q network, which only considers travel time and fairness, and does not consider other fairness factors, such as the average queue length at the intersection at the current moment, the average queue length at the intersection at the previous moment, and the green light time utilization rate corresponding to the traffic direction of the current green light phase.

[0004] Therefore, in order to solve the short-sightedness problem in the reward mechanism, it is of practical significance to establish a reward mechanism that can effectively control traffic lights. In view of this, the applicant has filed this application after studying the existing technology. Summary of the Invention

[0005] The present invention aims to provide a single-intersection signal light phase control method and system based on FR-DDQN to solve the problem that the existing methods do not consider the impact of the current decision on subsequent decisions and the overall signal light control effect and are short-sighted.

[0006] In order to solve the above technical problems, the present invention is implemented through the following technical solutions:

[0007] A single intersection signal light phase control method based on FR-DDQN, comprising:

[0008] S1, obtain historical traffic data of the intersection;

[0009] S2. Construct an intelligent agent, which includes a state space, an action space, and a fairness reward function. The state space is the position distribution and speed of vehicles in the entrance lane; the action space is all non-conflicting phases of the intersection; and the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of each phase, the average travel time, the green light duration penalty, the red light waiting time penalty, and the traffic condition reward and penalty for each direction.

[0010] S3, constructing two networks with the same structure but different parameters, namely an estimation network and a target network, based on the historical traffic data and the intelligent agent; the estimation network calculates the estimated Q value corresponding to each action in the current intersection state; and the target network calculates the target value from the next intersection state to the terminal state after the action is executed;

[0011] S4, calculating the estimated Q value corresponding to each action in the current intersection state according to the estimation network, and selecting an action using the ε-greedy strategy; after selecting the action, calculating the reward value using the fairness reward function, and storing the reward value in the experience replay set; calculating the target value from the next intersection state to the terminal state after executing the action according to the target network;

[0012] S5, using the loss function MSE to calculate the error between the target value and the estimated Q value, and updating the parameters of the estimation network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, thereby constructing a single intersection signal light phase control algorithm based on a fairness reward mechanism.

[0013] The present invention also provides a single intersection signal light phase control system based on FR-DDQN, comprising:

[0014] A simulation data acquisition unit, used to acquire historical traffic data of the intersection;

[0015] An agent construction unit is configured to construct an agent comprising a state space, an action space, and a fairness reward function; wherein the state space is the position distribution and speed of vehicles in the entrance lane; the action space is all non-conflicting phases of the intersection; and the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of each phase, the average travel time, the green light duration penalty, the red light waiting time penalty, and the traffic condition reward and penalty for each direction;

[0016] An algorithm network construction unit is configured to construct two networks with identical structures but different parameters, namely an estimation network and a target network, based on the historical traffic data and the intelligent agent; the estimation network calculates an estimated Q value corresponding to each action at the current intersection state; and the target network calculates a target value from the next intersection state to the terminal state after the action is executed;

[0017] a network calculation unit, configured to calculate an estimated Q value corresponding to each action in the current intersection state based on the estimation network, and select an action using an ε-greedy strategy; after selecting an action, calculate a reward value using the fairness reward function, and store the reward value in an experience replay set; and calculate a target value from the next intersection state to the terminal state after executing the action based on the target network;

[0018] The network update unit is used to calculate the error between the target value and the estimated Q value using the loss function MSE, and update the parameters of the estimation network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, thereby constructing a single intersection signal light phase control algorithm based on the fairness reward mechanism.

[0019] The present invention also provides a single-intersection signal light phase control device based on FR-DDQN, including a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the single-intersection signal light phase control method based on FR-DDQN as described above.

[0020] The present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, the single-intersection signal light phase control method based on FR-DDQN is implemented as described above.

[0021] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0022] This paper proposes a single-intersection traffic light phase control method based on the FR-DDQN (Fairness Reward Mechanism with DDQN Algorithm). This method builds a traffic light phase control model based on reinforcement learning and a fairness reward mechanism. This model considers both traffic efficiency and fairness, providing a more rational reward function for the traffic light phase control model. The traffic efficiency metric assesses the short-term impact of the selected action on the environment, while the fairness reward controls the long-term impact of the action selection strategy on the environment. This method effectively improves the overall traffic efficiency of the intersection over a period of time, providing a reference for the design of practical traffic light phase control.

[0023] The dual deep Q network constructed in the present invention includes an estimation network and a target network. This network architecture helps reduce the overestimation bias caused by a single network, improving the stability and performance of the strategy. In addition to considering travel time, the fairness factor also considers multiple specific fairness factors, including the average queue length at the intersection at the current moment, the average queue length at the intersection at the previous moment, and the green light time utilization rate corresponding to the traffic direction of the current green light phase. These factors are more comprehensive when dealing with complex traffic scenarios.

[0024] The present invention uses a CNN network to fit the Q-value function of state-action pairs and updates the network by gradient descent, which solves the problem that it is impossible to list the Q function corresponding to each state due to the continuous large-scale states generated during the operation of traffic flow.

[0025] Since the intersection states at consecutive moments are highly correlated, the present invention uses an experience replay set to break this limitation. The state, action, reward, and next state are stored in the experience replay set, and a small batch sampling strategy is used during training to perform random and uniform sampling from the experience replay set. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 A schematic diagram of a single-intersection signal light phase control method based on FR-DDQN provided in Example 1;

[0028] Figure 2 A schematic diagram of the intersection structure provided in Example 1;

[0029] Figure 3 A schematic diagram of the action space of a four-fork intersection provided in Example 1;

[0030] Figure 4 A schematic diagram of the CNN network structure provided in Example 1;

[0031] Figure 5 A schematic diagram of the loss function update training process provided in Example 1;

[0032] Figure 6 A training flow chart of a single intersection signal light phase control method provided in Example 1;

[0033] Figure 7 A comparison chart of average travel time during peak hours at the Quanzhou Fengze Tianan intersection provided in Example 1;

[0034] Figure 8 A comparison chart of queue lengths in each direction at the Fengze-Tian'an intersection during peak hours provided in Example 1;

[0035] Figure 9 This is a schematic diagram of a single-intersection signal light phase control system based on FR-DDQN provided in Example 2.

[0036] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] Example 1

[0039] Embodiment 1 of the present invention provides a single-intersection signal light phase control method based on FR-DDQN, which can be implemented by a single-intersection signal light phase control device based on FR-DDQN (hereinafter referred to as the phase control device), and in particular, executed by one or more processors in the phase control device.

[0040] In this embodiment, the phase control device may be an electronic device equipped with a processor, which carries a computer program of the FR-DDQN-based single intersection signal light phase control method and can be executed, such as a computer, a smart phone, a smart tablet, a workstation, a server, or other computing device with data calculation and processing capabilities, which is not specifically limited in the present invention.

[0041] like Figure 1 As shown, a single intersection signal light phase control method based on FR-DDQN includes steps S1 to S5.

[0042] S1, obtain historical traffic data of the intersection.

[0043] In this embodiment, the historical traffic data set of the intersection can be obtained from various traffic platforms. The traffic data of a certain intersection includes the number of entrance lanes n3 of the intersection, the historical traffic flow q of each entrance of the intersection, the maximum speed limit v max wait.

[0044] S2. Construct an intelligent agent, which includes a state space, an action space, and a fairness reward function. The state space is the position distribution and speed of vehicles in the entrance lane; the action space is all non-conflicting phases of the intersection; and the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of each phase, the average travel time, the green light duration penalty, the red light waiting time penalty, and the traffic condition reward and penalty items in each direction.

[0045] In this embodiment, the basic structure of the intelligent agent is constructed, including the state space s, the action space a and the reward function r. Figure 2 As shown, the vehicle position distribution and speed on the entrance lane are selected to describe the traffic flow state space. The entrance lane L meters away from the intersection stop line is divided into cells of length c. If there is a vehicle in the cell, the position matrix The corresponding value is 1, otherwise the value is 0; when there is a vehicle in the cell, the vehicle's speed is compared with the maximum speed v of the road limit. max Normalized, recorded as velocity matrix The value of; then the position matrix of entrance lane j is expressed as The velocity matrix is ​​expressed as The position matrix of the intersection is expressed as The velocity matrix is ​​expressed as The state of the intersection at time t is recorded as s t =(pos t ,v t );

[0046] Among them, n3 represents the number of entrance lanes of the intersection, j represents the entrance lane number; n6 represents the number of cells divided by the lane, and m represents the number of cells.

[0047] In this embodiment, all non-conflicting phases of the intersection are selected to define the action space, and the right turn direction is defined as not controlled by the traffic light, such as Figure 3 As shown, a total of 12 phase action spaces are finally defined: "straight from north to south + turn left", "straight from east to west + turn left", "straight from south to north + turn left", "straight from west to east + turn left", "straight from north to south + turn left from east to west", "straight from east to west + turn left from south to north", "straight from south to north + turn left from west to east", "straight from west to east + turn left from north to south", "straight from south to north + straight from north to south", "straight from east to west + straight from west to east", "turn left from south to north + turn left from north to south", "turn left from east to west + turn left from west to east".

[0048] The number of types of phase action spaces may also be determined according to actual conditions, such as 8 types, 4 types, etc., which is not limited here.

[0049] When constructing the fairness reward function, traffic efficiency and fairness are considered separately. In terms of traffic efficiency, the impact of each decision period and a traffic light cycle is considered. In the decision period, the change of queue length and the green light time utilization rate of each phase are selected to reflect the short-term impact of the current decision on the traffic efficiency per unit time. In a traffic light cycle, the average travel time is selected to evaluate the impact of the selected action set on the traffic efficiency within a cycle. In terms of fairness, constraints are established and penalty functions are set based on the green light duration and red light waiting time of each phase and the traffic conditions of each traffic direction within a traffic light cycle.

[0050] The average queue length is obtained by calculating the difference between the average queue lengths of each approaching direction between two adjacent decision points. If the difference is a positive number, it means that the action has reduced the average queue length of the intersection; if it is a negative number, it means that the action has increased the average queue length of the intersection.

[0051] The green light time utilization rate is obtained by calculating the green light time utilization rate of the two traffic directions within the phase. The larger the green light time utilization rate, the more vehicles pass through the traffic direction corresponding to the phase within a unit time period, and the greater the contribution to improving the traffic efficiency of the intersection; otherwise, the contribution is smaller. The formula for the green light time utilization rate u is:

[0052]

[0053] Among them, u direpresents the green light time utilization rate of the intersection traffic direction di; di represents a certain traffic direction of the intersection, q' di represents the traffic flow through the intersection in the di direction, cp di Indicates the road capacity in the di direction;

[0054] In this embodiment, di={go straight from north to south, turn left from north to south, go straight from east to west, turn left from east to west, go straight from south to north, turn left from south to north, go straight from west to east, turn left from west to east}.

[0055] The green light duration penalty term e1 is the minimum green light duration g min and maximum green light duration g max Establish a penalty term. If the duration of the green light is within the green light range, e1 is 0. If it is not, the closer it is to the green light range, the smaller e1 is, and vice versa.

[0056] The red light waiting time penalty term e2 is based on the maximum red light waiting time rw max Establish a penalty item. If the waiting time for a red light in a certain direction of traffic exceeds rw max , then the penalty term e2 is If it does not exist, the penalty term e2 is 0;

[0057] The traffic condition reward and penalty items e3 in each direction are determined by judging each maximum signal light cycle cy max The traffic conditions of all traffic directions within the area are used to establish reward and penalty items; if the cycle cy max If all traffic directions are clear, e3 is a bonus. If you are not satisfied, you will be punished That is, negative numbers

[0058] The expression of the penalty function is as follows:

[0059]

[0060] in, Indicates the reward and punishment scale, g indicates the duration of the green light in the current phase, rw indicates the duration of the phase waiting for the red light, g min is the minimum green light duration, g max is the maximum green light duration, cy max is the maximum semaphore cycle, rw max The maximum waiting time for a red light;

[0061] The formula for the fairness reward function after weighting is:

[0062]

[0063] Among them, r t is the fairness reward function at the current time t; queue t is the average queue length at the intersection at the current time t, queue t-1 is the average queue length at the intersection at the previous moment; k1, k2, k3, k4, k5, k6 are weight coefficients; It indicates the green light time utilization rate of the traffic directions di1 and di2 corresponding to the current green light phase; travel_time is the average travel time. If travel_time is not 0, the smaller the travel_time is, the faster the vehicles pass through the intersection and the better the traffic conditions are; the larger the travel_time is, the worse the intersection conditions are; if travel_time is 0, it is necessary to judge whether there are no vehicles entering the corresponding section or whether there is traffic congestion that prevents vehicles from passing the corresponding section based on the queue length. If there are no vehicles in the queue, it means that no vehicles have entered the corresponding section, and if there are vehicles in the queue, it means that there is traffic congestion.

[0064] S3, based on the historical traffic data and the intelligent agent, construct two networks with the same structure but different parameters, namely the estimation network and the target network; the estimation network calculates the estimated Q value corresponding to each action in the current intersection state; the target network calculates the target value from the next intersection state to the terminal state after executing the action.

[0065] In this embodiment, the Q value is updated by constructing two networks with the same structure but different parameters. The estimation network is responsible for calculating the current state s t The Q value corresponding to each action is Q t (s t ,a;θ); the target network is responsible for calculating the intersection state s after executing the action t+1 The cumulative value to the terminal state, that is, the target value y t (s t+1 ,a';θ'). Calculate the target value y t (s t+1 ,a';θ') in state s t+1 When the corresponding maximum estimated Q value is reached, the state s is first calculated by estimating the network t+1 The action a' corresponding to the maximum estimated Q value is then used to calculate the state s using the target network. t+1 The target Q value corresponding to action a' is calculated as follows:

[0066]

[0067] Among them, y t (s t+1 ,a';θ') is the target value at time t; s t 、st+1 Respectively represent the execution of action a t Traffic flow status before and after; a t represents the action selected by the agent at time t; r t Indicates execution of action a t The reward after Q t (s t+1 ,a';θ) is the state s t+1 The estimated Q value corresponding to the next action a'; In state s t+1 The action a' corresponding to the maximum Q value in the estimated network is selected below, γ represents the discount coefficient, θ represents the parameters of the estimated network, and θ' represents the parameters of the target network.

[0068] S4, calculating the estimated Q value corresponding to each action in the current intersection state according to the estimation network, and selecting an action using the ε-greedy strategy; after selecting an action, calculating the reward value through the fairness reward function, and storing the reward value in the experience replay set; calculating the target value from the next intersection state to the terminal state after executing the action according to the target network.

[0069] In this embodiment, the intelligent agent adopts the ε-greedy strategy when selecting actions, and realizes exploration and utilization in reinforcement learning by adjusting the exploration rate ε; where ε represents the probability of randomly selecting an action, 1-ε represents the probability of selecting the action corresponding to the maximum estimated Q value, and ε decreases continuously with the increase of the number of iterative training, realizing the process of transitioning from exploration to utilization.

[0070] In this embodiment, since the intersection states at consecutive moments have a strong correlation, the experience replay set is used to break this limitation, and the current state s t , selected action a t , the state after executing the action s t+1 and reward value r t The data is stored in the experience replay set, and a small batch sampling strategy is used to randomly and uniformly sample from the experience replay set when training the algorithm network.

[0071] Then, according to the target network in the constructed network structure, the intersection state s after the action is executed is calculated t+1 To the target value y of the terminal state t (s t+1 ,a';θ').

[0072] S5, using the loss function MSE to calculate the error between the target value and the estimated Q value, and updating the parameters of the estimation network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, thereby constructing a single intersection signal light phase control algorithm based on a fairness reward mechanism.

[0073] In this embodiment, when updating the parameters of the algorithm network by the gradient descent method, a CNN network is used to fit the Q-value function corresponding to the state action to solve the problem that it is impossible to list the Q-value function corresponding to each state when continuous and large-scale states are generated during the operation of traffic flow. Figure 4 As shown in the figure, the CNN network is divided into 4 layers. The first layer consists of 32 convolution kernels with a step size of 2; the second layer consists of 64 convolution kernels with a step size of 1; the third layer consists of a fully connected layer consisting of 128 neurons; the fourth layer consists of a fully connected layer consisting of 64 neurons, and the fully connected layers all use the ReLU activation function.

[0074] In this embodiment, if Figure 5 As shown, the loss function MSE is used to calculate the error between the target value and the estimated Q value. The formula of the loss function MSE is:

[0075]

[0076] Where m is the number of cells into which the entrance lane is divided, Q t (s t ,a t ;θ) is the current state s t Each action a t The corresponding estimated Q value.

[0077] The update formula of the estimation network is:

[0078] Q t+1 (s t ,a t ;θ)=Q t (s t ,a t ;θ)+α[y t (s t+1 ,a';θ')-Q t (s t ,a t ;θ)];

[0079] Here, α represents the learning rate.

[0080] like Figure 6 The algorithm training flow chart shown in the figure shows that after the network model is built, the maximum number of simulations ST is set. maxEach simulation ST process has a simulation time T, and the calculation is iterated within T. If the current time t exceeds T, the number of simulations ST increases. If the number of simulations ST exceeds ST max Finally, the signal light phase control strategy of the intersection is output.

[0081] The details are as follows:

[0082] A1, initialize the network structure and parameters θ, θ', and the number of simulations ST = 0;

[0083] A2, reset the simulation environment and set t = 0;

[0084] A3, after selecting an action, randomly selects the probability of ε, and the probability of 1-ε is to select the optimal action;

[0085] A4: The traffic flow direction di corresponding to the selected action a turns on the green light for Δt seconds. After the execution is completed, the new state s of the traffic flow is obtained. t+1 ;

[0086] A5, calculate the reward value e t ;

[0087] A6, change the current state s t , selected action a t , the state after executing the action s t+1 and reward value r t Save experience replay collection;

[0088] A7, updates the estimated network and target network according to frequencies f1 and f2 respectively;

[0089] A8, let t = t + Δt;

[0090] A9 checks whether t is greater than or equal to T. If not, it returns to A2; if so, it proceeds to the next step.

[0091] A10, the number of simulations ST increases, that is, ST = ST + 1;

[0092] A11, determine whether ε is greater than the threshold value 0.1. If so, set ε = ε - 0.01; otherwise, proceed to the next step;

[0093] A12, determine whether ST exceeds the maximum number of simulation times ST max If not, continue the iteration and return to A2; if so, end the iteration and obtain the signal light phase control solution.

[0094] In order to verify the effectiveness of the model and model solution proposed in the present invention, the present invention provides another preferred embodiment to illustrate the application of the present invention.

[0095] The Fengze Tian'an intersection in Quanzhou and the Ningxi Xingye intersection in Zhuhai were selected as experimental subjects. The datasets were selected from road structure data provided by the Quanzhou Traffic Police Detachment, traffic flow data from the morning rush hour (7:30-8:45) and off-peak hours (10:00-12:00), and the corresponding signal timing plans for the same periods, as well as intersection geometry data from the open dataset for the Ningxi Road and Xingye Road intersections in Zhuhai, traffic flow data from the morning rush hour (7:00-9:00) and off-peak hours (9:30-10:30), and the corresponding signal timing plans for the same periods, provided by the Zhuhai Ningxi Road and Xingye Road intersections. Simulation modeling was then performed based on the intersection road structure data.

[0096] In terms of evaluation methods and indicators, this embodiment uses traffic flow data during peak hours for training, and then applies the trained signal light phase control model to peak and off-peak hours for verification to evaluate the control effect of the signal light phase control model.

[0097] The average queue length queue and average travel time travel_time of each direction and the traffic flow tr through the intersection are selected as evaluation indicators to compare the control effects of various methods. If the average queue length is large, the average travel time is long, or the traffic flow that can pass through the intersection within a fixed time period is small, it indicates that the control effect of the signal light phase control strategy is poor; if ... Smaller, average travel time The shorter the time or the more traffic that can pass through the intersection within a fixed time period, the better the control effect of the signal light phase control strategy. The specific calculation method is as follows:

[0098]

[0099] Among them, n6 represents the number of exit lanes, q' o represents the outgoing traffic flow of lane o.

[0100] Select the road length L = 80 meters from the stop line, the cell size c = 5 meters, and the maximum speed limit of the road is v max =60km / h. The unit green light time of each phase of the traffic light is Δt = 5 seconds, and the minimum green light duration is limited to g min = 20 seconds, the maximum green light duration is limited to g max = 40 seconds, the maximum red light waiting time is rw max = 80 seconds, the maximum signal light cycle is cy max= 180 seconds. Since the calculation interval of queue length difference Δqueue, phase green light time utilization u, green light duration penalty e1 and red light waiting time penalty e2 is the unit green light time Δt, and the calculation interval of average travel time travel_time and traffic flow penalty e3 is the maximum signal light cycle cy max , and in the fairness reward and penalty items, the importance is ranked according to e2>e1. Therefore, this embodiment sets the weight coefficient of each indicator according to the value, calculation interval and importance of each indicator. The parameter settings are shown in Table 1.

[0101] Table 1. Experimental parameter settings

[0102]

[0103] This example adopts the original signal timing scheme (FTC), uses the change of average queue length as the reward mechanism (QD-DDQN), uses the change of vehicle delay, waiting time and average queue length as the reward mechanism (DWQ-DDQN), and uses the change of average queue length, average travel time and green light time utilization as the reward mechanism (QTU-DDQN) to compare with the method in this chapter (FR-DDQN).

[0104] This example uses traffic flow data during peak hours to train a single intersection signal light phase control model, and performs 10 verifications with different random seed numbers during peak hours and off-peak hours. Figure 7 As shown in the figure, the queue length comparison of each direction at Fengze Tianan intersection during peak hours is as follows: Figure 8 Table 2 compares the simulation control results for the Quanzhou Fengze Tian'an intersection, taking the average of 10 simulations with different random seed numbers. Traffic flow is measured in hours, representing the volume of traffic passing through the intersection within an hour. Table 3 compares the simulation control results for the Zhuhai Ningxi Xingye intersection during peak and off-peak hours, taking the average of 10 simulations with different random seed numbers.

[0105] Table 2. Simulation control results of Quanzhou Fengze Tianan intersection

[0106]

[0107] Table 3. Simulation control results of the Ningxi Xingye intersection in Zhuhai

[0108]

[0109]

[0110] The simulation results in Tables 2 and 3 show that the proposed method (FR-DDQN) outperforms fixed timing control (FTC) and methods that only consider traffic efficiency incentives (QD-DDQN, DWQ-DDQN, and QTU-DDQN). The traffic light phase control model trained during peak hours also performs well during off-peak hours, demonstrating its universal applicability to traffic conditions at different times.

[0111] During peak hours, in the Quanzhou data set, average travel time was reduced by 19.6769 seconds, average queue length was reduced by 11.8641 meters, and traffic volume increased by 1,289 vehicles per hour compared to fixed-time control. In the Zhuhai data set, average travel time was reduced by 42.3652 seconds, average queue length was reduced by 45.9167 meters, and traffic volume increased by 863 vehicles per hour. This represents a 52.31% reduction in average travel time, a 61.72% reduction in average queue length, and a 48.40% increase in traffic volume passing through the intersection during the fixed time period.

[0112] During off-peak hours: In the Quanzhou data set, average travel time was reduced by 17.8894 seconds, average queue length was reduced by 7.662 meters, and traffic volume increased by 1,521 vehicles per hour compared to fixed-time control. In the Zhuhai data set, average travel time was reduced by 40.2988 seconds, average queue length was reduced by 40.3949 meters, and traffic volume increased by 843 vehicles per hour. This represents a 56.37% reduction in average travel time, an 85.24% reduction in average queue length, and a 59.13% increase in traffic volume passing through the intersection during the fixed time period.

[0113] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0114] This paper proposes a single-intersection traffic light phase control method based on FR-DDQN. This method constructs a traffic light phase control model based on reinforcement learning and a fairness reward mechanism. This model considers both traffic efficiency and fairness, providing a more reasonable reward function for the traffic light phase control model. The traffic efficiency metric assesses the short-term impact of the selected action on the environment, while the fairness reward controls the long-term impact of the action selection strategy on the environment. This method effectively improves the overall traffic efficiency of the intersection over a period of time, providing a reference for the design of practical traffic light phase control.

[0115] Example 2

[0116] like Figure 9 As shown, the second embodiment of the present invention further provides a single intersection signal light phase control device based on FR-DDQN, comprising:

[0117] A simulation data acquisition unit, used to acquire historical traffic data of the intersection;

[0118] An agent construction unit is configured to construct an agent comprising a state space, an action space, and a fairness reward function; wherein the state space is the position distribution and speed of vehicles in the entrance lane; the action space is all non-conflicting phases of the intersection; and the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of each phase, the average travel time, the green light duration penalty, the red light waiting time penalty, and the traffic condition reward and penalty for each direction;

[0119] An algorithm network construction unit is configured to construct two networks with identical structures but different parameters, namely an estimation network and a target network, based on the historical traffic data and the intelligent agent; the estimation network calculates an estimated Q value corresponding to each action at the current intersection state; and the target network calculates a target value from the next intersection state to the terminal state after the action is executed;

[0120] a network calculation unit, configured to calculate an estimated Q value corresponding to each action in the current intersection state based on the estimation network, and select an action using an ε-greedy strategy; after selecting an action, calculate a reward value using the fairness reward function, and store the reward value in an experience replay set; and calculate a target value from the next intersection state to the terminal state after executing the action based on the target network;

[0121] The network update unit is used to calculate the error between the target value and the estimated Q value using the loss function MSE, and update the parameters of the estimation network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, thereby constructing a single intersection signal light phase control algorithm based on the fairness reward mechanism.

[0122] Example 3

[0123] The third embodiment of the present invention also provides a single-intersection signal light phase control device based on FR-DDQN, which includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the single-intersection signal light phase control method based on FR-DDQN as described above.

[0124] Example 4

[0125] The fourth embodiment of the present invention further provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of the device where the computer-readable storage medium is located, the single-intersection signal light phase control method based on FR-DDQN as described above is implemented.

[0126] In the several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device and method embodiments described above are merely illustrative. For example, the flowcharts in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of a code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.

[0127] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0128] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. It should be noted that, in this article, the terms "include", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0129] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0130] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0131] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0132] The "first" and "second" mentioned in the embodiments are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the specific order or precedence of "first" and "second" can be interchanged where appropriate. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0133] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A single intersection signal light phase control method based on FR-DDQN, characterized in that: include: S1, obtain historical traffic data of the intersection; S2. Construct an intelligent agent, which includes a state space, an action space, and a fairness reward function. The state space is the position distribution and speed of vehicles in the entrance lane; the action space is all non-conflicting phases of the intersection; and the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of each phase, the average travel time, the green light duration penalty, the red light waiting time penalty, and the traffic condition reward and penalty for each direction. S3, constructing two networks with the same structure but different parameters, namely an estimation network and a target network, based on the historical traffic data and the intelligent agent; the estimation network calculates the estimated Q value corresponding to each action in the current intersection state; and the target network calculates the target value from the next intersection state to the terminal state after the action is executed; S4, calculating the estimated Q value corresponding to each action in the current intersection state according to the estimation network, and selecting an action using the ε-greedy strategy; after selecting the action, calculating the reward value using the fairness reward function, and storing the reward value in the experience replay set; calculating the target value from the next intersection state to the terminal state after executing the action according to the target network; S5, using the loss function MSE to calculate the error between the target value and the estimated Q value, and updating the parameters of the estimation network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, thereby constructing a single intersection signal light phase control algorithm based on a fairness reward mechanism.

2. A single intersection signal light phase control method based on FR-DDQN according to claim 1, characterized in that ,The historical traffic data includes: the number of entrance lanes at the intersection, the ,historical traffic volume at each entrance of the intersection, and the ,maximum speed limit of the road.

3. The single intersection signal light phase control method based on FR-DDQN according to claim 2, characterized in that: When constructing the fairness reward function, we consider both traffic efficiency and fairness. For traffic efficiency, we consider the impact of each decision period and a traffic light cycle. In the decision-making period, the changes in queue length and the green light time utilization rate of each phase are selected to reflect the short-term impact of the current decision on the traffic efficiency per unit time; within a traffic light cycle, the average travel time is selected to evaluate the impact of the selected action set on the traffic efficiency within a cycle; in terms of fairness, constraints are established and penalty functions are set based on the green light duration and red light waiting time of each phase and the traffic conditions of each traffic direction within a traffic light cycle; The average queue length is obtained by calculating the difference between the average queue lengths of each approaching direction between two adjacent decision points. If the difference is a positive number, it means that the action has reduced the average queue length of the intersection; if it is a negative number, it means that the action has increased the average queue length of the intersection. The green light time utilization rate is obtained by calculating the green light time utilization rate of the two traffic directions within the phase. The larger the green light time utilization rate, the more vehicles pass through the traffic direction corresponding to the phase within a unit time period, and the greater the contribution to improving the traffic efficiency of the intersection; otherwise, the contribution is smaller. The formula for the green light time utilization rate u is: Among them, di represents a certain traffic direction at the intersection, q' di represents the traffic flow through the intersection in the di direction, cp di Indicates the road capacity in the di direction; The green light duration penalty term e1 is the minimum green light duration g min and maximum green light duration g max Establish a penalty term. If the duration of the green light is within the green light range, e1 is 0. If it is not, the closer it is to the green light range, the smaller e1 is, and vice versa. The red light waiting time penalty term e2 is based on the maximum red light waiting time rw max Establish a penalty item. If the waiting time for a red light in a certain direction of traffic exceeds rw max , then the penalty term e2 is If it does not exist, the penalty term e2 is 0; The traffic condition reward and penalty items e3 in each direction are determined by judging each maximum signal light cycle cy max The traffic conditions of all traffic directions within the area are used to establish reward and penalty items; if the cycle cy max If all traffic directions are clear, e3 is a bonus. If you are not satisfied, you will be punished That is, negative numbers The expression of the penalty function is as follows: in, represents the reward and punishment scale, g represents the duration of the green light in the current phase, and rw represents the duration of the phase waiting for the red light; The formula of the fairness reward function is obtained as follows: Among them, r t is the fairness reward function at the current time t; queue t is the average queue length at the intersection at the current time t, queue t-1 is the average queue length at the intersection at the previous moment; k1, k2, k3, k4, k5, k6 are weight coefficients; They respectively represent the green light time utilization rates of the traffic directions di1 and di2 corresponding to the current green light phase; travel_time is the average travel time. If travel_time is not 0, the smaller the travel_time is, the faster the vehicles pass through the intersection and the better the traffic conditions are; the larger the travel_time is, the worse the intersection conditions are; if travel_time is 0, it is necessary to judge whether there are no vehicles entering the corresponding section or whether there is traffic congestion that prevents vehicles from passing through the corresponding section based on the queue length. If there are no vehicles in the queue, it means that no vehicles have entered the corresponding section, and if there are vehicles in the queue, it means that there is traffic congestion.

4. The single intersection signal light phase control method based on FR-DDQN according to claim 3 is characterized in that , the calculation formula of the target value is: Among them, y t (s t+1 ,a';θ') is the target value at time t; s t 、s t+1 Respectively represent the execution of action a t Traffic flow status before and after; a t represents the action selected by the agent at time t; r t Indicates execution of action a t The reward after Q t (s t+1 ,a';θ) is the state s t+1 The estimated Q value corresponding to the next action a'; In state s t+1 The action a' corresponding to the maximum Q value in the estimated network is selected below, γ represents the discount coefficient, θ represents the parameters of the estimated network, and θ' represents the parameters of the target network.

5. The single intersection signal light phase control method based on FR-DDQN according to claim 1 is characterized in that When the ε-greedy strategy is used to select actions, the exploration and utilization in reinforcement learning are achieved by adjusting the exploration rate ε; where ε represents the probability of randomly selecting an action, 1-ε represents the probability of selecting the action corresponding to the maximum estimated Q value, and ε decreases as the number of iterative training increases.

6. The single intersection signal light phase control method based on FR-DDQN according to claim 1 is characterized in that When updating the parameters of the algorithm network by the gradient descent method, the CNN network is used to fit the Q-value function corresponding to the state action to solve the problem that it is impossible to list the Q-value function corresponding to each state when continuous large-scale states are generated during the operation of traffic flow; wherein, the CNN network is divided into 4 layers, the first layer consists of 32 convolution kernels with a step size of 2; the second layer consists of 64 convolution kernels with a step size of 1; the third layer is a fully connected layer composed of 128 neurons; the fourth layer is a fully connected layer composed of 64 neurons, and the fully connected layers all use the Relu activation function.

7. The single intersection signal light phase control method based on FR-DDQN according to claim 4 is characterized in that , the formula of the loss function MSE is: Where m is the number of cells into which the entrance lane is divided, Q t (s t ,a t ;θ) is the current state s t Each action a t The corresponding estimated Q value.

8. The single intersection signal light phase control method based on FR-DDQN according to claim 7 is characterized in that ,The update formula of the estimation network is: Q t+1 (s t ,a t ;θ)=Q t (s t ,a t ;θ)+α[y t (s t+1 ,a';θ')-Q t (s t ,a t ;i)]; Here, α represents the learning rate.

9. The single intersection signal light phase control method based on FR-DDQN according to claim 8, characterized in that , the experience replay set includes state s t 、Action a t , reward r t and state s t+1 , when training the algorithm network, a small batch sampling strategy is adopted to randomly and uniformly sample from the experience replay set.

10. A single intersection signal light phase control system based on FR-DDQN, characterized in that: include: A simulation data acquisition unit, used to acquire historical traffic data of the intersection; An agent construction unit is configured to construct an agent comprising a state space, an action space, and a fairness reward function; wherein the state space is the position distribution and speed of vehicles in the entrance lane; the action space is all non-conflicting phases of the intersection; and the fairness reward function is obtained by weighting the average queue length, the green light time utilization rate of each phase, the average travel time, the green light duration penalty, the red light waiting time penalty, and the traffic condition reward and penalty for each direction; An algorithm network construction unit is configured to construct two networks with identical structures but different parameters, namely an estimation network and a target network, based on the historical traffic data and the intelligent agent; the estimation network calculates an estimated Q value corresponding to each action at the current intersection state; and the target network calculates a target value from the next intersection state to the terminal state after the action is executed; a network calculation unit, configured to calculate an estimated Q value corresponding to each action in the current intersection state based on the estimation network, and select an action using an ε-greedy strategy; after selecting an action, calculate a reward value using the fairness reward function, and store the reward value in an experience replay set; and calculate a target value from the next intersection state to the terminal state after executing the action based on the target network; The network update unit is used to calculate the error between the target value and the estimated Q value using the loss function MSE, and update the parameters of the estimation network and the target network respectively by the gradient descent method until the maximum number of simulations is reached, thereby constructing a single intersection signal light phase control algorithm based on the fairness reward mechanism.

Images