An adaptive control method for signalized intersections based on improved deep Q-network

Abstract

The present invention relates to a signal intersection adaptive control method based on an improved deep Q network, comprising the following steps: obtaining vehicle passing data in an intersection environment, and calculating real-time vehicle queuing information; defining an intersection signal control agent based on the queuing information, including an action space, a state space, and a reward function, wherein a minimum green light duration constraint is set in the action space, the state space is constructed using the maximum queuing principle, and the reward function is constructed by predicting the number of queuing vehicles; online learning and storing the interaction strategy between the agent and the intersection environment based on a deep Q network; selecting the optimal behavior strategy of the agent from the interaction strategy and executing it to achieve traffic signal control. Compared with the prior art, the present invention has the advantages of enhancing robustness, reducing the average delay of vehicles, and improving the traffic efficiency of intersections.

Description

Technical Field

[0001] The present invention relates to the field of traffic signal control, and in particular to a signal intersection adaptive control method based on an improved deep Q network. Background Art

[0002] Adaptive control of signalized intersections is considered to be the most basic means to improve traffic efficiency. In recent decades, methods based on deep reinforcement learning have become a research hotspot in the field of adaptive traffic signal control. Deep reinforcement learning uses one or more neural networks to approximate the mapping relationship between traffic states and corresponding signal control plans. Through a large number of sequential learning, the value function drives the parameters to decrease so that the mapping approaches the optimal result. Deep reinforcement learning has the advantages of fast calculation and description of complex problems. Existing research has achieved superior results in reducing delays based on deep reinforcement learning methods.

[0003] However, the real world is far more complex than the simulated environment. In the adaptive traffic signal control algorithm approximated by artificial neural networks, the sudden increase in traffic demand may cause instability. In addition, the deep reinforcement learning-based method has been accused of lacking interpretability and robustness. The control error or performance of any node in the network may lead to unfavorable situations or even the collapse of the entire network. Therefore, in addition to discussing the average performance of traffic signal control performance, more attention needs to be paid to the examination or enhancement of its robustness. Summary of the invention

[0004] The purpose of the present invention is to provide a signal intersection adaptive control method based on an improved deep Q network with enhanced robustness.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A signalized intersection adaptive control method based on an improved deep Q network comprises the following steps:

[0007] Obtain vehicle passing data in the intersection environment and calculate real-time vehicle queue information;

[0008] Based on the queuing information, an intersection signal control agent is defined, including an action space, a state space and a reward function, wherein a minimum green light duration constraint is set in the action space, the state space is constructed using the maximum queuing principle, and the reward function is constructed by predicting the number of vehicles in the queue;

[0009] Online learning and storage of interaction strategies between the agent and the intersection environment based on a deep Q network;

[0010] The optimal behavior strategy of the intelligent agent is selected from the interaction strategies and executed to achieve traffic signal control.

[0011] Furthermore, the queuing information of the vehicles is the number of queuing vehicles, and the calculation formula is:

[0012]

[0013]

[0014]

[0015] Where wq is the velocity of the gathering wave, m / s; KI is the arrival flow density, veh / km / ln; k jam is the congestion density, veh / km / ln; v f is the free flow velocity, m / s; wd is the evacuation wave velocity, m / s; SFR is the saturation flow rate, veh / h / ln; k c is the critical density, veh / km / ln; τ is the time interval; l is the lane; t is the time; NI is the number of input vehicles; i is the intersection; N l i (t) is the number of vehicles in the queue; It is the release permission for lane l at phase φ of intersection i. When the value is 1, vehicles in all lanes corresponding to the phase are released. When the value is 0, vehicles are not released.

[0016] Furthermore, the process of defining the state space specifically includes:

[0017] The queue information is processed based on the maximum queue principle to obtain a state vector, which is expressed as:

[0018]

[0019] In the formula, s i (t) is the state vector, i is the intersection, t is the time; l is the lane; a i (t-τ) is the previous action, τ is the time interval; phaseφ is the phase number; is the number of queued vehicles in lane l at phase φ.

[0020] A state space is constructed based on the state vector.

[0021] Furthermore, the expression of the action space is:

[0022]

[0023] Where i is the intersection; a i (t) is the action space of intersection i at time t; is the φth phase plan created for intersection i at time t.

[0024] Furthermore, the expression of the minimum green light duration constraint is:

[0025]

[0026] Where GMIN is the minimum green light duration, i is the intersection, p is the phase; τ is the time interval; SFR is the saturated flow rate, veh / h / ln; N l i (t) is the number of vehicles in the queue.

[0027] Furthermore, the reward function construction process specifically includes:

[0028] Using traffic wave theory to predict the number of queued vehicles, the expression is:

[0029]

[0030]

[0031] Where CI means that if the signal controller switches to a new phase at the next time step, it will last at least GMIN seconds, otherwise it will only last τ seconds, GMIN is the minimum green light duration, τ is the time interval, i is the intersection, l is the lane, t is the time, and a is the time. i (t-τ) is the previous action; is the predicted number of vehicles in the queue; N l i (t) is the number of vehicles in the queue; wq is the speed of the assembly wave, m / s; k jam is the congestion density, veh / km / ln; wd is the evacuation wave speed, m / s; is the clearance for lane l at phase φ of intersection i; NM is the maximum length of the vehicle queue at the intersection.

[0032] Based on the predicted number of queued vehicles, the expected reward is calculated and used as the reward function. The calculation expression of the expected reward is:

[0033]

[0034] In the formula, r i It is the expectation of reward.

[0035] Furthermore, the learning process of the interaction strategy specifically includes:

[0036] The deep Q network uses Q-values ​​to evaluate the benefits of taking actions in traffic conditions at intersections;

[0037] Collect transfer samples and put them into the experience playback memory pool;

[0038] The multi-layer perceptron in deep learning is used to model the behavior strategy, with the input being the state vector and the output being the Q value corresponding to the action taken under the state vector;

[0039] Update neural network parameters through back-propagation and gradient descent algorithms;

[0040] Repeat the above steps to obtain the interaction strategy between the agent and the intersection environment.

[0041] Furthermore, the calculation expression of the income is:

[0042]

[0043] In the formula, Q(s i (t),a i (t)) represents the traffic state s i (t) Take action a i (t) The benefit obtained, i.e., the Q value; r i (t+τ) means taking action a i (t) is the immediate reward, γ is the discount factor, Indicates that in state s i At (t+τ), the agent obtains the maximum Q value by adopting the optimal behavior strategy.

[0044] Furthermore, the loss function expression of the deep Q network is:

[0045]

[0046] Where L is the loss value; is the predicted Q value of the neural network for the sample, s i (t) is the state, a i (t) is the action, θ is the neural network parameter; Q(s i (t),a i (t)) is the target Q value of the sample.

[0047] Furthermore, the expression for updating the neural network parameters is:

[0048]

[0049] Where θ is the neural network parameter; L is the Q network loss value; α is the learning rate.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] (1) The present invention minimizes the impact of unpredictable dynamic interference from the outside by improving mechanisms such as predicting the number of queued vehicles, the maximum queue and the minimum green light duration, and maintains rationality in a constantly changing environment. The control strategy is gradually changed by the interaction between the intelligent agent and the intersection environment through Q network online learning, thereby significantly enhancing the robustness of the control method.

[0052] (2) The present invention calculates the number of vehicles queuing at an intersection in real time, uses the queuing information to generate the next decision and publish a signal timing plan, allowing the traffic signal control agent to make decisions in real time according to the current traffic status. Compared with other adaptive methods, it significantly reduces the average delay of vehicles and improves the traffic efficiency of the intersection. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a schematic diagram of a method flow of an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of a method for estimating the number of vehicles in a queue provided by an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of a signal control phase plan diagram provided by an embodiment of the present invention;

[0056] Figure 4 A schematic diagram of a parameter updating method for a traditional deep Q network provided by an embodiment of the present invention;

[0057] Figure 5 A schematic diagram of a parameter updating method for an improved deep Q network provided by an embodiment of the present invention;

[0058] Figure 6 It is a schematic diagram of a single intersection and simulation verification in an embodiment of the present invention;

[0059] Figure 7 It is a schematic diagram of the comparison of average delays under different traffic demand levels in the empirical and simulation verification in the embodiment of the present invention;

[0060] Figure 8 It is a schematic diagram of the comparison of average delays under different traffic demand allocation ratios in the empirical and simulation verifications in the embodiments of the present invention;

[0061] Fig. 9 It is a schematic diagram of the comparison of average delays under different steering ratios in the empirical and simulation verification in the embodiment of the present invention;

[0062] Fig.10 It is a schematic diagram for comparing the average delays of stimulation tests in the empirical and simulation verifications in the embodiments of the present invention. DETAILED DESCRIPTION

[0063] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0064] This embodiment provides a signal intersection adaptive control method based on an improved deep Q network, such as Figure 1 As shown, the method comprises the following steps:

[0065] S1. Obtain vehicle passing data in the intersection environment and calculate real-time vehicle queuing information.

[0066] like Figure 2 As shown, the signalized intersection controller uses geomagnetism and coils to collect vehicle passing data and calculate the real-time number of vehicles queuing in the intersection entrance lane.

[0067]

[0068]

[0069]

[0070] Where wq is the velocity of the gathering wave, m / s; KI is the arrival flow density, veh / km / ln; k jam is the congestion density, veh / km / ln; v f is the free flow velocity, m / s; wd is the evacuation wave velocity, m / s; SFR is the saturation flow rate, veh / h / ln; k c is the critical density, veh / km / ln; τ is the time interval; l is the lane; t is the time; NI is the number of input vehicles; i is the intersection; N l i (t) is the number of vehicles in the queue; It is the release permission for lane l corresponding to phase φ in intersection i. When the value is 1, vehicles in all lanes corresponding to the phase are released, and when the value is 0, vehicles are not released.

[0071] S2. Based on the queuing information, an intersection signal control agent is defined, including an action space, a state space and a reward function, wherein a minimum green light duration constraint is set in the action space, the state space is constructed using the maximum queuing principle, and the reward function is constructed by predicting the number of vehicles in the queue.

[0072] This embodiment uses queuing information to generate the next decision and publish a signal timing plan. Action a represents the agent's behavioral response to the current traffic status, that is, the signal timing plan; state s describes the traffic status of the intersection; and reward r is the evaluation feedback of the intersection environment on the behavior taken.

[0073] S21, Action Space: Considering that fixed phase sequence is disadvantageous for the agent to cope with traffic flow fluctuations, this embodiment patent assumes that the signalized intersection adaptively adjusts the phase duration according to the traffic state. The agent's actions include extending the current phase duration or switching to another phase. In this embodiment patent, regardless of how the traffic light is adjusted, the vehicle's right turn is legal. Based on the NEMA ring concept, 8 optional phases can be created, such as Figure 3 As shown, the action space can be expressed as follows:

[0074]

[0075] The construction feature of the action space in this embodiment of the patent is that, assuming the agent chooses to switch to another phase, the duration of the new phase should be greater than the minimum green light duration GMIN to avoid unreasonable phase switching. The calculation formula is as follows:

[0076]

[0077] Among them, <> means rounding the internal numbers.

[0078] S22, state space: The state space in this embodiment describes the traffic state of the intersection, and is characterized by using the maximum queue principle to perform secondary processing on the queue information. This improvement enables the agent to identify the key state that determines each phase. The state vector s is constructed according to the maximum value of the traffic state of the lane corresponding to each phase as follows:

[0079]

[0080] In order to make the agent understand the current signal control situation, the previous action a i (t-τ) is also appended to the state vector. A state space is constructed based on the state vector.

[0081] S23, Reward function: The traffic signal control agent aims to reduce the number of queued vehicles to improve traffic efficiency. Therefore, the present embodiment of the patent uses the sum of the number of queued vehicles in all entrances of the intersection as the main criterion. According to the traditional method, the action a(t) decided at the current time t is used to calculate the reward feedback r(t+τ) based on the passing vehicle data collected by the detector at the next time step t+τ, such as Figure 4 However, since the arrival of vehicles is completely unpredictable and the evolution of traffic volume is a random process, it is difficult for the agent to converge and achieve acceptable performance when interacting with such a complex and random environment. t+τ This embodiment uses traffic wave theory to predict the number of queued vehicles (See attached Figure 5 ), the calculation formula is as follows:

[0082]

[0083]

[0084] Where CI means that if the signal controller switches to a new phase at the next time step, it will last at least GMIN seconds, otherwise it will only last τ seconds, GMIN is the minimum green light duration, τ is the time interval, i is the intersection, l is the lane, t is the time, and a is the time. i (t-τ) is the previous action; is the predicted number of vehicles in the queue; N l i (t) is the number of vehicles in the queue; wq is the speed of the assembly wave, m / s; k jam is the congestion density, veh / km / ln; wd is the evacuation wave speed, m / s; is the clearance for lane l at phase φ of intersection i; NM is the maximum length of the vehicle queue at the intersection.

[0085] Based on the predicted number of vehicles in the queue, the expected reward is calculated and used as the reward function. The calculation formula is as follows:

[0086]

[0087] In this embodiment, using the predicted number of queued vehicles is one of the methods to achieve robust control, which reduces the uncertainty between the current time step and the next time step, allowing the agent to achieve timely online learning and make the most intuitive and conservative decisions to adapt to dynamically changing traffic conditions.

[0088] S3. Based on the deep Q network, the interaction strategy between the intelligent agent and the intersection environment is learned and stored online.

[0089] This step uses the deep Q network to learn the behavior strategy a=π(s) online, and applies the value function, i.e., the Q value, to evaluate the benefits of taking action a at intersection i in traffic state s. The learning goal of the deep Q network is to select the appropriate behavior strategy for the intersection so as to achieve the maximum benefit in the current traffic environment. Specifically:

[0090] S31. Deep Q network uses Q value to evaluate intersection i in traffic state s i (t) Take action a i (t) The profit obtained is calculated as follows:

[0091]

[0092] Among them, r i (t+τ) means taking action a i (t) is the immediate reward, γ is the discount factor, Indicates that in state s i At (t+τ), the Q value obtained by the agent adopting the optimal behavior strategy.

[0093] S32, collect transfer samples and put them into the experience playback pool. Collect transfer sample data (s n ,a n ,r n+τ ,s n+τ ) and stored in the experience replay memory pool.

[0094] S33. Use the multi-layer perceptron in deep learning to model the behavior strategy a=π(s), with the input being the state vector s and the output being the Q value corresponding to taking action a under this traffic state. Train the neural network based on the sample data in the experience playback memory pool, and the loss function L is defined as:

[0095]

[0096] Among them, θ is the parameter of the neural network, is the predicted Q value of the neural network for the sample, Q(s i (t),a i (t)) is the target Q value of the sample.

[0097] S34. Update the neural network parameters through back propagation and gradient descent algorithm, as follows:

[0098]

[0099] S4. Selecting and executing the optimal behavior strategy of the intelligent agent from the interaction strategy to achieve traffic signal control.

[0100] The intersection signal control agent observes and processes the intersection traffic state s(t) at time t, and takes the action a(t) corresponding to the maximum Q value according to the behavior strategy learned by the Q network. The total number of queued vehicles at the next time step t+τ is pre-calculated based on the traffic wave theory. and status Added experience replay memory pool. Specifically:

[0101] S41, calculating the number of real-time queued vehicles in the intersection. Geomagnetic devices, coils, etc. are set at the upstream of the intersection entrance and the stop line to collect vehicle passing data, and the number of real-time queued vehicles in the intersection entrance is calculated every τ seconds as described above, and the obtained queue information is processed twice using the maximum queue principle to obtain a state vector.

[0102] S42, make a signal control decision for the next time step based on the deep Q network. The deep Q network that has completed pre-training takes the state vector as input, outputs the Q values ​​corresponding to all actions, and selects the action corresponding to the maximum Q value as the signal control solution for the next time step:

[0103]

[0104] S43, Q network online learning. According to the above prediction at time t, the instant reward is obtained and the next time step state Transferring samples Add experience replay memory pool and update Q network parameters.

[0105] S44, update the traffic status and re-assign the value, repeat the above steps to enter the next round of signal control plan decision.

[0106] This embodiment also verifies and analyzes the adaptive control model of a single signal intersection based on an improved deep Q network based on an empirical verification scenario, and uses SUMO (Simulation of Urban Mobility) to establish a simulation model of a signal intersection. At each time step, all vehicle information is transmitted from SUMO to Python through TraCI (Traffic Control Interface) to make decisions on the next signal control plan and return to SUMO. This embodiment conducts comparative analysis under different traffic demand levels, traffic demand distribution ratios, and turning ratios, and sets up a stimulation experiment to verify the stability of the adaptive control method for signal intersections under unstable traffic demand scenarios.

[0107] (1) The signalized intersection simulation model was established using SUMO software to compare and verify the adaptive control method of signalized intersections based on the improved Q network in different traffic environments. The vehicle parameters in the signalized intersection simulation were set according to the default settings of SUMO, with a vehicle speed limit of 50 km / h, a saturation flow rate of 1200 veh / h / ln, a time interval of τ of 5 s, a decay of γ of 0.9, and a learning rate of α of 0.1.

[0108] For ease of explanation, the maximum traffic flow of each entrance lane is used to represent the demand level of the entrance lane, such as V N ,V S ,V W and V E The verification part of this embodiment assumes that two-phase signal control is adopted, and the entrances in each direction in the same phase have the same traffic demand (i.e. V N =V S ,V W =V E). At the same time, all the entrances in the simulation scenario have the same steering ratio setting, denoted as V R :V TH :V L (See attached Figure 6 ). First, the benchmark scenario 1 is set as follows: V N (V S ,)=V W (V E )=360veh / h / ln,V R :V TH :V L =0.1:0.8:0.1, and another 6 demand scenarios are generated by adjusting the traffic demand level, demand allocation ratio and steering ratio.

[0109] In addition, three signal control methods are used to compare the control performance of the method described in this embodiment, namely DQN-S (Deep Q-Network-State), DQN-R (Deep Q-Network-Reward) and LQF (the Longest Queue First).

[0110] Compared with the proposed method, DQN-S extracts vehicle information from all lanes. Taking the intersection scenario with four entrances and three lanes as an example, it has information on the number of vehicles in 12 lanes:

[0111]

[0112] DQN-R uses the actual number of vehicles observed in the next time step to construct the reward function as follows:

[0113]

[0114] LQF is an adaptive control method proposed by Wunderlich et al. in the paper A novel signal-scheduling algorithm with Quality-of-service provisioning for an isolated intersection. For each time step τ, the LQF model counts the number of waiting vehicles in each lane and selects the phase corresponding to the lane with the largest number of vehicles. Considering the randomness of vehicle arrival, 50 random seeds are generated for each scenario for 50 rounds of verification. In each round, the first 15 minutes are warm-up time, and the average delay of all vehicles is evaluated for 60 minutes. The simulation results are displayed in groups.

[0115] (2) To verify the traffic signal control performance under different traffic demand levels, the following three demand levels are set: (a) V N (V S )=V W (V E )=240veh / h / ln;(b)V N (V S )=V W (V E )=360veh / h / ln;(c)V N (V S )=V W (V E )=480veh / h / ln. Figure 7 As shown, under all three demand level settings, the scheme proposed in this embodiment patent maintains good control performance compared with the other two signal control methods. First, the proposed method has lower average delays in 50 rounds of experiments in three scenarios. As the demand level increases, the difference gradually widens. The results show that the reward function and state settings have achieved the expected effect and greatly improved the traffic conditions at a single signal intersection. The results also show that this effect is more significant under high traffic pressure. Secondly, by observing the box shape, it can be found that the variance of the proposed method is significantly smaller than that of DQN-S and DQN-R. From the simulation records, it can be found that the single intersection learning scheme can correctly handle frequent and highly variable arrival flows without making too many adverse behaviors that are far from normal human understanding. When the traffic volume changes significantly, the proposed method can quickly converge to a reasonable range. However, when faced with environmental changes, DQN-S and DQN-R may spend a long time in the wrong search space. Therefore, compared with the other two methods, the proposed method is more stable when dealing with random arrival flows. In particular, as the traffic demand increases from 240veh / h / ln to 480veh / h / ln, the standard deviation of DQN-S becomes significantly larger. The results further show that reducing the state space using the LQF theory can significantly improve the robustness of the DQN-based control method. Third, there is a significant gap between the results of the LQF method and the other three reinforcement learning-based methods. At the same time, the gap between them increases with the increase of traffic demand. This finding strongly supports our original intention of applying reinforcement learning, that is, artificial intelligence with appropriate state settings and action spaces can significantly outperform rule-based control algorithms in complex and stochastic problems.

[0116] (3) To verify the traffic signal control performance under different traffic demand distributions, the following three scenarios are set: (a) V N (V S )=V W (V E)=360veh / h / ln;(b)V N (V S )=480veh / h / ln,V W (V E )=240veh / h / ln;(c)V N (V S )=540veh / h / ln,V W (V E )=180veh / h / 1n. Figure 8 As shown, with V N (V S ):V W (V E ) increases, and no obvious increasing or decreasing trend is found in the results of the four methods. Nevertheless, compared with the other three methods, the method proposed in this embodiment still has advantages.

[0117] (4) To verify the traffic signal control performance under different steering ratios, the following three scenarios are set: (a) V R :V TH :V L =0.1:0.8:0.1; (b) V R :V TH :V L =0.15:0.7:0.15; (c) V R :V TH :V L =0.2:0.6:0.2. Fig. 9 As shown in the figure, as the right-turn / left-turn traffic ratio increases, the gap between the average performance of the proposed method and DQN-A becomes smaller. The proposed method mainly targets the most congested lanes, which is suitable for the common situation of unbalanced lane usage in the approaches of most urban intersections. However, it inevitably underestimates the scenario where all lanes in the approach are used evenly but there are no extremely long queues. In particular, when the left-turn or right-turn traffic ratio increases, this effect becomes relatively frequent, making the advantage of the proposed method less prominent. In addition, in these scenarios, the robustness and stability of DQN-A and DQN-R are still poor.

[0118] (5) To verify the adaptability under different flow conditions, a stimulation experiment with two different input stages was designed. In the first stage, V N (V S )=V W (V E )=240veh / h / ln,V R :V TH :V L=0.1:0.8:0.1; in the second stage, set to V N (V S )=V W (V E )=240veh / h / ln,V R :V TH :V L = 0.1:0.8:0.1. In the experiment, the proposed robust learners DQN-S and DQN-R were compared, considering the traffic increase as a stimulus to the environment. The focus of the comparative study was on the adaptation process of each control algorithm (i.e., the fluctuation of the average delay in the second stage), as shown in the attached figure. Fig.10 shown.

[0119] First, the phase switching starts at 600 seconds, and the average delays of the three methods start to increase and fluctuate sharply from around 630 seconds. The fluctuation process represents the process of the agent adapting to the newly changed environment. Since the average delay is measured by the vehicle that completes the trip, there is a specific lag between the performance change and the phase transition. Second, the drastic fluctuations of DQN-S and DQN-R last for about 120s and 100s, respectively. In contrast, the cycle of the proposed method lasts only 70 seconds. It can be inferred that using the queue prediction model for the reward function and highlighting the longest queue length at each stage of the state function helps to quickly adapt to the sudden increase in traffic demand.

[0120] This embodiment relates to a signal intersection adaptive control method based on an improved deep Q network. The method sets a geomagnetic coil to collect vehicle data and calculate the number of real-time queued vehicles in the entrance lane of the intersection. Secondly, the queue information is processed twice using the maximum queue principle to obtain a state vector and input it into the deep Q network. Considering the minimum green light duration constraint, the signal control timing plan is output; at the same time, the traffic wave theory is introduced to calculate the total number of queued vehicles (reward expectation) in the next time step, which is used to train the deep Q network online. In this embodiment, the deep Q network is used to learn the behavior strategy of the signal control agent interacting with the intersection environment online. By introducing improvement mechanisms such as learning reward expectation, maximum queue, and minimum green light duration, the control stability of the intelligent signal control method under the condition of drastic fluctuations in traffic demand is effectively improved, highlighting the advantages of strong fluctuation adaptability and high traffic efficiency.

[0121] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0122] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0123] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0124] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0126] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0127] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A signalized intersection adaptive control method based on an improved deep Q network, characterized in that: The following steps are involved: Obtain vehicle passing data in the intersection environment and calculate real-time vehicle queue information; Based on the queuing information, an intersection signal control agent is defined, including an action space, a state space and a reward function, wherein a minimum green light duration constraint is set in the action space, the state space is constructed using the maximum queuing principle, and the reward function is constructed by predicting the number of vehicles in the queue; Online learning and storage of interaction strategies between the agent and the intersection environment based on a deep Q network; Selecting and executing the optimal behavior strategy of the agent from the interaction strategy to achieve traffic signal control; The queuing information of the vehicles is the number of queuing vehicles, and the calculation formula is: Where wq is the velocity of the gathering wave, m / s; KI is the arrival flow density, veh / km / ln; k jam is the congestion density, veh / km / ln; v f is the free flow velocity, m / s; wd is the evacuation wave velocity, m / s; SFR is the saturation flow rate, veh / h / ln; k c is the critical density, veh / km / ln; τ is the time interval; l is the lane; t is the time; NI is the number of input vehicles; i is the intersection; N l i (t) is the number of vehicles in the queue; is the release permission for lane l in phase φ at intersection i. When the value is 1, vehicles in all lanes corresponding to the phase are released, and when the value is 0, vehicles are not released; The reward function construction process specifically includes: Using traffic wave theory to predict the number of queued vehicles, the expression is: Where CI means that if the signal controller switches to a new phase at the next time step, it will last at least GMIN seconds, otherwise it will only last τ seconds, GMIN is the minimum green light duration, τ is the time interval, i is the intersection, l is the lane, t is the time, and a is the time. i (t-τ) is the previous action; is the predicted number of queued vehicles; Nli(t) is the number of queued vehicles; wq is the speed of the gathering wave, m / s; k jam is the congestion density, veh / km / ln; wd is the evacuation wave speed, m / s; is the clearance for the next lane l at phase φ of intersection i; NM is the maximum length of the vehicle queue at the intersection; Based on the predicted number of queued vehicles, the expected reward is calculated and used as the reward function. The calculation expression of the expected reward is: In the formula, r i It is the expectation of reward.

2. The signalized intersection adaptive control method based on improved deep Q network according to claim 1 is characterized in that: The expression of the action space is: Where i is the intersection; a i (t) is the action space of intersection i at time t; is the φth phase plan created for intersection i at time t.

3. The signalized intersection adaptive control method based on improved deep Q network according to claim 1 is characterized in that: The process of defining the state space specifically includes: The queue information is processed based on the maximum queue principle to obtain a state vector, which is expressed as: In the formula, s i (t) is the state vector, i is the intersection, t is the time; l is the lane; a i (t-τ) is the previous action, τ is the time interval; phaseφ is the phase number; is the number of queued vehicles in lane l at phase φ; A state space is constructed based on the state vector.

4. The signalized intersection adaptive control method based on improved deep Q network according to claim 1 is characterized in that: The expression of the minimum green light duration constraint is: Where GMIN is the minimum green light duration, i is the intersection, p is the phase; τ is the time interval; SFR is the saturated flow rate, veh / h / ln; N l i (t) is the number of vehicles in the queue.

5. The signalized intersection adaptive control method based on improved deep Q network according to claim 1 is characterized in that: The learning process of the interaction strategy specifically includes: The deep Q network uses Q-values ​​to evaluate the benefits of taking actions in traffic conditions at intersections; Collect transfer samples and put them into the experience playback memory pool; The multi-layer perceptron in deep learning is used to model the behavior strategy, with the input being the state vector and the output being the Q value corresponding to the action taken under the state vector; Update neural network parameters through back-propagation and gradient descent algorithms; Repeat the above steps to obtain the interaction strategy between the agent and the intersection environment.

6. The signalized intersection adaptive control method based on improved deep Q network according to claim 5, characterized in that: The calculation expression of the income is: In the formula, Q(s i (t),a i (t)) represents the benefit obtained by taking action ai(t) in traffic state si(t), i.e., Q value; r i (t+τ) represents the immediate reward obtained by taking action ai(t), γ represents the discount factor, Indicates that in state s i At (t+τ), the agent obtains the maximum Q value by adopting the optimal behavior strategy.

7. The signalized intersection adaptive control method based on improved deep Q network according to claim 5, characterized in that: The loss function expression of the deep Q network is: Where, L is the loss value; is the predicted Q value of the neural network for the sample, s i (t) is the state, a i (t) is the action, θ is the neural network parameter; Q(s i (t),a i (t)) is the target Q value of the sample.

8. The signalized intersection adaptive control method based on improved deep Q network according to claim 5, characterized in that: The expression for updating the neural network parameters is: Where θ is the neural network parameter; L is the Q network loss value; α is the learning rate.

Images