An adaptive signal control method for intersections based on cellular deduction and multi-step decision making

Abstract

The invention discloses an intersection adaptive signal control method based on cellular deduction multi-step decision, and relates to the technical field of traffic signal light control. The invention establishes a signal continuous control model based on a cellular transmission model, and the model uses cellular deduction to simulate the operation and transformation of traffic flow at an intersection, so as to realize the state transfer of the intersection; at the same time, the state obtained by cellular deduction is spliced ​​with the current phase scheme of the road, the lane-level vehicle arrival rate and departure rate of the previous cycle after feature extraction as the state space input of the model, so as to improve the accuracy of the intelligent agent's decision; after completing the pre-decision of all phases in the cycle by adopting a multi-step decision mechanism, the discrete action sequence output in the cellular environment is integrated into a signal timing scheme of a complete cycle and transmitted to the signal light for continuous control; while retaining the advantages of the flexibility and diversity of discrete actions, the problem that the intersection signal control model based on the deep reinforcement learning of discrete states is difficult to realize continuous action output is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic signal light control, and in particular to an intersection adaptive signal control method based on cellular deduction multi-step decision. Background Art

[0002] Traffic congestion has had a huge negative impact on economic, environmental and social benefits. Intersections are key nodes in the road network, and improving intersection efficiency is the key to solving traffic congestion problems. Among them, optimizing signal timing is the fastest, most effective and lowest-cost way to improve intersection efficiency. Currently, the widely used traffic signal control methods are mainly fixed signal timing and inductive signal control. The former controls intersection signals according to a pre-set cycle. Although the implementation cost is low, it lacks sufficient applicability in the face of complex and changeable dynamic traffic flows. The latter detects the arrival of vehicles at the intersection through pre-buried coil detectors and makes instant signal adjustments. Although it can effectively reduce the number of green lights left unused, it only considers the instantaneous traffic status, and the model setting is relatively simple, which cannot guarantee the control benefits in complex traffic scenarios.

[0003] In recent years, with the development of sensing technology, it is more convenient to obtain the state information of road traffic flow; reinforcement learning (RL) adopts the Markov decision process, which can find the optimal strategy through continuous interaction with the environment. It is suitable for traffic signal control problems oriented to sequential decisions, and has shown excellent application prospects in the field of traffic signal control research. However, most reinforcement learning uses manual feature extraction to simplify the state space. The actual traffic environment is complex and changeable, and the increase in the amount of traffic data will greatly reduce the control effect of this method; deep learning (DL) uses deep neural networks to automatically extract data features and can effectively process high-dimensional state spaces; deep reinforcement learning algorithms (DRL) combine the strengths of both and are currently a research hotspot in the field of intelligent signal control.

[0004] To ensure the effectiveness of the model, the research on deep reinforcement learning signal control model has successively designed a variety of different state spaces. The state spaces widely used at this stage are roughly divided into two forms: image matrix and vector matrix. The image matrix obtains the state matrix including vehicle position, speed and other information through intersection snapshots. The vector matrix constructs a feature vector representing the state using the statistical data of each lane of the intersection after statistical analysis of the operating data of independent vehicles such as intersection vehicle speed and queue length. On the setting side of the action space, the current model has not yet realized the continuous control of traffic signals, and most of the action plans are set to dynamic delay of phase or random switching of phase. However, in practice, the scheme for optimizing the phase will cause sudden changes in traffic lights, which will have a negative impact on traffic stability and safety, and cannot be applied to traffic lights with countdown functions, which hinders the practical application of deep reinforcement learning adaptive signal control and makes it difficult to completely realize the adaptive continuous control of traffic signals. Summary of the invention

[0005] The embodiment of the present invention provides an intersection adaptive signal control method based on cellular deduction multi-step decision, which can solve the problem in the prior art that it is difficult to completely realize the adaptive continuous control of traffic signals.

[0006] The embodiment of the present invention provides an intersection adaptive signal control method based on cellular deduction multi-step decision, comprising the following steps:

[0007] Discretize the road according to the current road image to obtain discrete grids, and obtain the position and speed of the vehicle on the current road corresponding to each discrete grid;

[0008] The cellular transmission model CTM and the deep reinforcement learning adaptive signal discrete control model DRLTSC are constructed respectively, and the cellular transmission model CTM and the deep reinforcement learning adaptive signal discrete control model DRLTSC are cascaded to construct the intersection deep reinforcement learning adaptive signal continuous control model CTMDRL; and the intersection deep reinforcement learning adaptive signal continuous control model CTMDRL is trained using the DDQN algorithm and the D3QN algorithm to obtain the trained intersection deep reinforcement learning adaptive signal continuous control model CTMDRL;

[0009] The cellular transmission model CTM in the trained intersection deep reinforcement learning adaptive signal continuous control model CTMDRL is used to set the discretized grid of the road as an independent cell, and the driving process of the vehicle on the road is set as a cellular transmission process. The cellular environment is initialized according to the position and speed of the vehicle, and the operation and transformation of the intersection traffic flow are simulated by cellular deduction based on the full speed difference FVD following model to obtain a state matrix containing the intersection vehicle position and vehicle speed; the state matrix is ​​subjected to feature extraction through the output layer of the cellular transmission model CTM to obtain the eigenvector of the vehicle information state matrix;

[0010] Using the trained deep reinforcement learning adaptive signal discrete control model DRLTSC in the intersection deep reinforcement learning adaptive signal continuous control model CTMDRL, the eigenvector of the vehicle information state matrix, the current execution phase of the intersection, and the lane-level vehicle arrival rate and departure rate in the previous cycle are concatenated to form a state vector; the intelligent agent in the deep reinforcement learning adaptive signal discrete control model DRLTSC is used to pre-decide on multi-step discrete actions for the state vector to obtain discrete actions, and the discrete actions are integrated into a signal timing plan for a complete cycle; the signal timing plan is transmitted to the traffic light to realize continuous control of the traffic signal.

[0011] Preferably, obtaining a discrete grid comprises:

[0012] The vehicle position and speed at the lane level are obtained from the intersection snapshot according to the vehicle network method, the partition length is set to 5m, and it is stipulated that each discrete grid contains at most one vehicle;

[0013] The 1000m×1000m environment centered at the intersection is divided into 200×200 square grids of the same size at 5m intervals. The state information contained in each grid is a binary vector of the vehicle's position and speed.

[0014] Preferably, the initializing the cellular environment includes:

[0015] The cellular environment is initialized with a 200×200 discrete grid centered at the intersection at the beginning of each cycle. The size of the unit cell is consistent with the discrete grid, and each cell can only accommodate one vehicle, that is, the state information carried in the cell is consistent with the data structure of the discrete grid;

[0016] The step length of the cellular deduction model is set to 1s, and the full velocity difference FVD following model is used to describe the following trajectory of the artificially driven vehicle in the cell. The acceleration of the following vehicle is expressed as:

[0017] a cc (t) = α[V(Δd i (t))-v i(t)]+β[v i+1 (t)-v i (t)]

[0018] V(Δd i (t))=V1+V2tanh[c1(Δx i (t)-l)-c2]

[0019] Among them: a cc (t) represents the acceleration of the vehicle at time t; V(Δd i (t)) represents the optimized speed function; l is the vehicle length; V1, V2, c1, c2 are the coefficients of the optimized speed function; α is the driver sensitivity coefficient; β is the speed difference reaction coefficient; v i (t) represents the speed of vehicle i in the cell at time t; v i+1 (t) represents the speed of the i+1th vehicle in the cell at time t; Δx i (t) represents the distance traveled by vehicle i in the cell at time t when the speed is optimized;

[0020] The speed of the vehicle at the next moment is:

[0021] v i (t+1)=v i (t)+a cc (t)·Δt

[0022] Where: Δt is the time interval; it can be obtained that during the cellular deduction process, the total displacement s of vehicle i at time t+1 i (t+1) is:

[0023]

[0024] Where: s t represents the displacement of the vehicle at time t; according to the above formula, the vehicle displacement s i (t+1) divided by 5 is used to calculate the cell position of vehicle i after it performs a discrete action in the environment at time t.

[0025] Preferably, extracting features from the state matrix through an output layer of a cellular transport model CTM comprises:

[0026] The state matrix obtained in the cellular deduction contains lane-level vehicle information and has a size of 60×60×2. After feature extraction through the output layer of the cellular transfer model CTM, the state matrix has an output size of 13×13×128.

[0027] Preferably, the method of using the DDQN algorithm and the D3QN algorithm to train the intersection deep reinforcement learning adaptive signal continuous control model CTMDRL includes:

[0028] The optimal action value function based on DQN is:

[0029]

[0030] Where: R t is the reward obtained at time t; γ is the discount factor after considering the long-term reward decay; Q is the action value function; θ is the parameter of the action value function; s t represents the displacement of the vehicle at time t; s t+1 represents the displacement of the vehicle at time t+1; a t represents the acceleration of the vehicle at time t; a t+1 represents the acceleration of the vehicle at time t+1;

[0031] The agent selects the maximum The corresponding action achieves reward G t The goal of maximization is set as the optimal strategy:

[0032]

[0033] At the same time, the parameters of the action value are updated by maximizing the mean square error loss function. The update method is:

[0034]

[0035] DDQN adopts the form of a dual Q network and introduces a target network into the network structure. Except for the weight parameters, the target network is exactly the same as the training network. The training network is used to estimate the current Q θ (s t , a t ), the target network is used to estimate The deviation caused by bootstrapping can be alleviated, and the optimal action value function is:

[0036]

[0037] Where: γ is the discount factor; Q′ θ′ is the target Q value; θ′ is the target network parameter; argmax a∈A Q θ (s t+1 ,a) from the current network;

[0038] Dueling DQN sets the optimal action value Q * Decompose into optimal state value V * (s t ) and optimal advantage D * (s t , a t ), its optimal action value function is expressed as:

[0039]

[0040] Among them: When the action takes the action corresponding to the optimal state value, The action-value function achieves the optimal value;

[0041] The D3QN algorithm combines the advantages of the DDQN and Dueling DQN algorithms to further improve the learning efficiency and stability of the algorithm. It is very effective in dealing with discrete action space problems. DDQN and D3QN algorithms are used to update the intelligent agent.

[0042] Preferably, obtaining the discrete action comprises:

[0043] The agent makes a pre-action decision based on the state information obtained from the cellular deduction and sets the action set to (0,1); when the agent selects action a=0, the model extends the current phase by 3s and stores it in the pre-decision action space; when the agent selects action a=1, the model switches to the yellow light phase for 3s and then switches to the next phase in the pre-set phase sequence and executes for 3s, which is also stored in the pre-decision action space. The agent performs multi-step pre-decision until the pre-decision action space contains four actions with a=1; this round of decision-making stops and outputs discrete actions.

[0044] Preferably, the reward function used in the training of the agent is:

[0045]

[0046] Where: r t Represents the execution of action a t The reward given to the agent by the environment at that time; l represents the entrance lane; L represents the total number of entrance lanes at the intersection; c represents vehicles; C represents the total number of vehicles on the corresponding entrance lane; w l，c Represents the execution of action a t is the waiting time of vehicle c in the lth lane.

[0047] The embodiment of the present invention provides an intersection adaptive signal control method based on cellular deduction multi-step decision-making. Compared with the prior art, the method has the following beneficial effects:

[0048] The present invention establishes a deep reinforcement learning adaptive signal continuous control model for intersections based on a cellular transmission model. The model uses cellular deduction to simulate the operation and transformation of traffic flow at an intersection to achieve state transfer at the intersection. At the same time, the state obtained by cellular deduction is extracted and then spliced ​​with the current phase plan of the road, the vehicle arrival rate and departure rate of the previous cycle as the state space input of the model to improve the accuracy of the intelligent agent's decision-making, and a multi-step decision-making mechanism is used to complete the pre-decision of all phases in the cycle, that is, the discrete action sequence of the state space input is integrated into a signal timing plan for a complete cycle and transmitted to the traffic light for continuous control. While retaining the flexible and changeable advantages of discrete actions, the model solves the problem that the deep reinforcement learning signal control model based on discrete states is difficult to achieve continuous action output. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of the overall flow of an intersection adaptive signal control method based on cellular deduction multi-step decision-making provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of an adaptive signal control model based on deep reinforcement learning for an intersection adaptive signal control method based on cellular deduction multi-step decision-making provided in an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of road discretization of an intersection adaptive signal control method based on cellular deduction multi-step decision-making provided by an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of the state processing process of an intersection adaptive signal control method based on cellular deduction multi-step decision-making provided by an embodiment of the present invention;

[0053] Figure 5 A schematic diagram of a phase-preserving cellular deduction process of an intersection adaptive signal control method based on cellular deduction multi-step decision-making provided by an embodiment of the present invention;

[0054] Figure 6 A schematic diagram of the action output of a CTMDRL model of an intersection adaptive signal control method based on cellular deduction multi-step decision-making provided by an embodiment of the present invention;

[0055] Figure 7 A schematic diagram of a DDQN-based intelligent signal control neural network structure diagram of an intersection adaptive signal control method based on cellular deduction multi-step decision-making provided in an embodiment of the present invention;

[0056] Figure 8 A schematic diagram of cumulative rewards for training scenario 1 of an intersection adaptive signal control method based on cellular deduction multi-step decision-making provided by an embodiment of the present invention;

[0057] Fig. 9 A schematic diagram of cumulative rewards for training scenario 2 of an intersection adaptive signal control method based on cellular deduction multi-step decision-making provided by an embodiment of the present invention;

[0058] Fig.10 A schematic diagram of cumulative rewards for training scenario 3 of an intersection adaptive signal control method based on cellular deduction multi-step decision-making provided by an embodiment of the present invention;

[0059] Fig.11 A schematic diagram of cumulative rewards for training scenario 4 of an intersection adaptive signal control method based on cellular deduction multi-step decision-making provided by an embodiment of the present invention;

[0060] Fig.12 A schematic diagram of average waiting time for training scenario 1 of an intersection adaptive signal control method based on cellular deduction multi-step decision-making provided by an embodiment of the present invention;

[0061] Fig.13 A schematic diagram of average waiting time for training scenario 2 of an intersection adaptive signal control method based on cellular deduction multi-step decision-making provided in an embodiment of the present invention;

[0062] Fig.14 A schematic diagram of average waiting time for training scenario 3 of an intersection adaptive signal control method based on cellular deduction multi-step decision-making provided by an embodiment of the present invention;

[0063] Fig.15 A schematic diagram of average waiting time for training scenario 4 of an intersection adaptive signal control method based on cellular deduction and multi-step decision-making provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0065] See also Figure 1The embodiment of the present invention provides an intersection adaptive signal control method based on cellular deduction multi-step decision. In order to solve the problem that the intelligent agent in the adaptive signal control model based on deep reinforcement learning can only rely on the current state to make discrete control of the traffic light, a multi-step decision mechanism is introduced to establish an intersection deep reinforcement learning signal continuous control model; cellular deduction is used to simulate the operation and transformation of the intersection traffic flow to realize the state transfer of the intersection; the state obtained by cellular deduction is extracted after feature extraction and spliced ​​with the current phase plan, the vehicle arrival rate and departure rate of the previous cycle as the state input of the model to improve the accuracy of the intelligent agent's decision; the multi-step decision mechanism is used to complete the pre-decision of all phases in the cycle, and then integrated into a complete four-phase signal timing plan and transmitted to the traffic light to realize the adaptive signal continuous control.

[0066] Specifically:

[0067] In order to realize deep reinforcement learning adaptive signal continuous control, the present invention introduces CTM into the model to realize intersection state transfer, and establishes a CTMDRL adaptive signal continuous control model based on the multi-step discrete decision-making of the intelligent agent, which solves the problem that deep reinforcement learning adaptive control traffic lights do not have a countdown function.

[0068] 1. State space.

[0069] The present invention uses road discretization to obtain vehicle information at four entrances as the state input of the model. Based on tools such as vehicle-mounted networks, the position and speed of vehicles on the current road are obtained from the intersection snapshot. The division length is defined as 5m, and it is stipulated that each discrete grid contains at most one vehicle. Figure 3 As shown in the figure; the model divides the environment of 1000m×1000m with the intersection as the center into 200×200 square grids of the same size at 5m intervals; when dividing the road, vehicles crossing the boundaries of adjacent grids are divided according to the center position of the vehicle; the state information contained in each grid is defined as a binary vector of the vehicle <position, speed>. The position is a binary value, indicating whether there is a vehicle in the grid; if there is a vehicle in the grid, the value in the grid is (1, speed); otherwise, it is (0,0); the speed represents the current speed of the vehicle in the grid.

[0070] The model state processing process is as follows Figure 4 As shown in the figure; in order to improve the decision-making efficiency of the intelligent agent, the information in the 60×60 grid centered on the intersection is used for decision-making. After obtaining the vehicle position matrix and speed matrix at the intersection, they are input into the 3-layer convolutional network, and the extracted feature information is spliced ​​with the current phase scheme, the lane-level vehicle arrival rate and departure rate of the previous cycle as the state input of the subsequent neural network.

[0071] Although the state space of discrete grids has certain advantages in adaptive control of traffic signals, it can only reflect instantaneous static information and cannot make decisions on continuous actions. To address this problem, the present invention then considers introducing CTM to simulate vehicle movement to achieve intersection state transfer. The intelligent agent makes multi-step discrete decisions based on the transferred state, and integrates the discrete actions and then transmits them to the traffic light to achieve the conversion of discrete actions to continuous actions.

[0072] The cellular environment is initialized with a 200×200 discrete grid centered at the intersection at the beginning of each cycle. The size of the unit cell is consistent with the discrete grid, and each cell can only accommodate one vehicle, that is, the state information carried in the cell is completely consistent with the data structure of the discrete grid. The step size of the cellular deduction model is set to 1s, and the full velocity difference (FVD) following model is used to describe the following trajectory of the artificially driven vehicle in the cell. The acceleration of the following vehicle is expressed as:

[0073] a cc (t) = α[V(Δd i (t))-v i (t)]+β[v i+1 (t)-v i (t)]

[0074] V(Δd i (t))=V1+V2tanh[c1(Δx i (t)-l)-c2]

[0075] Among them: a cc (t) represents the acceleration of the vehicle at time t; V(d i (t)) represents the optimized speed function, which is the expected speed calculated based on the distance between the front and rear vehicles; l is the vehicle length, which is 5; V1, V2, c1, and c2 are the coefficients of the optimized speed function, whose values ​​are V1 = 6.75 m·s -1 、V2=7.19m·s -1 、c1=0.13m -1 、c2=1.57m -1 ; α=0.41s -1 , is the driver sensitivity coefficient; β=0.5s -1 , is the speed difference reaction coefficient; when there is no leading vehicle and the lane is red, the stop line can be regarded as the leading vehicle with a speed of 0, and the vehicle acceleration is determined based on the following model; when there is no leading vehicle on the lane and the lane is green, its acceleration a is specified. cc (t)′=max(v max -v i (t), 1m·s -2 , where vi (t) represents the speed of vehicle i in the cell at time t. To ensure that the vehicle speed is within a reasonable range, v is specified. i (t)∈[0km / h,50km / h], then the speed of the vehicle at the next moment should be:

[0076] v i (t+1)=v i (t)+a cc (t)·Δt

[0077] Where: Δt is the time interval, from which we can get the total displacement s of vehicle i at time t+1 during the cellular deduction process i (t+1) is:

[0078]

[0079] Based on the above formula, the vehicle displacement s i (t+1) divided by 5 can be used to calculate the cell position of vehicle i after it performs a discrete action in the environment at time t; i Under the premise that (t)≤1000m, if random<corresponding vehicle arrival rate, a vehicle is generated at the starting point of the lane in the cellular environment, and the initial speed of the vehicle is the maximum speed; when the vehicle displacement is greater than 1000m, the vehicle leaves the cellular environment.

[0080] For the CTMTSC model proposed in the present invention, when the agent action output a=0, the traffic light maintains the current phase and executes for 3s, and the cell performs 3-step deduction; when the agent action output a=1, the traffic light switches to the next phase and executes for 3s, and outputs a 3s yellow light as a transition in the middle, and the cell performs a total of 6 steps of deduction; for example, when the agent action output is a=0, if the current phase is east-west straight release, the cell deduction process is as follows Figure 5 As shown; Figure 5 Only the 12×12 cell deduction process around the intersection is shown.

[0081] After the model generates a complete cycle of signal timing plans, CTM will stop the deduction, and the intelligent agent will transmit the complete cycle of signal timing plans to the intersection signal lights for signal control, and then continue to initialize the cellular environment based on the actual road network after the signal plan is implemented; there is no need to ensure strict consistency in modeling to achieve intersection state transfer, and the training of deep reinforcement learning adaptive signal control models also does not need to ensure strict consistency between the environment and the actual intersection environment; the negative impact caused by environmental differences can be compensated by the inherent advantages of deep reinforcement learning algorithms: an acceptable deep reinforcement learning adaptive signal control model is trained in a simulation environment and applied to intersections. The model will autonomously collect actual data from the intersection when making decisions to update itself, continuously improve its decision-making ability, and finally obtain a model that can be applied in practice and has excellent control effects; therefore, research on optimizing the training effect of deep reinforcement learning signal control models should focus on how to reflect more spatiotemporal dynamic information of traffic flow in the state space.

[0082] 2. Action space.

[0083] The present invention proposes to introduce a multi-step decision mechanism into an adaptive signal control model, and output a complete four-phase signal timing scheme based on the pre-decision of multi-step discrete actions, thereby retaining the flexible and changeable advantages of discrete actions while realizing adaptive continuous control of traffic signals.

[0084] The agent makes a preliminary action decision based on the state information obtained by cellular deduction and sets the action set to (0,1); Figure 6 As shown in the figure, when the agent selects action a=0, the model extends the current phase by 3s and stores it in the "pre-decision action space"; when the agent selects action a=1, the model will switch to the next phase according to the pre-set phase scheme sequence and execute it for 3s, and also store it in the "pre-decision action space". The agent performs multiple steps of pre-decision until the "pre-decision action space" contains four actions with a=1, that is, after a complete four-phase signal timing scheme is generated, the cellular deduction of the next signal cycle ends, and the agent stops this round of decision-making; the pre-decision action is only implemented in the cellular environment, and the traffic light outputs a yellow light for 3s between adjacent green light phases as a buffer. During the execution time of the last yellow light, the signal timing scheme of the complete cycle is transmitted to the signal light for intersection signal control, and the cellular state is initialized with the environment at the end of the last yellow light at the intersection, and the next round of action output is performed.

[0085] 3. Reward function.

[0086] The ultimate goal of deep reinforcement learning adaptive signal control is to improve the efficiency of intersection traffic and reduce vehicle waiting time. Considering the intuitiveness of the reward function, the present invention will reward r t Defined as the discrete action pre-decision process, executing action a tThe negative value of the sum of the accumulated waiting time of all lane vehicles in the cell grid, the reward function is defined as follows:

[0087]

[0088] Where: r t Represents the execution of action a t The reward given to the agent by the environment at that time; l represents the entrance lane; L represents the total number of entrance lanes at the intersection; c represents vehicles; C represents the total number of vehicles on the corresponding entrance lane; w l，c Represents the execution of action a t The waiting time of vehicle c in lane l. In order to guide the agent to converge in the direction of maximum cumulative reward, the reward feedback given by the environment to the agent needs to be negative based on the waiting time.

[0089] 4. Agent update strategy.

[0090] This paper adopts the more typical DDQN and D3QN algorithms in the value-based algorithm to update the intelligent agent. The traditional deep reinforcement learning based on the DQN algorithm solves the problem of too large state dimension by approximating the action value function. Its value function is expressed as:

[0091] Q θ (s, a) = E[G t |(S=s t , A=a t )]

[0092] Where: Q is the action value function; θ is the parameter of the action value function; G t Return refers to the state s at time t. t From the beginning to the end, the sum of the decay of all rewards is:

[0093]

[0094] Where: R t is the reward obtained at time t; γ is the discount coefficient after considering the long-term reward decay, γ∈(0,1).

[0095] According to the Bellman equation, the Q-value function under different states can be obtained by recursion, so the optimal action value function can be obtained:

[0096]

[0097] The agent selects the maximum The corresponding actions can achieve the goal of maximizing returns, that is, the optimal strategy:

[0098]

[0099] At the same time, the parameters of the action value function are updated by maximizing the mean square error loss function:

[0100]

[0101] DDQN decouples the selection of target Q-value actions from the calculation of target Q-values, effectively alleviating the overestimation problem of DQN; its action value function is defined as follows:

[0102]

[0103] Where: γ is the discount factor; Q′ θ′ is the target Q value; θ′ is the target network parameter; argmax a∈A Q θ (s t+1 , a) from the current network; DDQN introduces a target network into the network structure; except for the weight parameters, the target network is exactly the same as the training network; the training network is used to estimate the current Q θ (s t , a t ), the target network is used to estimate It can alleviate the deviation caused by bootstrapping; the Duling DQN algorithm decomposes the Q-value network into value function and advantage function, improving the accuracy and stability of the algorithm; D3QN combines the advantages of DDQN and Dueling DQN algorithms to further improve the learning efficiency and stability of the algorithm. It is very effective in dealing with discrete action space problems and is gradually widely used in traffic signal control.

[0104] 5. Neural network structure.

[0105] The model first obtains the initial state of the cellular deduction based on the discretized intersection, then obtains the matrix containing the vehicle position and vehicle speed based on the cellular deduction and inputs it into the three-layer convolutional network for processing. The obtained feature vector is concatenated with the current execution phase and the lane-level vehicle arrival rate and departure rate to obtain a vector with a length of 21660. See the state vector acquisition for details. Figure 4 , where the vehicle information obtained based on the cell grid is a 60×60×2 matrix, and the convolutional network output size is 13×13×128; after the state vector is input into the model, it is processed by two layers of fully connected networks to output actions, and each fully connected layer contains 128 neurons. The action output only contains two actions (0,1), so the action space size is 2; in order to make the model have excellent signal control performance while taking into account the algorithm complexity, the present invention adopts a value-based algorithm in the model, and the neural network structure is as follows Figure 7 shown.

[0106] The experimental part of the present invention selects the more classic DDQN and D3QN in the value-based algorithm as the update algorithm of the intelligent agent, respectively, to verify the universality of the modeling ideas and cellular deduction multi-step decision-making mechanism of the present invention; D3QN and DDQN algorithms are both improvements of the DQN algorithm. Compared with DDQN, the only difference of D3QN is that the competition architecture is introduced in the algorithm. One of the two fully connected competition networks outputs the state value and the other outputs the action advantage. The rest is exactly the same as the DDQN network. Therefore, the present invention no longer displays the structure diagram of the intelligent signal control neural network based on D3QN; for the convenience of description, the cellular deduction signal control model based on the DDQN algorithm is referred to as CTMDDQN, and the cellular deduction signal control model based on the D3QN algorithm is referred to as CTMD3QN; the specific iteration process of the algorithm is as follows:

[0107] Step 1: Initialize model parameters, set the experience pool capacity to D, and initialize the target network weights.

[0108] Step 2: Preheat the model and randomly run several signal cycles to make the road generate traffic flow that meets the training requirements.

[0109] Step 3: Create an initial cellular environment based on intersection vehicle information.

[0110] Step 4: Get state input s t , based on the Q value corresponding to all actions output by the current Q network, select action a according to the greedy strategy t =0 / 1 and store it in the pre-decision space A.

[0111] Step 5: Execute action a in the cellular environment t (i.e., perform cellular deduction) and get reward r t and the new state s t+1 , will (s t ,a t ,s t+1 ,r t ) is stored in the experience pool as a sample.

[0112] Step 6: When the number of samples in the experience pool reaches the sample capacity requirement, randomly collect a batch of samples from the replay experience pool to update the agent.

[0113] Step 7: Determine whether A generates a complete cycle signal timing plan. If so, integrate the actions in A to generate a complete cycle signal timing plan and transmit it to the intersection signal light for continuous signal control. After the intersection signal light executes a complete cycle, use the current actual environment to continue to initialize the cellular environment. If not satisfied, go to step 4.

[0114] Step 8: Determine whether the number of training rounds reaches the preset number of training rounds. If so, end the agent training; if not, jump to step 2.

[0115] Experimental analysis:

[0116] The present invention uses the traffic data in the off-peak phase for simulation training and experiments. The specific traffic flow data is shown in Table 1.

[0117] Table 1 Traffic flow data

[0118] 8:00-9:00 9:00-10:00 10:00-11:00 11:00-12:00 Go straight east-west 497 422 403 446 Turn left southeast 231 246 212 227 Northeast turn right 372 332 322 300 Go straight from west to east 444 403 408 418 Turn left northwest 155 97 124 154 Turn right southwest 317 284 261 301 Go straight north and south 368 302 278 319 Turn left on Southwest 198 176 172 203 Turn right south east 169 164 162 179 Go straight north-south 462 405 379 394 Turn left from the north east 182 161 185 203 Turn right northwest 189 182 190 206 total 3584 3174 3096 3350

[0119] 1. Model training.

[0120] The present invention adopts the method of dynamically loading model parameters to divide a long time scene into several short time segments and train the intelligent agent based on the time segments. In practical applications, the model parameters can be dynamically loaded according to the short segment scenes to achieve optimal control of the intelligent agent; the training and experimental scene settings of the model are shown in Table 1. The training is divided into 4 scenes at intervals of 60 minutes to train the intelligent agent respectively, 8:00-9:00 is test scene 1, 9:00-10:00 is test scene 2, 10:00-11:00 is test scene 3, and 11:00-12:00 is test scene 4; the arrival of traffic follows Poisson distribution, and the training in each scene is 200 rounds in total; the optimal parameters of the CTMDRL control model are determined through multiple experiments, and the model hyperparameter settings are shown in Table 2.

[0121] Table 2 Model hyperparameter settings

[0122]

[0123]

[0124] In order to verify the convergence effect and adaptability of the model, the present invention selects the deep reinforcement learning adaptive signal control model based on discrete grid states (DRLTSCbased on discrete girds, DGDRL) and the deep reinforcement learning adaptive signal control model based on feature vector states (DRLTSC based on featurevector, FVDRL) and combines them with DDQN and D3QN algorithms respectively as the comparison algorithms of this model. To simplify the expression, the two models based on discrete grid states are referred to as DGD3QN and DGDDQN, and the two models based on feature vector states are referred to as FVD3QN and FVDDQN in the following text; to ensure consistency, the hyperparameters of the comparison model are selected to be consistent with the CTMDRL model.

[0125] The cumulative waiting time of all vehicles in each round of the 6 models under 4 scenarios changes as follows: Figure 8-11 As shown. Since the training results will take negative cumulative returns, Figure 8-11 The cumulative return is positively correlated with the ideal degree of training effect; Figure 8-11 It can be seen that all six models produced large fluctuations in the initial stage of training. The CTMD3QN and CTMDDQN models reached convergence after about 20 rounds of training in the four scenarios, and achieved relatively fast and stable convergence effects among the six algorithm models; except for the DGDDQN model, which reached convergence after about 60 rounds of training in scenario 2, DGD3QN and DGDDQN also reached convergence after about 20 rounds of training in the other scenarios; the FVD3QN and FVDDQN models converged slower and had poorer training effects than the other four comparison models; compared with the other four comparison models, the CTMD3QN and CTMDDQN models achieved continuous control of intersection signals by using a multi-step decision-making mechanism while ensuring rapid convergence of the model, solving the problem that deep reinforcement learning adaptive signal control does not have a countdown function.

[0126] Figure 8-11 The convergence effect of the algorithm was evaluated only from the perspective of cumulative return. Since factors such as vehicle flow in reality will also affect the cumulative return, it is impossible to make an objective evaluation of the signal control effect only from the perspective of cumulative waiting time. To further verify the convergence of the model, the average waiting time of all vehicles during the training of each model under the four scenarios is shown in the following figure. Figure 12-15 As shown; since the average waiting time of each vehicle is the ratio of the cumulative return to the traffic flow and Figure 8-11 The cumulative return is negative. Figure 8-11 and Figure 12-15 showed the opposite trend; Figure 12-15 It can be seen that the CTMD3QN and CTMDDQN models converge smoothly under different traffic scenarios, and the average waiting time of vehicles is comparable to that of the DGD3QN and DGDDQN models, and is better than that of the FVD3QN and FVDDQN models.

[0127] 2. Model testing.

[0128] The experiment selected the average waiting time and fuel consumption of vehicles as the performance evaluation indicators of the experimental results; the traffic environment adopted the measured traffic flow shown in Table 1, and the experiment was divided into four scenarios according to the hour, and the arrival of the traffic flow obeyed the Poisson distribution; the simulation rounds of the model in each scenario were set to 50 times, and the simulation test time was 3600s; in order to verify the applicability of the CTMDRL model, it was combined with the DDQN and D3QN algorithms with better control effects in the value algorithm, so that the experimental models can be divided into four groups: the first group is the CMTDRL model combined with the two algorithms; the second group is the DGDRL model combined with the two algorithms, and the model hyperparameters are consistent with the model in this paper; the third group is the FVDRL model combined with the two algorithms, and the model hyperparameters are consistent with the model in this paper; the fourth group is the Webster signal control method, which is a common deep reinforcement learning model at present, and the green light timing of each phase is calculated by the Webster optimization algorithm.

[0129] In order to comprehensively evaluate the control benefits of each model in different scenarios, the specific data of the overall performance of the seven traffic signal control models in four experimental scenarios are shown in Table 3.

[0130] Table 3 Performance of the model in four experimental scenarios

[0131]

[0132] As shown in Table 3, the FVD3QN and FVDDQN models are based on deep reinforcement learning algorithms, which can dynamically adjust the signal according to the arrival of vehicles at the intersection. Compared with the Webster signal control method, they have achieved better control effects in terms of average waiting time and average fuel consumption indicators. Compared with the FVDRL control model, the DGDRL control model uses a discrete state space, which makes the agent training more efficient, the learned rules more applicable, and the control effect better. The CTMDRL control model uses CTM to transform the intersection state and realizes multi-step pre-decision of discrete actions. Although the cellular environment is an approximate expression of the actual environment, the intersection of discrete cells and discrete grids has natural consistency, which can include the state information required for the agent decision-making without affecting the agent. The model outputs the action, and at the same time introduces the vehicle arrival rate, departure rate and current phase plan of the previous cycle, so that the intelligent agent can dynamically adjust the signal according to the latest traffic status, avoid the oversaturation of the road in time, and enhance the reliability and stability of the pre-decision action. Finally, the control effect of the CTMDRL control model is equivalent to that of the DGDRL model, which is better than the Webster signal control model and the FVDRL model. From the specific data analysis, the FVDRL model based on the DDQN algorithm and the D3QN algorithm are compared with the CTMDRL model based on the same algorithm in 4 scenarios. The Webster signal control model is compared with the CTMDRL model based on the two algorithms respectively. Finally, the average of the comparison results of different algorithm models is taken according to the scenario division. It was found that under scenarios 1-4, the CTMDRL control model reduced the average waiting time and fuel consumption by approximately 11.17%, 22.34%, 19.02%, 9.30% and 5.67%, 12.11%, 9.73%, 4.67% respectively compared with the Webster signal control model; compared with the FVDRL model, the average waiting time and fuel consumption were reduced by approximately 9.80%, 16.69%, 12.32%, 16.85% and 4.56%, 8.70%, 5.63%, 9.26% respectively.

[0133] The present invention proposes to apply a multi-step decision-making mechanism to deep reinforcement learning adaptive signal control, and establishes an adaptive signal continuous control model that can be applied to countdown traffic lights; the main idea of ​​modeling is to realize the adaptive continuous control of intersection signals by overlapping the cellular virtual environment with the actual road network environment: CTM is used to realize the state transfer of the intersection, and the intelligent agent makes multi-step discrete action pre-decision based on the cellular environment until the discrete actions are spliced ​​into a continuous action, that is, a complete cycle of signal timing plan, which is then transmitted to the actual intersection for phase control; the model obtains intersection vehicle information in the form of discrete grids, and the intelligent agent makes multi-step discrete decisions based on the vehicle information deduced in the cellular environment, the current phase plan, and the vector spliced ​​by the lane-level vehicle arrival rate and departure rate of the previous cycle in the actual environment. The action output is a complete cycle of signal timing plan, and the reward is the accumulated waiting time of all vehicles.

[0134] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An adaptive signal control method for intersections based on cellular deduction and multi-step decision making, characterized in that: The following steps are involved: Discretize the road according to the current road image to obtain discrete grids, and obtain the position and speed of the vehicle on the current road corresponding to each discrete grid; The cellular transmission model CTM and the deep reinforcement learning adaptive signal discrete control model DRLTSC are constructed respectively, and the cellular transmission model CTM and the deep reinforcement learning adaptive signal discrete control model DRLTSC are cascaded to construct the intersection deep reinforcement learning adaptive signal continuous control model CTMDRL; and the intersection deep reinforcement learning adaptive signal continuous control model CTMDRL is trained using the DDQN algorithm and the D3QN algorithm to obtain the trained intersection deep reinforcement learning adaptive signal continuous control model CTMDRL; The cellular transmission model CTM in the trained intersection deep reinforcement learning adaptive signal continuous control model CTMDRL is used to set the discretized grid of the road as an independent cell, and the driving process of the vehicle on the road is set as a cellular transmission process. The cellular environment is initialized according to the position and speed of the vehicle, and the operation and transformation of the intersection traffic flow are simulated by cellular deduction based on the full speed difference FVD following model to obtain a state matrix containing the intersection vehicle position and vehicle speed; the state matrix is ​​subjected to feature extraction through the output layer of the cellular transmission model CTM to obtain the eigenvector of the vehicle information state matrix; Using the trained deep reinforcement learning adaptive signal discrete control model DRLTSC in the intersection deep reinforcement learning adaptive signal continuous control model CTMDRL, the eigenvector of the vehicle information state matrix, the current execution phase of the intersection, and the lane-level vehicle arrival rate and departure rate in the previous cycle are concatenated to form a state vector; the intelligent agent in the deep reinforcement learning adaptive signal discrete control model DRLTSC is used to pre-decide on multi-step discrete actions for the state vector to obtain discrete actions, and the discrete actions are integrated into a signal timing plan for a complete cycle; the signal timing plan is transmitted to the traffic light to realize continuous control of the traffic signal.

2. The method for adaptive intersection signal control based on cellular deduction multi-step decision-making according to claim 1 is characterized in that: The obtaining of a discrete grid comprises: The vehicle position and speed at the lane level are obtained from the intersection snapshot according to the vehicle network method, the partition length is set to 5m, and it is stipulated that each discrete grid contains at most one vehicle; The 1000m×1000m environment centered at the intersection is divided into 200×200 square grids of the same size at 5m intervals. The state information contained in each grid is a binary vector of the vehicle's position and speed.

3. The method for adaptive intersection signal control based on cellular deduction multi-step decision-making according to claim 1 is characterized in that: The initialization of the cellular environment includes: The cellular environment is initialized with a 200×200 discrete grid centered at the intersection at the beginning of each cycle. The size of the unit cell is consistent with the discrete grid, and each cell can only accommodate one vehicle, that is, the state information carried in the cell is consistent with the data structure of the discrete grid; The step length of the cellular deduction model is set to 1s, and the full velocity difference FVD following model is used to describe the following trajectory of the artificially driven vehicle in the cell. The acceleration of the following vehicle is expressed as: a cc (t)=α[V(Δd i (t))-v i (t)]+β[v i+1 (t)-v i (t)] V(Δd i (t))=V1+V2tanh[c1(Δx i (t)-l)-c2] Among them: a cc (t) represents the acceleration of the vehicle at time t; V(Δd i (t)) represents the optimized speed function; l is the vehicle length; V1, V2, c1, c2 are the coefficients of the optimized speed function; α is the driver sensitivity coefficient; β is the speed difference reaction coefficient; v i (t) represents the speed of vehicle i in the cell at time t; v i+1 (t) represents the speed of the i+1th vehicle in the cell at time t; Δx i (t) represents the distance traveled by vehicle i in the cell at time t when the speed is optimized; The speed of the vehicle at the next moment is: v i (t+1)=v i (t)+a cc (t)·Δt Where: Δt is the time interval; it can be obtained that during the cellular deduction process, the total displacement s of vehicle i at time t+1 i (t+1) is: Where: s t represents the displacement of the vehicle at time t; according to the above formula, the vehicle displacement s i (t+1) divided by 5 is used to calculate the cell position of vehicle i after it performs a discrete action in the environment at time t.

4. The method for adaptive intersection signal control based on cellular deduction multi-step decision-making according to claim 1 is characterized in that: The feature extraction of the state matrix through the output layer of the cellular transport model CTM includes: The state matrix obtained in the cellular deduction contains lane-level vehicle information and has a size of 60×60×2. After feature extraction through the output layer of the cellular transfer model CTM, the state matrix has an output size of 13×13×128.

5. The method for adaptive intersection signal control based on cellular deduction multi-step decision-making according to claim 1, characterized in that: The method of using the DDQN algorithm and the D3QN algorithm to train the intersection deep reinforcement learning adaptive signal continuous control model CTMDRL includes: The optimal action value function based on DQN is: Where: R t is the reward obtained at time t; γ is the discount factor after considering the long-term reward decay; Q is the action value function; θ is the parameter of the action value function; s t represents the displacement of the vehicle at time t; s t+1 represents the displacement of the vehicle at time t+1; a t represents the acceleration of the vehicle at time t; a t+1 represents the acceleration of the vehicle at time t+1; The agent selects the maximum The corresponding action achieves reward G t The goal of maximization is set as the optimal strategy: At the same time, the parameters of the action value are updated by maximizing the mean square error loss function. The update method is: DDQN adopts the form of a dual Q network and introduces a target network into the network structure. Except for the weight parameters, the target network is exactly the same as the training network. The training network is used to estimate the current Q θ (s t , a t ), the target network is used to estimate The deviation caused by bootstrapping can be alleviated, and the optimal action value function is: Where: γ is the discount factor; Q′ θ″ is the target Q value; θ′ is the target network parameter; argmax a∈A Q θ (s t+1 ,a) from the current network; DuelingDQN converts the optimal action value Q * Decompose into optimal state value V * (s t ) and optimal advantage D * (s t , a t ), its optimal action value function is expressed as: Among them: When the action takes the action corresponding to the optimal state value, The action-value function achieves the optimal value; The D3QN algorithm combines the advantages of the DDQN and Dueling DQN algorithms to further improve the learning efficiency and stability of the algorithm. It is very effective in dealing with discrete action space problems. DDQN and D3QN algorithms are used to update the intelligent agent.

6. The method for adaptive intersection signal control based on cellular deduction multi-step decision-making according to claim 1, characterized in that: The obtaining of discrete actions comprises: The agent makes a pre-action decision based on the state information obtained from the cellular deduction and sets the action set to (0,1); when the agent selects action a=0, the model extends the current phase by 3s and stores it in the pre-decision action space; when the agent selects action a=1, the model switches to the yellow light phase for 3s and then switches to the next phase in the pre-set phase sequence and executes for 3s, which is also stored in the pre-decision action space. The agent performs multi-step pre-decision until the pre-decision action space contains four actions with a=1; this round of decision-making stops and outputs discrete actions.

7. The method for adaptive intersection signal control based on cellular deduction multi-step decision-making according to claim 1, characterized in that: The reward function used in the training of the agent is: Where: r t Represents the execution of action a t The reward given to the agent by the environment at that time; l represents the entrance lane; L represents the total number of entrance lanes at the intersection; c represents vehicles; C represents the total number of vehicles on the corresponding entrance lane; w l，c Represents the execution of action a t is the waiting time of vehicle c in the lth lane.

Images