A single-point traffic signal control method based on intersection holographic data

Abstract

The present invention relates to the field of traffic control technology, and specifically to a single-point traffic signal control method based on intersection holographic data. A deep reinforcement learning model including multiple intelligent agents is constructed, and then a MARDDPG algorithm is used for neural network training. Finally, the trained neural network is used for signal control. Different from traditional traffic detection technology, the present invention makes full use of intersection holographic data to fully express traffic demand. At the same time, deep reinforcement learning of the MARDDPG algorithm is adopted. Each intelligent agent controls each phase individually, and collaborative optimization is performed between single intelligent agents, which makes it easier to achieve optimal intersection control efficiency. Furthermore, since the intelligent agent controls each phase individually, the state space of each intelligent agent is smaller, the results converge faster, and the traffic signal can be effectively and more accurately controlled.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic control, and in particular to a single-point traffic signal control method based on intersection holographic data. Background Art

[0002] Currently, the most widely used inductive traffic signal control systems in China include the UK's SCOOT (Split Cycle Offset Optimization Technique) system and Australia's SCATS (Sydney Coordinated Area Traffic System). With the accelerating pace of urbanization and the exponential growth in the number of motor vehicles, traditional inductive traffic signal control methods are unable to effectively manage and cope with the dramatic real-time fluctuations in traffic flow. Traditional inductive control methods adjust green light duration based on headway, occupancy, queue length, and congestion levels.

[0003] The shortcomings are mainly reflected in the following aspects:

[0004] 1. The detection data of traditional inductive traffic signal control systems cannot fully and effectively represent the traffic demand at intersections. The coil detection error is large and the reliability is low. It is difficult to effectively utilize the algorithm and control method.

[0005] 2. Current deep reinforcement learning traffic control methods generally lack collaborative optimization. Traffic signal control by a single agent can only achieve system optimization, and the discrete decision space of the combinatorial optimization problem is too complex.

[0006] 3. The dimension of the state space of a single agent is too high, the calculation time is long, and it cannot be used in practice. Summary of the Invention

[0007] The purpose of the present invention is to provide a single-point traffic signal control method based on intersection holographic data, which utilizes intersection vehicle holographic detection means to obtain vehicle position and speed information in real time, and control traffic signals more effectively and accurately.

[0008] To achieve the above object, the present invention provides a single-point traffic signal control method based on intersection holographic data, comprising the following steps:

[0009] Collect initial holographic traffic data;

[0010] Holographic traffic data processing;

[0011] Build deep reinforcement learning models with multiple agents;

[0012] Use MARDDPG algorithm for neural network training;

[0013] Use trained neural networks for signal control.

[0014] Among them, the holographic traffic data includes target vehicle operation data, lane-level traffic data, intersection design status and traffic signal control status. The target vehicle operation data includes the target vehicle ID with timestamp, vehicle type, vehicle longitudinal speed, vehicle lane number and vehicle distance from the stop line. The lane-level traffic data includes the queue length of the target lane, the total waiting time of vehicles, the average delay and the number of vehicles passing the stop line. The intersection design status includes the number of lanes at each entrance of the intersection and the lane function distribution. The traffic signal control status includes the current phase sequence of the intersection and the distribution of the duration of each phase.

[0015] The holographic traffic data processing process includes the following steps:

[0016] Delete redundant data;

[0017] Delete abnormal trajectory data;

[0018] Linear function interpolation method is used to complete the noisy trajectory data.

[0019] Among them, in the process of building a deep reinforcement learning model containing multiple agents, the traffic signal control agent Agent uses the MARDDPG algorithm for deep reinforcement learning, and defines the state space S, action space A and reward value R respectively.

[0020] The process of using the MARDDPG algorithm to train a neural network includes the following steps:

[0021] Step 1: Initialize the parameterized action selection strategy of the actor network and the critic network in is the historical memory data of the i-th agent at a certain time step t in the participant network, and the historical memory data of the critic network It is also defined as the historical memory data of the i-th agent in the network at a certain time step t; construct and parameterize the value function in the critic network The input to the value function includes the historical state h and the action a selected by the participant network; respectively, initialize the weights θ' of all target networks i and Initialize the replay buffer D, reset the time step to zero, and read the initial state s of the participant network t,i ;

[0022] Step 2: Every 5 seconds, a single agent selects a strategy through action selection in Explore the noise for the current time step; the actor network selects action a from the action set Ai t,iAnd execute, receive reward value r after execution t,i With the new state s t+1,i , generate new historical data

[0023] Step 3: Convert the 1,i , a 1,i , r 1,i , s 2,i , a 2,i , r 2,i , ....} is stored in the replay buffer D, with the number of samples being no less than 20,000; agent i samples M historical training steps stored in the replay buffer D for training the participant and critic networks;

[0024] Step 4: After the agent selects M historical training step data, it uses a minibatch (represented by m) in M ​​to make the critic network pass the value function constructed in step 1 Estimate the Q value and update the parameters in the critic target network by minimizing the average loss function Similarly, the participant network calculates the policy gradient through the loss function to update the parameters θ in the participant target network i ;

[0025] For update The loss function is as follows:

[0026]

[0027] represents the average Q value of the actions selected by M minibatches of all agent-critic networks; is the total reward value; γ is the reduction coefficient; represents the sum of the Q values ​​of all agents in the next time step;

[0028] To update θ i The objective function is as follows:

[0029]

[0030] In order to maximize the future expected reward to update the participant network, define J(θ i ) to find the direction that maximizes the cumulative reward;

[0031] Step 5: The agent uses a “soft update” method to update the parameters of the target network and θ' iFirst, define a rate τ (0 < τ < 1) for updating the target network based on the main network; update the target network using a convex combination of the current network parameters and the target network parameters, as follows:

[0032]

[0033] θ' i =τθ i +(1-τ)θ' i

[0034] Repeat steps 2 to 5. When , the optimal state in the traversable state space has been reached and the agent completes training.

[0035] The process of using the trained neural network for signal control is to collect real-time holographic data according to the requirements of the intelligent agent construction, input signals to control the intelligent agent, change the action by the duration of each phase signal output by the intelligent agent, and generate the phase-time matrix G':

[0036] G'=[G1' G2' G3' … Gn'].

[0037] The present invention provides a single-point traffic signal control method based on intersection holographic data, constructs a deep reinforcement learning model including multiple intelligent agents, then uses the MARDDPG algorithm to train the neural network, and finally uses the trained neural network to perform signal control. Different from traditional traffic detection technology, the present invention makes full use of intersection holographic data to fully express traffic needs, and at the same time adopts deep reinforcement learning of the MARDDPG algorithm. Each intelligent agent controls each phase individually, and collaborative optimization is performed between single intelligent agents, which makes it easier to achieve optimal intersection control efficiency. Furthermore, since the intelligent agent controls each phase individually, the state space of each intelligent agent is smaller, the results converge faster, and the traffic signal can be effectively and more accurately controlled. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 It is a flow chart of a single-point traffic signal control method based on intersection holographic data of the present invention.

[0040] Figure 2 It is a schematic diagram of the convolutional network structure of the deep reinforcement learning model of the present invention.

[0041] Figure 3 Schematic diagram comparing the vehicle positions at the intersection of the present invention with the position matrix and the speed matrix. DETAILED DESCRIPTION

[0042] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0043] See also Figure 1 The present invention provides a mobile terminal identification method for crop diseases and insect pests based on deep learning, comprising the following steps:

[0044] S1: Collect initial holographic traffic data;

[0045] S2: holographic traffic data processing;

[0046] S3: Build a deep reinforcement learning model with multiple agents;

[0047] S4: Use MARDDPG algorithm to train neural network;

[0048] S5: Use the trained neural network for signal control.

[0049] Specifically, in step S1, the holographic traffic data includes target vehicle operation data, lane-level traffic data, intersection design status, and traffic signal control status.

[0050] Among them, the target vehicle operation data includes: target vehicle identification number C id 、Vehicle Type C s , vehicle longitudinal speed V p 、Lane number of the vehicle i , Distance Y between the vehicle and the stop line i .

[0051] Lane-level traffic data includes: target lane queue length Q i , total vehicle waiting time W i , average delay D i , Number of vehicles passing the stop line N c .

[0052] The current status of intersection design includes: the number of lanes at each entrance to the intersection and the functional distribution of the lanes.

[0053] The current status of traffic signal control includes: the current phase sequence of the intersection and the duration distribution of each phase.

[0054] In step S2, the purpose of holographic traffic data processing is to improve data accuracy and make traffic control more efficient, which specifically includes the following steps:

[0055] Step 2.1: Delete redundant data. Due to transmission interference and the recognition rate of the device, recognition anomalies may occur: Normal output data should be a character string. If the number of data bits does not match or the data is garbled, all traffic data of the relevant target vehicle must be deleted.

[0056] Step 2.2: Delete abnormal trajectory data. If the lane steering does not correspond to the data collected by the equipment, it is an abnormal trajectory and all traffic data of the relevant target vehicle must be deleted.

[0057] Step 2.3: Completing the noise trajectory data. Since the equipment collects traffic data at a fixed time step, a linear function interpolation method can be used. First, the continuous position points of the target vehicle whose trajectory offset is greater than the threshold are formed into a trajectory. The formed trajectory is fitted into a linear function, and then this offset trajectory point is inserted into the median of the previous and next trajectory points in the fitting curve.

[0058] In step S3, a deep reinforcement learning model including multiple agents is constructed, which specifically includes the following steps:

[0059] Assume that the intersection is a classic n-phase signal-controlled intersection with m lanes available in each phase. Without adjusting the phase sequence, a signal control agent is deployed in each phase to adjust the green light duration of its own phase to control traffic flow. After each adjustment, the agent can obtain status and feedback from the environment. All signal control agents can collaborate and optimize to reduce overall traffic congestion at the intersection.

[0060] Step 3.1. The traffic signal control agent uses a multi-agent deep reinforcement learning method. Based on the high precision of holographic traffic data and the temporal continuity of traffic flow, the multi-agent recursive deep deterministic policy gradient algorithm (MARDDPG) is selected. This algorithm is a MADDPG algorithm with LSTM (long short-term memory).

[0061] The input of the actor network is the agent Sti = (P, V, L), where P and V are matrices of the same dimension, with a dimension of 45×m. The P matrix and the V matrix are combined into a two-channel image and input into the stacked sub-network. The sub-network contains two convolutional layers. The first convolutional layer contains 32 filters, each of size 4×4, with a stride of (2, 2); the second convolutional layer contains 64 filters, each of size 2×2, with a stride of (2, 2). The phase matrix L is encoded into an 8-dimensional vector using a fully connected layer. Then, the outputs of all networks are connected into a vector, and the vectors synthesized by all agents are sent to LSTMs containing 64 hidden units, which output the action prediction value Q(S, A) through the softmax activation function. The specific convolutional network structure is as follows: Figure 2 As shown,

[0062] The critic network structure is similar to the participant network. In addition to the input state space, the global action set A of all agents at the intersection must also be input.

[0063] The replay buffer D is used to randomly sample samples in each training step and update the participant network and the critic network simultaneously.

[0064] Step 3.2: Define the state space S. Read the collected target vehicle operation data. Each agent uses the vehicle position and speed in the relevant passable lane of its own assigned phase to define the state.

[0065] like Figure 3 As shown in the figure, to represent the vehicle position, the lane is divided into discrete units with an equal distance of 6 meters. Each discrete unit is a cell. If there is a car in the corresponding cell, the corresponding position value is 1, otherwise the corresponding position value is 0. The matrix Pi is placed horizontally in different directions with the right side as the stop line. The position matrix P of the intelligent agent is composed of the matrices Pi in each direction:

[0066]

[0067] To represent the vehicle speed, read the target vehicle speed and form the speed matrix Vi according to the arrangement of the P matrix. The speed matrix V of the agent is composed of the matrices Vi in each direction:

[0068]

[0069] The intersection phase scheme should be represented in the state space. The green light time of each phase is Gi, and its ratio to the cycle length is Li, forming the green light time matrix G and the phase matrix L:

[0070] G=[G1 G2 G3 … Gn]

[0071] L=[L1 L2 L3 … Ln]

[0072] In summary, the intersection state is defined as St = (P, V, L) at discrete time step t.

[0073] Step 3.3: Define the action space A. The agent controlling the current phase selects an action at the end of the phase green light time. To ensure system stability, the phase green light time should vary within a small range, and the green light time Gi of each phase should be limited between the maximum green light time Gmax and the minimum green light time Gmin (Gmin≤Gi≤Gmax).

[0074] The maximum green light time calculation formula is:

[0075]

[0076] Where: G max - Maximum green time,

[0077] C max ——The maximum cycle time is recommended to be 180 seconds;

[0078] L——total lost time,

[0079] y——critical flow ratio of this phase,

[0080] Y——the sum of key flow ratios of each phase;

[0081] Minimum green light time calculation formula:

[0082]

[0083] Where: G min - Minimum green time,

[0084] PL p ——longitudinal length of the crosswalk,

[0085] Pv p ——The pedestrian crossing speed is usually 1.2m / s.

[0086] I——Green light time interval;

[0087] Its action set is set to Ai = (-5, -4, -3, -2, -1, 0, +1, +2, +3, +4, +5). If the agent chooses ai = +3, it means that the green light duration of the current phase is increased by 3 seconds, and the changed phase time is converted and used to update the phase matrix in the state space.

[0088] Step 3.4: Definition of reward value R. Since different indicators have different degrees of influence on each agent's strategy, the harmonic weight value of multiple intersection traffic parameters is used as the agent's reward value, that is, R = W1R l +W2R w +W3R d +W4R c , (in the present invention, W1 is -0.25, W2 is 0.2, W3 is -1, and W4 is 1), wherein:

[0089] (1) Queue length Rl: the sum of the queue lengths lij of all relevant roads controlled by the agent, which is obtained from lane-level traffic data collection:

[0090]

[0091] Dij — queue lengths of all lanes controlled by the agent;

[0092] (2) Waiting time Rw: the sum of the waiting time Wij of all vehicles on the relevant roads controlled by the agent, which is given by the combination of the target vehicle operation data and lane-level traffic data:

[0093]

[0094] Wijn — the queue time of all vehicles in all lanes controlled by the agent;

[0095] (3) Average delay Rd: The average delay of all relevant roads controlled by the agent, which is obtained from lane-level traffic data collection:

[0096]

[0097] dijn – the average delay across all lanes controlled by the agent;

[0098] (4) The vehicle Rc passing through the intersection is obtained from the target vehicle operation data collection:

[0099]

[0100] Cijn is the sum of all vehicles passing the stop line in all lanes controlled by the agent during the green light time of the phase;

[0101] In step S4, neural network training is performed using the MARDDPG algorithm in deep reinforcement learning to train the traffic signal control agent. This algorithm selects the optimal action by updating the action strategy. Specifically, it includes the following sub-steps:

[0102] Step 4.1: Initialize the parameterized action selection strategy of the actor network and the critic network in Memory the historical data of the i-th agent in the network at a certain time step t, including state, reward, and action information; construct and parameterize the value function in the critic network The inputs to the value function include the historical state h and the action a chosen by the participant network;

[0103] Separately, initialize the weights θ' of all target networks i , Initialize the replay buffer D, reset the time step to zero, and read the initial state s of the participant network t,i .

[0104] Step 4.2: Every 5 seconds, a single agent selects a strategy through action selection in Explore the noise for the current time step; the actor network selects action a from the action set Ai t,i And execute, receive reward value r after execution t,i With the new state s t+1,i , generate new historical data

[0105] Step 4.3, the {s 1,i , a 1,i , r 1,i , s 2,i , a 2,i , r 2,i ,....} is stored in the replay buffer D. In the present invention, the buffer storage size is set to 20,000; the agent i samples M historical training steps stored in the replay buffer D for training the participant and critic network.

[0106] Step 4.4: After the agent selects M historical training step data, the number of samples of M in the present invention is 64. Use the 64 historical data to randomly divide into several minibatches (represented by m). Specifically, first take 64 data from the buffer, but do not update the 64 data. The 64 data are randomly divided into several minibatches, and then the minibatch data is used to update the loss function; the state and the action selected by the participant network are input into the critic network, so that the critic network passes the value function constructed in step 4.1. Estimate the Q value and update the parameters in the critic target network by minimizing the average loss function The participant network uses the reward value returned by the critic network to calculate the policy gradient through the loss function to update the parameters θ in the participant target network. i .

[0107] For update The loss function is as follows:

[0108]

[0109] represents the average Q value of the actions selected by M minibatches of all agent-critic networks; is the total reward value; γ is the reduction coefficient, which is 0.99 in this invention; Represents the sum of the Q values ​​of all agents in the next time step.

[0110] To update θ i The objective function is as follows:

[0111]

[0112] In order to maximize the future expected reward to update the participant network, define J(θ i ) to find the direction that maximizes the cumulative reward.

[0113] Step 4.5: All agents in the system use the “soft update” method to update the parameters of the target network. and θ' i First, define a rate τ (0<τ<1) for updating the target network based on the main network. In this invention, the value is 0.001. Then, the target network is updated using a convex combination of the current network parameters and the target network parameters, as follows:

[0114]

[0115] θ' i =τθ i +(1-τ)θ' i

[0116] Repeat steps 4.2 to 4.5 when (In the present invention, Δp is taken as 0.05), the optimal state in the traversable state space has been reached and the agent has completed training.

[0117] Step S5: Use the trained neural network for signal control. According to the agent construction requirements in steps 3.2 and 3.4, real-time holographic data is collected, and the input signal controls the agent. The action is changed by the duration of each phase signal output by the agent to generate a phase-time matrix:

[0118] G'=[G1' G2' G3' … Gn']

[0119] Where Gn' is the green light time optimized by the agent controlling phase n. After combining them, they are input into the traffic signal before the start of the next cycle, and the optimized signal control scheme is used for traffic signal control.

[0120] The above disclosure is only a preferred embodiment of the present invention, and certainly cannot be used to limit the scope of the rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A single-point traffic signal control method based on intersection holographic data, characterized in that: The following steps are involved: Collect initial holographic traffic data; Holographic traffic data processing; Build deep reinforcement learning models with multiple agents; In the process of building a deep reinforcement learning model containing multiple agents, the traffic signal control agent Agent uses the MARDDPG algorithm for deep reinforcement learning and defines the state space S, action space A and reward value R respectively; The input of the participant network is the agent Sti = (P, V, L), where P and V are matrices of the same dimension, 45×m. The P matrix and the V matrix are combined into a two-channel image and input to the stacked sub-network; the sub-network contains two convolutional layers. The first convolutional layer contains 32 filters, each of size 4×4, with a stride of (2, 2); the second convolutional layer contains 64 filters, each of size 2×2, with a stride of (2, 2); the phase matrix L is encoded into an 8-dimensional vector using a fully connected layer; the outputs of all networks are concatenated into a vector, and the vectors synthesized by all agents are sent to LSTMs with 64 hidden units, which output action predictions Q(S, A) through a softmax activation function; Use MARDDPG algorithm for neural network training; The process of training a neural network using the MARDDPG algorithm includes the following steps: Step 1: Initialize the parameterized action selection strategy of the actor network and the critic network in is the historical memory data of the i-th agent at a certain time step t in the participant network, and the historical memory data of the critic network It is also defined as the historical memory data of the i-th agent in the network at a certain time step t; construct and parameterize the value function in the critic network The input to the value function includes the historical state h and the action a selected by the participant network; respectively, initialize the weights θ' of all target networks i and Initialize the replay buffer D, reset the time step to zero, and read the initial state s of the participant network t,i ; Step 2: Every 5 seconds, a single agent selects a strategy through action selection in Explore the noise for the current time step; the actor network selects action a from the action set Ai t,i And execute, receive reward value r after execution t,i With the new state s t+1,i , generate new historical data Step 3: Convert the 1,i , a 1,i , r 1,i , s 2,i , a 2,i , r 2,i , ...} are stored in the replay buffer D, with the number of samples being no less than 20,000; agent i samples M historical training steps stored in the replay buffer D for training the participant and critic networks; Step 4: After the agent selects M historical training step data, it uses a minibatch in M ​​to make the critic network pass the value function constructed in step 1 Estimate the Q value and update the parameters in the critic target network by minimizing the average loss function Where minibatch is represented by m; similarly, the participant network calculates the policy gradient through the loss function to update the parameters θ in the participant target network i ; For update The loss function is as follows: represents the average Q value of the actions selected by M minibatches of all agent-critic networks; is the total reward value; γ is the reduction coefficient; represents the sum of the Q values ​​of all agents in the critic network at the next time step; To update θ i The objective function is as follows: In order to maximize the future expected reward to update the participant network, define J(θ i ) to find the direction that maximizes the cumulative reward; Step 5: The agent uses a "soft update" method to update the parameters of the target network and θ' i First, define a rate τ (0 < τ < 1) for updating the target network based on the main network; update the target network using a convex combination of the current network parameters and the target network parameters, as follows: I will i =tθ i +(1-τ)θ' i Repeat steps 2 to 5. When , the optimal state in the traversable state space has been reached and the agent has completed training; Use trained neural networks for signal control.

2. The single-point traffic signal control method based on intersection holographic data according to claim 1, characterized in that: The holographic traffic data includes target vehicle operation data, lane-level traffic data, intersection design status and traffic signal control status, wherein the target vehicle operation data includes the target vehicle ID with timestamp, vehicle type, vehicle longitudinal speed, vehicle lane number and vehicle distance from the stop line; the lane-level traffic data includes the target lane queue length, total vehicle waiting time, average delay and the number of vehicles passing the stop line; the intersection design status includes the number of lanes at each entrance lane of the intersection and the lane function distribution; the traffic signal control status includes the current phase sequence of the intersection and the distribution of the duration of each phase.

3. The single-point traffic signal control method based on intersection holographic data according to claim 1, characterized in that: The process of holographic traffic data processing includes the following steps: Delete redundant data; Delete abnormal trajectory data; Linear function interpolation method is used to complete the noisy trajectory data.

4. The single-point traffic signal control method based on intersection holographic data according to claim 1, characterized in that: The process of using the trained neural network for signal control is to collect real-time holographic data according to the requirements of the intelligent agent construction, input signals to control the intelligent agent, change the action according to the duration of each phase signal output by the intelligent agent, and generate the phase time matrix G': G'=[G1'G2'G3'…Gn'].

Images