Traffic light control method for single intersection based on improved deep reinforcement learning

Abstract

The present invention relates to a traffic light control method for a single intersection based on improved deep reinforcement learning, which obtains the density of current vehicles, the current vehicle speed and the current vehicle waiting time of the currently controlled single intersection, and defines the intelligent agent state value S, the action space A, the action selection strategy ε and the reward function based on the current vehicle density, the current vehicle speed and the current vehicle waiting time. r ; Create a neural network, a target network and an experience pool E, and transform the agent state value S, action space A, action selection strategy ε and reward function r Stored in the experience pool E, a batch is sampled from it to train the created neural network; the parameters of the trained neural network are updated to the target network; the traffic lights at a single intersection are controlled based on the trained neural network and the target network with updated parameters. The introduction of the experience replay mechanism utilizes past experience, improves the utilization rate of samples, speeds up learning, and allows the model to converge faster.

Description

Technical Field

[0001] The present invention belongs to the technical field of traffic light control, and in particular to a single intersection traffic light control method based on improved deep reinforcement learning. Background Art

[0002] In recent years, the rapid development of artificial intelligence technology has provided an opportunity for the application of intelligent transportation systems (ITS), which can help optimize traffic flow and improve traffic efficiency and safety. Among them, traffic flow detection and traffic signal control are particularly important. As a source of information for traffic control, traffic flow detection allows traffic managers to quickly understand the actual situation of the current road surface, thereby speeding up the handling of emergency or complex events and better providing real-time information and suggestions to drivers and passengers. As the core tool for traffic flow control, traffic lights are changing from traditional static modes to intelligent and dynamic control methods to adapt to changing traffic needs and alleviate urban traffic congestion.

[0003] Traffic flow detection is mainly based on vehicle flow target detection methods. Traditional methods usually rely on related hardware equipment, such as geomagnetic sensors, infrared sensors or microwave radar lights to perform statistical analysis on the vehicle body, which has high real-time and certain accuracy. Another more mainstream method is to use computer vision (CV) technology to detect vehicles in the video under the camera. This method can obtain real-time and comprehensive traffic information and is easy to install and maintain. It also has the characteristics of high accuracy. The current traffic light control algorithms are mainly the following:

[0004] By analyzing historical traffic information such as the presence of pedestrians, the classification of vehicles in the area, the appropriate assessment of travel time, and different levels of traffic congestion, the discrete time model is used to describe the system, periodic sampling is performed, and the timing plan is obtained by solving mathematical programming problems. Then, the time interval of each phase signal light is pre-set, and no relevant adjustments are made due to changes in road conditions, thereby completing the static fixed duration control of traffic lights. This static fixed duration control of traffic lights is currently the most commonly used signal control method because it is simple and easy to maintain and does not require dynamic adjustment, and users can predict signal changes. However, this method may lead to a waste of time and energy. When the vehicle density in a certain direction changes frequently, the shortcomings of fixed duration control become apparent, which can easily cause traffic congestion.

[0005] The Max-pressure model is constructed by factors such as traffic flow, signal control strategy of intersections and vehicle queue length. The real-time traffic flow data of the intersection, vehicle arrival rate and queue situation are input. The model is used to calculate the "pressure" of each intersection (i.e. the ratio of vehicle arrival rate to queue length), and the direction with the greatest pressure is selected for signal release. With the acquisition of real-time information and pressure calculation, the phase and duration of the signal light are dynamically adjusted to achieve dynamic duration control of the traffic light. Model-based traffic light control methods usually rely on specific traffic flow models, which may not fully reflect the complexity of actual traffic conditions, resulting in unreliable results. This limitation is particularly obvious in a dynamically changing traffic environment, which may lead to frequent signal adjustments and unnecessary traffic congestion. Its algorithm also has some inherent defects, such as weak local optimization ability, low calculation accuracy and poor convergence.

[0006] The system collects and processes real-time traffic flow data, such as the number of vehicles, speed, and density, through various sensors and monitoring equipment. SCOOT then analyzes the traffic congestion and flow change trends based on the collected traffic flow data to determine the appropriate control cycle and signal phase duration. Then, based on the control cycle and phase duration, the offset and phase setting of the traffic signal are calculated, so that the traffic light can be adaptively controlled. Adaptive traffic light control methods are combined with a variety of algorithms to solve traffic problems, but algorithmic defects may lead to modeling difficulties and the generation of local optimal solutions. Since this type of method requires efficient data collection and processing capabilities, as well as powerful algorithm optimization capabilities, it is sensitive to changes in road networks and traffic flow, which may lead to instability and uncertainty in traffic signal control.

[0007] The intersection signal control problem is transformed into a Markov decision process, which is represented by a state space, an action space, a state transition probability, a reward function, and a discount factor. Among them, the state in the state space mainly includes traffic flow information related to queue length, cumulative waiting time, average vehicle speed, and number of vehicles. The action space is represented by changing the phase of the traffic light. The state transition probability is the probability distribution of the next possible state obtained at a certain moment under the combined action of the state and the action of the agent. The value function is determined by the state value function and the action value function. The A2C algorithm of deep reinforcement learning usually requires a large number of samples to train, and the utilization rate of past experience is low, resulting in a long training time. Moreover, with the addition of a large amount of new data, the training process may face instability, especially when the learning rate is not adjusted properly, which will lead to performance fluctuations.

[0008] Deep reinforcement learning (DRL) is a method that combines deep learning (DL) and reinforcement learning (RL) to learn through continuous trial and error. The three core elements of DRL are state, action, and reward. State is information that describes the current environment of the agent, and action is the response of the agent based on the state, which may be discrete or continuous. Reward is the immediate feedback obtained by the agent after taking an action. Usually, a mechanism is set to incentivize or punish the behavior of the agent. Positive rewards encourage the agent to learn a certain action, while negative rewards inhibit inappropriate behavior. The purpose of DRL is to maximize the cumulative reward and optimize the action selection through the interaction between the agent and the environment. The agent observes the state of the environment at each moment, selects the corresponding action, and obtains rewards based on the results. The behavior is adjusted through the reward mechanism to achieve the optimal response to the environment. Through these mechanisms, such as Figure 1 As shown. The Advantage Actor Critic (A2C) algorithm is used to update and iterate the parameters of the policy function. By continuously adding data and optimizing the action selection strategy using the reward function, the agent can achieve rapid convergence, thereby allowing the agent to complete the control of the traffic light. The A2C algorithm based on deep reinforcement learning usually requires a large number of samples for training, and the utilization rate of past experience is low, resulting in a long training time. Moreover, with the addition of a large amount of new data, the training process may face instability, especially when the learning rate is not adjusted properly, which will cause performance fluctuations.

[0009] Therefore, how to improve the existing deep reinforcement learning traffic light control method to reduce model training samples, improve sample utilization, speed up learning, enable the model to converge faster, and at the same time improve the stability and accuracy of the model is a technical problem that needs to be solved urgently. Summary of the invention

[0010] The purpose of the present invention is to provide a single intersection traffic light control method based on improved deep reinforcement learning, so as to reduce the model training samples, improve the sample utilization rate, speed up the learning speed, enable the model to converge faster, and at the same time improve the stability and accuracy of the model.

[0011] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0012] The traffic light control method for a single intersection based on improved deep reinforcement learning includes the following steps:

[0013] S1: Obtain the current vehicle density, current vehicle speed and current vehicle waiting time of the currently controlled single intersection, and define the agent state value S, action space A, action selection strategy ε and reward function based on the current vehicle density, current vehicle speed and current vehicle waiting time r ;

[0014] S2: Create a neural network Q(s, a ;θ) and the target network Q'(s', a θ - ) and experience pool E, the agent state value S, action space A, action selection strategy ε and reward function r Stored in the experience pool E, and a batch of pairs of neural networks Q(s, a ; θ) for training;

[0015] S3: The trained neural network Q(s, a ; θ) parameters are updated to the target network Q'(s'), a θ - );

[0016] S4: Based on neural network Q(s, a ;θ) and the target network Q'(s', a ; θ-) jointly control the traffic lights at a single intersection.

[0017] Preferably, the specific process of defining the state value S in step S1 is:

[0018] S110: Evenly distribute the roads of the controlled single intersection and traverse the vehicle list;

[0019] S111: When there is a vehicle on the main road, the distance between it and the traffic light intersection is calculated to determine the cell position of the vehicle at the current time step, and the number of vehicles in the cell is counted;

[0020] S112: according to the preset maximum number of vehicles carried by each cell, the vehicle density of the current cell is calculated and represented by using discrete traffic coding;

[0021] S113: introducing speed parameters, constructing the average speed of vehicles in the current cell and setting it as a speed matrix;

[0022] S114: Merge two separate channels S p and S v , and obtain the agent state value S.

[0023] Preferably, the specific formula for calculating and expressing the vehicle density of the current cell using discrete traffic coding in step S112 is:

[0024] ;

[0025] in, For the i Line The vehicle density values ​​in the column, l is the length of a single traffic road, x The distance to count for the partition.

[0026] In step S113, the speed parameter is introduced , the specific formula for constructing the average speed of the vehicle in the current cell and setting it as the speed matrix is:

[0027] ;

[0028] In step S114, two separate channels are merged. S p and S v , the specific formula for obtaining the agent state value S is as follows:

[0029] .

[0030] Preferably, the action space A is represented based on the way of changing the phase of the traffic light, and the action phase includes 4 types, namely {0, 1, 2, 3}, where 0 represents the green light for turning right when going straight in the north-south direction; 1 represents the green light for turning left in the north-south direction; 2 represents the green light for turning right when going straight in the east-west direction; 3 represents the green light for turning left in the east-west direction.

[0031] Preferably, the cumulative reward data is used to construct the action selection strategy function ε( g ), the action selection strategy function ε( g ) is as follows:

[0032] ;

[0033] The specific formula of the strategy expression π(a | s) is:

[0034] ;

[0035] in, g i For the i The cumulative reward obtained from the selection, a is the learning rate, a * is the optimal learning rate, k is the correction factor.

[0036] Preferably, the reward function r The reward function is based on the waiting time T of the vehicle and the length L of the queue as the criteria for rewarding and punishing the agent after its action. r Correct the coefficients and the reward function after correction r t The specific formula is:

[0037] ;

[0038] in, T i It is i The waiting time of the vehicle after the action, Li It is i The length of the queue after the action, α ∈(0,1), β ∈(0,1), α and β These are correction parameters.

[0039] Preferably, the neural network structure in step S2 includes an input layer, a hidden layer and an output layer;

[0040] The first layer of the hidden layer is a 5×5 convolution layer with 32 convolution kernels; the second layer is a 3×3 maximum pooling layer; the third layer is a 3×3 convolution layer with 64 convolution kernels; the fourth layer is a 3×3 convolution layer with 64 convolution kernels; the fifth layer is a 3×3 convolution layer with 64 convolution kernels;

[0041] The output layer is provided with two different branches, one output branch is used to output the state value function, and the other output branch is used to output the state-related action advantage function. Finally, the output state value function and the result are combined with the result of the action advantage function and then output.

[0042] Preferably, the input of the neural network is the current state value S, and the output is a combined value of the state value function and result of the original Q value decomposition and the result of the action advantage function;

[0043] The calculation formula of the original Q value is as follows:

[0044] ;

[0045] The specific formula for updating the target Q value Q' is as follows:

[0046] ;

[0047] The training of the neural network is performed by the stochastic gradient descent algorithm, and the loss function is expressed by calculating the difference between the actual 𝑄 value 𝑄(𝑠,𝑎, 𝜃) and the target 𝑄 value 𝑄′. The specific formula of the loss function is:

[0048] .

[0049] Preferably, the specific training process of step S2 is as follows:

[0050] S21: Initialize the neural network Q(s, a ; θ), target network Q'(s', a ; θ - ), experience pool E and action selection strategy ε;

[0051] S22: Get the status value of the current environment at the current time S t ;

[0052] S23: Based on the current network Q(s, a ;θ) and the selection strategyπ( a |s) Get the current action a t ;

[0053] S24: Enforcement Action a t , get rewards through the environment r t and the state S at time t+1 t+1 ;

[0054] S25: t , a t ,r t , s t+1 ) is stored in the experience pool E, and the number of data in the experience pool E is obtained n ;

[0055] S26: Use the action selection strategy function ε( g ) to optimize the action selection strategy ε;

[0056] S27: Sample M data from the experience pool E {(S t , a t , r t , S t+1 )} for training.

[0057] Preferably, in step S2, the current neural network Q(S', a θ -) parameters are updated to the target network Q'(s', a ;θ - ) The specific process is as follows:

[0058] Using the target network Q'(S', a θ - ) to calculate the target Q value Q' and minimize the loss function And update the neural network Q(S, a ; θ), the current neural network Q(s, a ; θ) parameters are updated to the target network Q'(s', a ; θ - ) middle.

[0059] The beneficial effects of the present invention include:

[0060] The improved deep reinforcement learning single intersection traffic light control method provided by the present invention obtains the current vehicle density, current vehicle speed and current vehicle waiting time of the currently controlled single intersection, and defines the agent state value S, action space A, action selection strategy ε and reward function based on the current vehicle density, current vehicle speed and current vehicle waiting time r ; Create a neural network, a target network and an experience pool E, and transform the agent state value S, action space A, action selection strategy ε and reward function r The data are stored in the experience pool E, and a batch is sampled therefrom to train the created neural network; the parameters of the trained neural network are updated to the target network; the traffic lights at the single intersection are controlled based on the trained neural network and the target network with updated parameters.

[0061] First, by optimizing the action selection strategy ε, a balance is achieved between exploring new strategies and using existing knowledge to avoid premature convergence to the local optimal solution. At the same time, an experience replay mechanism is introduced to repeatedly use past experience, improve the utilization rate of samples, thereby accelerating the learning speed and allowing the model to converge faster.

[0062] Secondly, by optimizing the Q-value neural network, the introduction of the ResNet structure in the feature extraction direction can obtain deeper and more comprehensive feature information. By introducing the attention module, the stable and accurate output capabilities of the results are enhanced, and the excellent control effect is basically maintained in the traffic flow pattern. In the traffic flow pattern with extremely unbalanced vehicle arrival rates in all directions, it can still quickly reduce the vehicle waiting time and queue length. Compared with deep reinforcement learning alone, it has more stable optimization effects and better generalization capabilities.

[0063] Again, the improved deep reinforcement learning algorithm can not only process real-time data, but also be trained on offline data, which makes it more effective when it needs to process large amounts of historical data.

[0064] Finally, by improving the neural network of deep reinforcement learning, the feature extraction part is composed of a single residual structure, which retains the original features on the basis of obtaining deep features to prevent the disappearance of some features. The two branches are composed of two self-attention structures respectively. Finally, the two results are combined to generate an output Q function, which increases the ability of the neural network model to obtain fixed information. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Schematic diagram of the basic framework of deep reinforcement learning in existing technologies.

[0066] Figure 2 Schematic diagram of the neural network structure of the improved deep reinforcement learning of the present invention.

[0067] Figure 3 Schematic diagram of a single intersection signal control algorithm based on an improved reinforcement learning algorithm of the present invention. DETAILED DESCRIPTION

[0068] The following is combined with Figure 1~Figure 3 The present invention is further described in detail:

[0069] Example 1

[0070] The traffic light control method for a single intersection based on improved deep reinforcement learning includes the following steps:

[0071] S1: Obtain the current vehicle density, current vehicle speed and current vehicle waiting time of the currently controlled single intersection, and define the agent state value S, action space A, action selection strategy ε and reward function based on the current vehicle density, current vehicle speed and current vehicle waiting time r ;

[0072] S2: Create a neural network Q(s, a ;θ) and the target network Q'(s', a θ - ) and experience pool E, the agent state value S, action space A, action selection strategy ε and reward function r Stored in the experience pool E, and a batch of pairs of neural networks Q(s, a ; θ) for training;

[0073] S3: The trained neural network Q(s, a ; θ) parameters are updated to the target network Q'(s'),a θ - );

[0074] S4: Based on neural network Q(s, a ;θ) and the target network Q'(s', a ; θ-) jointly control the traffic lights at a single intersection.

[0075] See also Figure 1 The existing deep reinforcement learning basic framework is mainly composed of an agent and an environment. The agent is a neural network model, and the environment is represented by available information content, which mainly includes traffic light data and vehicle data. The agent takes action A t After being transmitted to the environment, the environment obtains the current state S according to the behavior t and reward R t After that, the information is fed back to the agent, which helps the agent adjust and optimize its own strategy, thus completing the entire cycle. Usually, a large number of samples are required for training, and the utilization rate of past experience is low, resulting in a long training time. Moreover, with the addition of a large amount of new data, the training process may face instability, especially when the learning rate is not adjusted properly, which will lead to performance fluctuations.

[0076] See also Figure 2 The schematic diagram of the neural network structure of the improved deep reinforcement learning. The feature extraction part is composed of a single residual structure. On the basis of obtaining deep features, the original features are retained to prevent the disappearance of some features. The two branches are composed of two self-attention structures respectively. Finally, the two results are combined to generate an output Q function, which increases the model's ability to obtain fixed information. By optimizing the Q value neural network, the introduction of the resnet structure in the feature extraction direction can obtain deeper and more comprehensive feature information. By introducing the attention module, the stable and accurate output ability of the results is enhanced. The excellent control effect is basically maintained in the traffic flow mode. In the traffic flow mode with extremely unbalanced vehicle arrival rates in all directions, the waiting time of vehicles and the queue length can still be quickly reduced. Compared with the single deep reinforcement learning, it has a more stable optimization effect and better generalization ability. In addition, the improved deep reinforcement learning algorithm can not only process real-time data, but also be trained on offline data, which makes it more effective when it needs to process a large amount of historical data.

[0077] Example 2

[0078] Based on Example 1, the specific process of defining the state value S in step S1 is as follows:

[0079] S110: Evenly distribute the roads of the controlled single intersection and traverse the vehicle list;

[0080] S111: When there is a vehicle on the main road, the distance between it and the traffic light intersection is calculated to determine the cell position of the vehicle at the current time step, and the number of vehicles in the cell is counted;

[0081] S112: according to the preset maximum number of vehicles carried by each cell, the vehicle density of the current cell is calculated and represented by using discrete traffic coding;

[0082] S113: introducing speed parameters, constructing the average speed of vehicles in the current cell and setting it as a speed matrix;

[0083] S114: Merge two separate channels S p and S v , and obtain the agent state value S.

[0084] The specific formula for calculating and expressing the vehicle density of the current cell using discrete traffic coding in step S112 is:

[0085] ;

[0086] in, For the i Line The vehicle density values ​​in the column, l is the length of a single traffic road, x The distance to count for the partition.

[0087] In this embodiment, in step S113, the speed parameter is introduced , the specific formula for constructing the average speed of the vehicle in the current cell and setting it as the speed matrix is:

[0088] ;

[0089] In step S114, two separate channels are merged. S p and S v , the specific formula for obtaining the agent state value S is as follows:

[0090] .

[0091] The action space A is represented based on the way of changing the phase of the traffic light. The action phase includes 4 types, namely {0, 1, 2, 3}, where 0 represents a green light for turning right when going straight in the north-south direction; 1 represents a green light for turning left in the north-south direction; 2 represents a green light for turning right when going straight in the east-west direction; 3 represents a green light for turning left in the east-west direction. At the same time, the minimum duration of each action is set to 10s. In order to prevent accidents caused by alternating vehicles at the intersection, the duration of the yellow light is increased by 3s.

[0092] In the prior art, the action selection strategy ε is usually set to a fixed value, but this will slow down the convergence of the algorithm, and the action selection strategy ε is often strongly correlated with the cumulative reward g. The action selection strategy function ε( g ), increase the convergence speed of the algorithm, the action selection strategy function ε( g ) is as follows:

[0093] ;

[0094] The specific formula of the strategy expression π(a | s) is:

[0095] .

[0096] in, g i For the i The cumulative reward obtained from the selection, a is the learning rate, a * is the optimal learning rate, k is the correction factor.

[0097] By optimizing the action selection strategy ε, a balance is achieved between exploring new strategies and using existing knowledge to avoid premature convergence to the local optimal solution. At the same time, the experience replay mechanism is introduced to repeatedly use past experience, improve the utilization rate of samples, thereby accelerating the learning speed and allowing the model to converge faster.

[0098] The reward function r The reward function is based on the vehicle waiting time T and the queue length L as the criteria for rewarding and punishing the agent after it takes action. The longer the vehicle waiting time and the queue length are, the worse the effect of the agent after taking action is. Considering that the vehicle waiting time and the queue length have a certain correlation, the reward function is used to reward the agent after taking action. r Correct the coefficients and the reward function after correction r t The specific formula is:

[0099] ;

[0100] in, T i It is i The waiting time of the vehicle after the action, Li It is i The length of the queue after the action, α ∈(0,1), β ∈(0,1), α andβ These are correction parameters.

[0101] Example 3

[0102] On the basis of Example 1 or Example 2, the neural network structure in step S2 includes an input layer, a hidden layer and an output layer. The first layer of the hidden layer is a 5×5 convolution layer with 32 convolution kernels; the second layer is a 3×3 maximum pooling layer; the third layer is a 3×3 convolution layer with 64 convolution kernels; the fourth layer is a 3×3 convolution layer with 64 convolution kernels; the fifth layer is a 3×3 convolution layer with 64 convolution kernels.

[0103] Since the state is represented by the discrete coding matrix S, the improved reinforcement learning algorithm uses a convolutional neural network (CNN) as the backbone network of the action value prediction function. According to the Dueling network concept, the original single-channel output is changed to two different branches. The two different branches of the output layer, one output branch is used to output the state value function, and the other output branch is used for the state-related action advantage function. Finally, the output state value function and the result and the result of the action advantage function are combined and output.

[0104] The feature extraction part of the neural network structure is composed of a single residual structure. On the basis of obtaining deep features, the original features are retained to prevent the disappearance of some features. The two branches are composed of two self-attention structures respectively. Finally, the two results are combined to produce an output Q function, which increases the model's ability to obtain fixed information. By optimizing the Q value neural network, the introduction of the resnet structure in the feature extraction direction can obtain deeper and more comprehensive feature information. By introducing the attention module, the stable and accurate output capability of the results is enhanced. In the traffic flow mode, the excellent control effect is basically maintained. In the traffic flow mode with extremely unbalanced vehicle arrival rates in all directions, the waiting time of vehicles and the queue length can still be quickly reduced. Compared with the single deep reinforcement learning, it has a more stable optimization effect and better generalization ability. In addition, the improved deep reinforcement learning algorithm can not only process real-time data, but also be trained on offline data, which makes it more effective when it needs to process a large amount of historical data.

[0105] The input of the neural network is the current state value S, and the output is the combined value of the state value function and result of the original Q value decomposition and the result of the action advantage function;

[0106] The calculation formula of the original Q value is as follows:

[0107] ;

[0108] The specific formula for updating the target Q value Q' is as follows:

[0109] ;

[0110] The training of the neural network is performed by the stochastic gradient descent algorithm, and the loss function is expressed by calculating the difference between the actual 𝑄 value 𝑄(𝑠,𝑎, 𝜃) and the target 𝑄 value 𝑄′. The specific formula of the loss function is:

[0111] .

[0112] Experience replay technology is used to break the correlation between data and speed up the convergence of the model. In this method, the state s, action a, reward r at each moment and the state s' at the next moment are stored in the experience pool, and a batch of data is sampled from it for training.

[0113] See also Figure 3 , the specific training process of step S2 is as follows:

[0114] S21: Initialize the neural network Q(s, a ; θ), target network Q'(s', a; θ - ), experience pool E and action selection strategy ε;

[0115] S22: Get the status value of the current environment at the current time S t ;

[0116] S23: According to the current network Q(s, a ;θ) and the selection strategyπ( a |s) Get the current action a t ;

[0117] S24: Enforcement Action a t , get rewards through the environment r t and the state S at time t+1 t+1 ;

[0118] S25: t , a t ,r t , s t+1 ) is stored in the experience pool E, and the number of data in the experience pool E is obtained n ;

[0119] S26: Use the action selection strategy function ε( g ) to optimize the action selection strategy ε;

[0120] S27: Sample M data from the experience pool E {(S t , a t , r t , S t+1 )} for training.

[0121] In this embodiment, the current neural network Q(S', a θ - ) parameters are updated to the target network Q'(s', a ; θ - ) The specific process is as follows:

[0122] Using the target network Q'(S', a θ - ) to calculate the target Q value Q' and minimize the loss function And update the neural network Q(S, a ; θ), the current neural network Q(s, a ; θ) parameters are updated to the target network Q'(s', a ; θ - ) middle.

[0123] In summary, the traffic light control method for a single intersection based on improved deep reinforcement learning provided by the present invention achieves a balance between exploring new strategies and using existing knowledge by optimizing the action selection strategy ε, thereby avoiding premature convergence to a local optimal solution. At the same time, an experience replay mechanism is introduced, which can repeatedly use past experience, improve the utilization rate of samples, thereby accelerating the learning speed and allowing the model to converge faster. By optimizing the Q-value neural network, in the feature extraction direction, the introduction of the resnet structure can obtain deeper and more comprehensive feature information. By introducing the attention module, the stable and accurate output capability of the results is enhanced, and the excellent control effect is basically maintained in the traffic flow mode. In the traffic flow mode with extremely unbalanced vehicle arrival rates in all directions, the vehicle waiting time and queue length can still be quickly reduced. Compared with deep reinforcement learning alone, it has a more stable optimization effect and better generalization ability.

[0124] The improved deep reinforcement learning algorithm can not only process real-time data, but also be trained on offline data, which makes it more effective when processing a large amount of historical data. By improving the neural network of deep reinforcement learning, the feature extraction part is composed of a single residual structure, which retains the original features on the basis of obtaining deep features to prevent the disappearance of some features. The two branches are composed of two self-attention structures respectively, and finally the two results are combined to generate an output Q function, which increases the ability of the neural network model to obtain fixed information.

Claims

1. A traffic light control method for a single intersection based on improved deep reinforcement learning, characterized in that: The following steps are involved: S1: Obtain the current vehicle density, current vehicle speed and current vehicle waiting time of the currently controlled single intersection, and define the agent state value S, action space A, action selection strategy ε and reward function based on the current vehicle density, current vehicle speed and current vehicle waiting time r ; S2: Create a neural network Q(s, a ;θ) and the target network Q'(s', a θ - ) and experience pool E, the agent state value S, action space A, action selection strategy ε and reward function r Stored in the experience pool E, and a batch of pairs of neural networks Q(s, a ; θ) for training; S3: The trained neural network Q(s, a ; θ) parameters are updated to the target network Q'(s'), a θ - ); S4: Based on neural network Q(s, a ;θ) and the target network Q'(s', a θ - ) jointly control traffic lights at single intersections; The specific process of defining the state value S in step S1 is as follows: S110: Evenly distribute the roads of the controlled single intersection and traverse the vehicle list; S111: When there is a vehicle on the main road, the distance between it and the traffic light intersection is calculated to determine the cell position of the vehicle at the current time step, and the number of vehicles in the cell is counted; S112: according to the preset maximum number of vehicles carried by each cell, the vehicle density of the current cell is calculated and represented by using discrete traffic coding; S113: introducing speed parameters, constructing the average speed of vehicles in the current cell and setting it as a speed matrix; S114: Merge two separate channels S p and S v , obtain the agent state value S; The specific formula for calculating and expressing the vehicle density of the current cell using discrete traffic coding in step S112 is: ； in, For the i Line The vehicle density values ​​in the column, l is the length of a single traffic road, x The distance counted for the partition; In step S113, the speed parameter is introduced , the specific formula for constructing the average speed of the vehicle in the current cell and setting it as the speed matrix is: ； In step S114, two separate channels are merged. S p and S v , the specific formula for obtaining the agent state value S is as follows: ； The action selection strategy function ε( g ), the action selection strategy function ε( g ) is as follows: ； The specific formula of the strategy expression π(a | s) is: ； in, gi For the i The cumulative reward obtained from the selection, a is the learning rate, a * is the optimal learning rate, k is the correction factor 2. The traffic light control method for a single intersection based on improved deep reinforcement learning according to claim 1 is characterized in that: The action space A is represented based on the way of changing the phase of the traffic light. The action phases of the action space A include four types, namely {0, 1, 2, 3}, where 0 represents a green light for turning right when going straight in the north-south direction; 1 represents a green light for turning left in the north-south direction; 2 represents a green light for turning right when going straight in the east-west direction; and 3 represents a green light for turning left in the east-west direction.

3. The traffic light control method for a single intersection based on improved deep reinforcement learning according to claim 1 is characterized in that: The reward function r The reward function is based on the waiting time T of the vehicle and the length L of the queue as the criteria for rewarding and punishing the agent after its action. r Correct the coefficients and the reward function after correction r t The specific formula is: ； in, T i It is i The waiting time of the vehicle after the action, Li It is i The length of the queue after the action, α ∈(0,1), β ∈(0,1), α and β These are correction parameters.

4. The traffic light control method for a single intersection based on improved deep reinforcement learning according to claim 1 is characterized in that: The neural network structure in step S2 includes an input layer, a hidden layer and an output layer; The first layer of the hidden layer is a 5×5 convolution layer with 32 convolution kernels; the second layer is a 3×3 maximum pooling layer; the third layer is a 3×3 convolution layer with 64 convolution kernels; the fourth layer is a 3×3 convolution layer with 64 convolution kernels; the fifth layer is a 3×3 convolution layer with 64 convolution kernels; The output layer is provided with two different branches, one output branch is used to output the state value function, and the other output branch is used to output the state-related action advantage function. Finally, the output state value function and the result are combined with the result of the action advantage function and then output.

5. The traffic light control method for a single intersection based on improved deep reinforcement learning according to claim 4 is characterized in that: The input of the neural network is the current state value S, and the output is the combined value of the state value function and result of the original Q value decomposition and the result of the action advantage function; The calculation formula of the original Q value is as follows: ； The specific formula for updating the target Q value Q' is as follows: ； The training of the neural network is performed by the stochastic gradient descent algorithm, and the loss function is expressed by calculating the difference between the actual 𝑄 value 𝑄(𝑠, 𝑎,𝜃) and the target 𝑄 value 𝑄′. The specific formula of the loss function is: 。 6. The traffic light control method for a single intersection based on improved deep reinforcement learning according to claim 1 is characterized in that: The specific training process of step S2 is as follows: S21: Initialize the neural network Q(s, a ; θ), target network Q'(s', a; θ - ), experience pool E and action selection strategy ε; S22: Get the status value of the current environment at the current time S t ; S23: According to the current network Q(s, a ;θ) and the selection strategyπ( a |s) Get the current action a t ; S24: Enforcement Action a t , get rewards through the environment r t and the state S at time t+1 t+1 ; S25: t , a t ,r t , s t+1 ) is stored in the experience pool E, and the number of data in the experience pool E is obtained n ; S26: Use the action selection strategy function ε( g ) to optimize the action selection strategy ε; S27: Sample M data from the experience pool E {(S t , a t , r t , S t+1 )} for training.

7. The traffic light control method for a single intersection based on improved deep reinforcement learning according to claim 6 is characterized in that: In step S2, the current neural network Q(S', a θ - ) parameters are updated to the target network Q'(s', a ; θ - )The specific process is as follows: Using the target network Q'(S', a θ - ) to calculate the target Q value Q' and minimize the loss function And update the neural network Q(S, a ; θ), the current neural network Q(s, a ; θ) parameters are updated to the target network Q'(s', a ; θ - ) middle.

Images