Traffic signal light strategy evaluation evolution method and system

Abstract

The application belongs to the technical field of intelligent control of traffic signal lights, and discloses a traffic signal light strategy evaluation evolution method and system; wherein the traffic signal light strategy evaluation evolution method comprises the following steps: modeling a signal light control strategy problem of a traffic road network as a Markov decision process model, and constructing a traffic signal light strategy training network; based on the traffic signal light strategy training network, constructing a traffic signal strategy evaluation evolution algorithm based on a deep Q network, and realizing traffic signal light strategy evaluation evolution; wherein in the traffic signal strategy evaluation evolution algorithm, the value function and the advantage function of the duel network are used to evaluate the advantage of a strategy, predict the best action, gradually optimize the strategy through parameter gradient updating, and perform advantage strategy evolution. The technical scheme disclosed by the application realizes the evaluation of the advantage traffic signal light control strategy through the duel network, and can solve the problem of slow strategy evolution convergence.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control technology for traffic lights, and specifically relates to a method and system for evaluating and evolving traffic light strategies. Background Technology

[0002] Traffic signal strategies are crucial for improving traffic efficiency and alleviating traffic congestion. A good traffic signal control strategy can intelligently adjust the duration of red and green lights, effectively alleviate traffic congestion, improve road utilization efficiency, reduce traffic accidents, and is of great significance to urban development.

[0003] Currently, traditional fixed-duration control methods are heavily reliant on expert experience and cannot adapt to complex and ever-changing dynamic traffic flows. In light of these shortcomings, researchers have proposed reinforcement learning-based traffic light control methods (for example, Colight, which enhances collaboration between intersections through dynamic communication and information sharing; and Presslight, which overcomes performance parameter sensitivity through maximum pressure state and reward design). These methods can overcome these shortcomings to some extent; however, they all require long learning processes and do not fully consider the temporal information of traffic intersections and the quality of control strategies. This results in slow convergence of traffic signal control strategies and large fluctuations in the early stages of evolution, highlighting the urgent need for a scientific and effective strategy evaluation method.

[0004] The relatively mature policy evaluation method, alpha-rank, introduces a multi-population discrete-time model and the idea of ​​genetic mutation in evolutionary theory. Given the policy payoff matrix, it can calculate the transition matrix and obtain the policy distribution and score ranking of multiple agents. However, it is more suitable for game environments, and the computational cost will be very large as the action and state space increases, making it unsuitable for deployment in environments with multiple traffic intersections for policy evaluation. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for evaluating and evolving traffic signal strategies, in order to solve one or more of the aforementioned technical problems. The technical solution provided by this invention evaluates superior traffic signal control strategies through a duel network, thus solving the problem of slow strategy evolution convergence. Furthermore, by using a spatiotemporal fusion graph neural network to fully capture the spatiotemporal information of the traffic network and vehicle flow, it can be deployed in real-world large-scale traffic intersection environments.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for evaluating and evolving traffic signal strategies, comprising the following steps:

[0008] By using multi-agent reinforcement learning, the traffic light control strategy problem of traffic road network is modeled as a Markov decision process model;

[0009] Based on the Markov decision process model, a traffic light policy training network is constructed. In the traffic light policy training network, a multilayer perceptron is used to extract the traffic conditions and neighbor information of each intersection, and a spatiotemporal fusion graph attention network is combined to dynamically capture the influence weights of neighboring intersections on the target intersection.

[0010] Based on the traffic signal policy training network, a traffic signal policy evaluation and evolution algorithm based on a deep Q-network is constructed to realize the evaluation and evolution of traffic signal policies. In the traffic signal policy evaluation and evolution algorithm, the value function and advantage function of the duel network are used to evaluate the advantage of the policy, predict the best action, and gradually optimize the policy through parameter gradient updates to carry out the evolution of the advantageous policy.

[0011] A further improvement to the method of the present invention is that the step of modeling the traffic light control strategy problem of the traffic road network into a Markov decision process model using multi-agent reinforcement learning includes:

[0012] Assume there are N traffic intersections in the system. Each intersection can be considered as an agent, and each agent takes a portion of the observed system state as its observation value. The state of an agent is defined as the observation value of agent i at time t. The action of agent i is defined as the current signal phase, represented by a one-hot vector, and the number of vehicles in each entering lane connected to the intersection; the action of agent i is defined as the set of candidate actions. One of the four signal phases As a decision for the next time period; among them, the four signal phases are east-west straight, north-south straight, east-west left turn, and north-south left turn;

[0013] Given the state s of the system at time t t and the combined action of multiple corresponding intelligent agents a t The system follows the state transition probability P(s) t+1 |s t ,a t ) Reach the next state s t+1 ;

[0014] The reward for agent i is defined as follows: Where γ is the penalty factor. Let be the number of vehicles queuing in lane l of agent i at time t;

[0015] Use the action value function Q of each agent i at the nth iteration. i (θn This approach minimizes the loss to make the deep neural network approximate the total reward. The expression is,

[0016]

[0017] in, It is the next observation value; It is the value of the next action; θ n This represents the relevant parameters.

[0018] A further improvement to the method of the present invention is that, in the traffic light policy training network, the step of extracting traffic conditions and neighbor information for each intersection using a multilayer perceptron includes:

[0019] The network inputs information about the road network and the origin and destination information of each vehicle in the traffic flow. It then extracts the signal phase and vehicle quantity information of each traffic intersection's entry lane as the input layer. A multilayer perceptron consisting of two fully connected layers is used to extract features, as shown in the formula. Among them, W e and b e The weight matrix and bias vector to be learned, h i It represents the traffic conditions at each intersection.

[0020] A further improvement of the method of the present invention is that, in the traffic signal light policy training network, the step of dynamically capturing the influence weights of neighboring intersections on the target intersection using a spatiotemporal fusion graph attention network includes:

[0021] Traffic condition information is transmitted to a spatiotemporal fusion graph neural network; in this network, the temporal module employs a long short-term memory network, and the spatial module employs a graph attention network.

[0022] In Long Short-Term Memory (LSTM) networks, the formula for capturing temporal features is h′. i =LSTM(h i );

[0023] In a graph attention network, the importance of neighboring intersections j of intersection i in determining the strategy for the target intersection i is first determined, using the formula e. ij =(h′) i W t )·(h′ j W s ) T Among them, W s and W t These are the multilayer perceptron parameters for the adjacent intersection and the target intersection, respectively; the importance score of the adjacent intersection j to the target traffic intersection i is calculated using the normalized formula,

[0024]

[0025] Where τ is the influence coefficient. It is the set of intersections adjacent to the target intersection;

[0026] Each target intersection has five neighboring intersections: the target intersection itself and four intersections geographically closest to it. The characteristics of these neighboring intersections are combined with their respective importance scores using the following formula:

[0027]

[0028] Among them, W c These are the parameters of the multilayer sensor at the adjacent intersection j; W q and b q These are the parameters to be trained;

[0029] A multi-head attention mechanism is employed, executing multiple sets of attention functions with trainable parameters in parallel, and finally averaging them. The formula is as follows:

[0030]

[0031] Where H is the number of attention heads, H = 5; It is the importance score of the h-th attention head.

[0032] A further improvement to the method of the present invention lies in that, in the traffic signal strategy evaluation evolution algorithm,

[0033] The architecture of the duel network is adopted, which consists of an evaluation network and a target network. The evaluation network and the target network are Q networks with different parameters but the same structure. Both consist of an input layer, a feature extraction layer, a spatiotemporal fusion graph neural network layer, a value function layer, an advantage function layer and an output layer.

[0034] At regular intervals, the parameters of the evaluation network are copied to the target network. The target network generates the predicted Q value and calculates the loss by comparing it with the Q value of the evaluation network, so that the evaluation network can update its gradient.

[0035] The Q network employs an experience replay mechanism, using a deep neural network to approximate the value function. It stores the trained data in a replay memory pool for subsequent random sampling and training.

[0036] At time t, the duel network generates a predicted Q-value, using the formula:

[0037]

[0038] Where θ is the parameter of the spatial module of the spatiotemporal fusion graph neural network layer, which is shared by the value function layer and the advantage function layer; α and β are the parameters of the advantage function layer and the value function layer, respectively. It is a collection of actions;

[0039] Advantage function layer generation A one-dimensional vector, the value function layer produces a one-dimensional scalar;

[0040] Features h obtained from the policy-trained network mi The formulas for the value function layer and the advantage function layer of the duel network are as follows:

[0041]

[0042]

[0043] Among them, W p1 W p2 b1 and b2 are the parameters to be learned;

[0044] The loss function formula for the evolution of the duel network optimization strategy is as follows:

[0045]

[0046] Where T is the total number of time steps affecting network updates; N is the number of intersections in the traffic network; and θ n This represents all trainable parameters.

[0047] A second aspect of the present invention provides a traffic signal strategy evaluation and evolution system, comprising:

[0048] The modeling unit is used to model the traffic light control strategy problem of traffic road network as a Markov decision process model using multi-agent reinforcement learning.

[0049] The strategy training network construction unit is used to construct a traffic light strategy training network based on a Markov decision process model. In the traffic light strategy training network, a multilayer perceptron is used to extract the traffic conditions and neighbor information of each intersection, and a spatiotemporal fusion graph attention network is combined to dynamically capture the influence weights of neighboring intersections on the target intersection.

[0050] The strategy evaluation and evolution unit is used to construct a traffic signal strategy evaluation and evolution algorithm based on a deep Q-network, based on a traffic signal strategy training network, to realize the evaluation and evolution of traffic signal strategies. In the traffic signal strategy evaluation and evolution algorithm, the value function and advantage function of the duel network are used to evaluate the advantage of the strategy, predict the best action, and gradually optimize the strategy through parameter gradient updates to carry out the evolution of the advantageous strategy.

[0051] A further improvement of the system of the present invention is that, in the modeling unit, the step of modeling the traffic light control strategy problem of the traffic road network into a Markov decision process model using multi-agent reinforcement learning includes:

[0052] Assume there are N traffic intersections in the system. Each intersection can be considered as an agent, and each agent takes a portion of the observed system state as its observation value. The state of an agent is defined as the observation value of agent i at time t. The action of agent i is defined as the current signal phase, represented by a one-hot vector, and the number of vehicles in each entering lane connected to the intersection; the action of agent i is defined as the set of candidate actions. One of the four signal phases As a decision for the next time period; among them, the four signal phases are east-west straight, north-south straight, east-west left turn, and north-south left turn;

[0053] Given the state s of the system at time t t and the combined action of multiple corresponding intelligent agents a t The system follows the state transition probability P(s) t+1 |s t ,a t ) Reach the next state s t+1 ;

[0054] The reward for agent i is defined as follows: Where γ is the penalty factor. Let be the number of vehicles queuing in lane l of agent i at time t;

[0055] Use the action value function Q of each agent i at the nth iteration. i (θ n This approach minimizes the loss to make the deep neural network approximate the total reward. The expression is,

[0056]

[0057] in, It is the next observation value; It is the value of the next action; θ n This represents the relevant parameters.

[0058] A further improvement of the system of the present invention is that, in the traffic light policy training network, the step of extracting traffic conditions and neighbor information for each intersection using a multilayer perceptron includes:

[0059] The network inputs information about the road network and the origin and destination information of each vehicle in the traffic flow. It then extracts the signal phase and vehicle quantity information of each traffic intersection's entry lane as the input layer. A multilayer perceptron consisting of two fully connected layers is used to extract features, as shown in the formula. Among them, W e and b e The weight matrix and bias vector to be learned, h i It represents the traffic conditions at each intersection.

[0060] A further improvement of the system of the present invention is that, in the traffic signal light policy training network, the step of dynamically capturing the influence weights of neighboring intersections on the target intersection using a spatiotemporal fusion graph attention network includes:

[0061] Traffic condition information is transmitted to a spatiotemporal fusion graph neural network; in this network, the temporal module employs a long short-term memory network, and the spatial module employs a graph attention network.

[0062] In Long Short-Term Memory (LSTM) networks, the formula for capturing temporal features is h′. i =LSTM(h i );

[0063] In a graph attention network, the importance of neighboring intersections j of intersection i in determining the strategy for the target intersection i is first determined, using the formula e. ij =(h′) i W t )·(h′ j W s ) T Among them, W s and W t These are the multilayer perceptron parameters for the adjacent intersection and the target intersection, respectively; the importance score of the adjacent intersection j to the target traffic intersection i is calculated using the normalized formula,

[0064]

[0065] Where τ is the influence coefficient. It is the set of intersections adjacent to the target intersection;

[0066] Each target intersection has five neighboring intersections: the target intersection itself and four intersections geographically closest to it. The characteristics of these neighboring intersections are combined with their respective importance scores using the following formula:

[0067]

[0068] Among them, W c These are the parameters of the multilayer sensor at the adjacent intersection j; W q and bq These are the parameters to be trained;

[0069] A multi-head attention mechanism is employed, executing multiple sets of attention functions with trainable parameters in parallel, and finally averaging them. The formula is as follows:

[0070]

[0071] Where H is the number of attention heads, H = 5; It is the importance score of the h-th attention head.

[0072] A further improvement to the system of the present invention lies in the fact that, in the traffic signal strategy evaluation evolution algorithm,

[0073] The architecture of the duel network is adopted, which consists of an evaluation network and a target network. The evaluation network and the target network are Q networks with different parameters but the same structure. Both consist of an input layer, a feature extraction layer, a spatiotemporal fusion graph neural network layer, a value function layer, an advantage function layer and an output layer.

[0074] At regular intervals, the parameters of the evaluation network are copied to the target network. The target network generates the predicted Q value and calculates the loss by comparing it with the Q value of the evaluation network, so that the evaluation network can update its gradient.

[0075] The Q network employs an experience replay mechanism, using a deep neural network to approximate the value function. It stores the trained data in a replay memory pool for subsequent random sampling and training.

[0076] At time t, the duel network generates a predicted Q-value, using the formula:

[0077]

[0078] Where θ is the parameter of the spatial module of the spatiotemporal fusion graph neural network layer, which is shared by the value function layer and the advantage function layer; α and β are the parameters of the advantage function layer and the value function layer, respectively. It is a collection of actions;

[0079] Advantage function layer generation A one-dimensional vector, the value function layer produces a one-dimensional scalar;

[0080] Features h obtained from the policy-trained network mi The formulas for the value function layer and the advantage function layer of the duel network are as follows:

[0081]

[0082]

[0083] Among them, W p1 Wp2 b1 and b2 are the parameters to be learned;

[0084] The loss function formula for the evolution of the duel network optimization strategy is as follows:

[0085]

[0086] Where T is the total number of time steps affecting network updates; N is the number of intersections in the traffic network; and θ n This represents all trainable parameters.

[0087] Compared with the prior art, the present invention has the following beneficial effects:

[0088] This invention provides a traffic signal strategy evaluation and evolution method, which is an effective strategy evaluation method in the actual engineering scenario of large-scale traffic intersections. It can effectively cope with the complex and ever-changing dynamic traffic flow scenarios of large-scale traffic intersections, realize the evaluation and evolution of superior traffic signal strategies, accelerate the evolution and convergence of traffic signal control strategies, minimize the average travel time of vehicles in the road network, maximize the throughput of the road network, and reduce the difficulty and cost of deployment.

[0089] To address the deployment problem of traffic signal strategies, this invention employs a spatiotemporal fusion graph neural network method. It uses a long short-term memory network to capture the temporal dependence of signal phase sequences and a graph attention network to obtain traffic flow characteristics and signal phase characteristics at traffic intersections. This allows traffic signal strategies to fully consider temporal and spatial characteristics, effectively enabling dynamic information sharing and collaboration among multiple intersections, extending to large-scale intersection environments, and making it practically deployable.

[0090] To address the problem of evaluating and evolving traffic signal strategies, this invention specifically employs a duel network method. The advantage function of the duel network is used to evaluate the advantage of the strategy and predict the signal phase action. This allows the traffic signal strategy to undergo advantage evaluation, which promotes the evolution of advantageous strategies, improves stability, avoids the exploration of ineffective strategies during the evolution process, improves learning efficiency, and accelerates the convergence of strategy evolution. Attached Figure Description

[0091] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art are briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0092] Figure 1 This is a flowchart illustrating a traffic light policy evaluation and evolution method based on deep reinforcement learning provided in an embodiment of the present invention.

[0093] Figure 2 This is a schematic diagram of the signal phase of the traffic signal control model constructed in an embodiment of the present invention;

[0094] Figure 3 This is a schematic diagram of the traffic light strategy training network constructed in an embodiment of the present invention;

[0095] Figure 4 This is a schematic diagram of the traffic signal policy evaluation evolution network based on a deep Q-network constructed in an embodiment of the present invention. Detailed Implementation

[0096] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0097] The present invention will now be described in further detail with reference to the accompanying drawings:

[0098] Please see Figure 1 The present invention provides a traffic light policy evaluation and evolution method based on deep reinforcement learning, comprising the following steps:

[0099] Step S1: Use multi-agent reinforcement learning to model the traffic light control strategy problem of the road network as a Markov decision process model;

[0100] Step S2: Based on the Markov decision process model built in Step S1, a traffic light policy training network is constructed. In the traffic light policy training network, a multilayer perceptron is first used to extract the traffic conditions and neighbor information of each intersection, and then a spatiotemporal fusion graph attention network is combined to dynamically capture the influence weights of neighboring intersections on the target intersection.

[0101] Step S3: Based on the traffic signal policy training network constructed in Step S2, construct a traffic signal policy evaluation and evolution algorithm based on a deep Q-network. Utilize the value function and advantage function of the duel network to evaluate the advantage of the policy, predict the best action, and gradually optimize the policy through parameter gradient updates to evolve the dominant policy.

[0102] The method provided in this invention can effectively cope with the complex and ever-changing dynamic traffic flow scenarios at large-scale traffic intersections, realize the evaluation and evolution of superior traffic signal strategies, accelerate the evolution and convergence of traffic signal control strategies, minimize the average travel time of vehicles in the road network, maximize the throughput of the road network, and reduce the difficulty and cost of deployment.

[0103] In a further preferred embodiment of the present invention, in step S1, when modeling the traffic light policy control problem using multi-agent reinforcement learning: multi-agent reinforcement learning is based on a Markov Decision Process (MDP), which is defined by a quadruple: P, R; where, Represents a set of states. Let represent the set of actions, P represent the state transition probability, and R represent the reward function. At each time t, the agent observes the current state s. t And select an action a t After the action is executed, the system transitions to the next state s. t+1 and receive an instant reward r t The goal of an agent is to find a policy that maximizes the expected long-term cumulative reward across all possible state transitions.

[0104] Traffic signal control problems require consideration of signal phase (i.e., the direction of the green light). The four most common signal phases are: Figure 2 As shown. Each intersection has traffic roads in four directions: east, south, west, and north. Each direction of traffic road is divided into three lanes: right turn lane, straight lane, and left turn lane. The four signal phases are: east-west straight (WSES), north-south straight (NSSS), east-west left turn (WLEL), and north-south left turn (NLSL).

[0105] In this embodiment of the invention, multi-agent reinforcement learning is used to model the traffic light control strategy problem of a traffic road network as a Markov decision process model; wherein, it is assumed that there are N traffic intersections in the system, and each intersection is regarded as an agent, and each agent can observe a part of the system state. As its observation value; the state of the agent is defined as the observation value of agent i at time t. It consists of the current signal phase represented by a one-hot vector and the number of vehicles in each entering lane connected to the intersection; the action of agent i is defined as its set of candidate actions. One of the four signal phases As a decision for the next time period. Given the state s of the system at time t. t and the combined action of multiple corresponding intelligent agents a t The system follows the state transition probability P(s) t+1 |s t ,a t ) Reach the next state s t+1 The reward for agent i is defined as follows: Where γ is the penalty factor. Let be the number of vehicles queuing in the entry lane l at intersection i at time t.

[0106] The present invention aims to minimize the average travel time of all vehicles in the traffic road network from their origin to their destination. This is an evaluation indicator that is difficult to optimize directly. Therefore, in the embodiments of the present invention, the average number of vehicles queuing at each traffic intersection is used to reflect this evaluation indicator.

[0107] In this embodiment of the invention, the action value function Q of each agent i at the nth iteration is used. i (θ n )(θ n (Representing relevant parameters), the deep neural network approximates the total reward by minimizing the loss.

[0108]

[0109] in, It is the next observation value; It is the action value for the next move.

[0110] In the technical solution of this invention embodiment, the traffic light policy control problem at multiple traffic intersections is transformed into a multi-agent reinforcement learning process by modeling a Markov decision process, which facilitates the use of deep Q-networks to reduce processing difficulty.

[0111] In a further preferred embodiment of the present invention, in step S2, the traffic light policy training network constructed based on the Markov decision process model of step S1 is as follows: Figure 3 As shown; where,

[0112] First, traffic scene information, including road network information and the origin and destination information of each vehicle in the traffic flow, is input into the network. Information such as the signal phase and number of vehicles entering the lane at each traffic intersection is extracted as the input layer. Then, a multilayer perceptron consisting of two fully connected layers (32 neurons each) is used to extract features, including traffic flow information and neighbor information such as the adjacency matrix. The formula is as follows:

[0113] Among them, W e and b e The weight matrix and bias vector to be learned, and the generated h i It represents the traffic conditions at each intersection.

[0114] Then, the traffic condition information is passed to the spatiotemporal fusion graph neural network. The temporal module uses a Long Short-Term Memory (LSTM) network, introducing gating units and unit states to control the information flow, which can solve the gradient vanishing problem in long-term time series prediction. The LSM network's gating units have three types: input gates, forget gates, and output gates, used to determine whether to add or delete information to the unit state. The formula for using the LSM network to process traffic condition information and capture temporal features is as follows:

[0115] h′ i =LSTM(h i );

[0116] The spatial module employs a graph attention network. First, it determines the importance of neighboring intersections j of intersection i in determining the strategy for the target intersection i, using the formula:

[0117] e ij =(h′) i W t )·(h′ j W s ) T ;

[0118] Among them, W s and W t These are the multi-layer sensor parameters for the adjacent intersection and the target intersection, respectively.

[0119] Furthermore, the importance score of neighboring intersection j to target intersection i is calculated using normalization, as shown in the formula.

[0120]

[0121] Where τ is the influence coefficient. It is the set of neighboring intersections of the target intersection.

[0122] In this embodiment of the invention, each target intersection is set to have 5 neighboring intersections, including the target intersection itself and 4 intersections geographically closest to the target intersection; the characteristics of multiple neighboring intersections are combined with their respective importance scores, using the following formula:

[0123]

[0124] Among them, W c These are the parameters of the multilayer sensor at the adjacent intersection j, W q and b q These are the parameters to be trained.

[0125] To better extract different features, this embodiment of the invention employs a multi-head attention mechanism, executing multiple sets of attention functions with trainable parameters in parallel, and finally averaging them, as shown in the formula:

[0126]

[0127] Where H is the number of attention heads, H = 5; It is the importance score of the h-th attention head. In an exemplary embodiment of the present invention, five attention heads are used.

[0128] The technical solution of this invention, through the construction of a traffic signal light policy training network, enables the policy-controlled intersection scale to have good scalability, and can be expanded to large-scale traffic intersections (such as close to 200 intersections).

[0129] In a further preferred embodiment of the present invention, in step S3, based on the policy training network of step S2, a traffic signal policy evaluation and evolution algorithm based on a deep Q-network is constructed using a duel network, such as... Figure 4 As shown.

[0130] In this embodiment of the invention, a duel network architecture is adopted, consisting of an evaluation network and a target network. Both are Q-networks with different parameters but identical structures, each comprising an input layer, a feature extraction layer, a spatiotemporal fusion graph neural network layer, a value function layer, an advantage function layer, and an output layer. At regular intervals, the parameters of the evaluation network are copied to the target network. The target network generates a predicted Q-value, and the loss is calculated using the Q-value of the evaluation network, facilitating gradient updates for the evaluation network. The deep Q-network employs an empirical replay mechanism, using a deep neural network to approximate the value function. The trained data is stored in a replay memory pool for subsequent random sampling for training, improving data utilization and reducing the variance of the data samples. In this embodiment, the storage size of the replay memory pool is set to 1000, i.e., storing 1000 samples.

[0131] At time t, the duel network generates a predicted Q-value, using the formula:

[0132]

[0133] Here, θ represents the parameters of the spatial module of the spatiotemporal fusion graph neural network layer, which are shared by the value function layer and the advantage function layer. α and β are the parameters of the advantage function layer and the value function layer, respectively. It is a collection of actions.

[0134] Advantage function layer generation A one-dimensional vector; the value function layer generates a one-dimensional scalar, which, for a specific state, can evaluate the advantage of taking an action relative to the average value obtainable in that state, thus playing a role in advantage assessment for traffic light strategies.

[0135] Features h obtained from the policy-trained network mi The formulas for the value function layer and the advantage function layer of the duel network are as follows:

[0136]

[0137]

[0138] Among them, W p1 W p2 b1 and b2 are the parameters to be learned.

[0139] The loss function formula for the evolution of the duel network optimization strategy is as follows:

[0140]

[0141] Where T is the total number of time steps affecting network updates, N is the number of intersections in the traffic network, and θ n This represents all trainable parameters.

[0142] In this embodiment of the invention, a traffic light strategy evaluation, which was lacking in previous technologies, is achieved through a duel network architecture. This allows for the assessment of the advantage of each control strategy, accelerating the evolution of advantageous strategies, and resulting in faster convergence speed and more stable algorithm performance.

[0143] In summary, this invention discloses a traffic signal policy evaluation and evolution method based on deep reinforcement learning. This method first utilizes multi-agent reinforcement learning to model the traffic signal control policy problem of a traffic road network as a Markov decision process model. Based on the Markov decision process model, a traffic signal policy training network is constructed. First, a multilayer perceptron is used to extract traffic conditions and neighbor information for each intersection. Then, a spatiotemporal fusion graph attention network is combined to dynamically capture the influence weights of neighboring intersections on the target intersection. Based on the policy training network, a traffic signal policy evaluation and evolution algorithm based on a deep Q-network is constructed. The value function and advantage function of the duel network are used to evaluate the advantage of the policy, predict the optimal action, and gradually optimize the policy through parameter gradient updates, thus evolving the advantageous policy. The technical solution disclosed in this invention can effectively cope with complex dynamic traffic flow scenarios at large-scale traffic intersections, achieving efficient traffic signal policy control.

[0144] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0145] In this embodiment of the invention, a traffic signal strategy evaluation and evolution system is provided, comprising:

[0146] The modeling unit is used to model the traffic light control strategy problem of traffic road network as a Markov decision process model using multi-agent reinforcement learning.

[0147] The strategy training network construction unit is used to construct a traffic light strategy training network based on a Markov decision process model. In the traffic light strategy training network, a multilayer perceptron is used to extract the traffic conditions and neighbor information of each intersection, and a spatiotemporal fusion graph attention network is combined to dynamically capture the influence weights of neighboring intersections on the target intersection.

[0148] The strategy evaluation and evolution unit is used to construct a traffic signal strategy evaluation and evolution algorithm based on a deep Q-network, based on a traffic signal strategy training network, to realize the evaluation and evolution of traffic signal strategies. In the traffic signal strategy evaluation and evolution algorithm, the value function and advantage function of the duel network are used to evaluate the advantage of the strategy, predict the best action, and gradually optimize the strategy through parameter gradient updates to carry out the evolution of the advantageous strategy.

[0149] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0151] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for evaluating and evolving traffic signal strategies, characterized in that, Includes the following steps: By using multi-agent reinforcement learning, the traffic light control strategy problem of traffic road network is modeled as a Markov decision process model; Based on the Markov decision process model, a traffic light policy training network is constructed. In the traffic light policy training network, a multilayer perceptron is used to extract the traffic conditions and neighbor information of each intersection, and a spatiotemporal fusion graph attention network is combined to dynamically capture the influence weights of neighboring intersections on the target intersection. Based on the traffic signal policy training network, a traffic signal policy evaluation and evolution algorithm based on deep Q network is constructed to realize the evaluation and evolution of traffic signal policies. In the traffic signal policy evaluation and evolution algorithm, the value function and advantage function of the duel network are used to evaluate the advantage of the policy, predict the best action, and gradually optimize the policy through parameter gradient update to carry out the evolution of the advantageous policy. in, In the traffic light strategy training network, the step of dynamically capturing the influence weights of neighboring intersections on the target intersection using a spatiotemporal fusion graph attention network includes: Traffic condition information is transmitted to a spatiotemporal fusion graph neural network; in this spatiotemporal fusion graph neural network, the temporal module uses a long short-term memory network, and the spatial module uses a graph attention network; wherein... In Long Short-Term Memory (LSTM) networks, the formula for capturing temporal features is: , Characterizing traffic intersections i Traffic conditions; In a graph attention network, the traffic intersections are first identified. i The nearby intersection j After identifying the target traffic intersection i The importance of the strategy is shown in the formula. ;in, and These are the multilayer perceptron parameters for the adjacent intersections and the target intersection; normalization calculation of adjacent intersections j For the target traffic intersection i Importance score, formula is: ； in, It is the influence coefficient. It is the set of intersections adjacent to the target intersection; Each target intersection has five neighboring intersections: the target intersection itself and four intersections geographically closest to it. The characteristics of these neighboring intersections are combined with their respective importance scores using the following formula: ； in, It is near the intersection j The parameters of the multilayer perceptron; and These are the parameters to be trained; A multi-head attention mechanism is employed, executing multiple sets of attention functions with trainable parameters in parallel, and finally averaging them. The formula is as follows: ； in, H It's about the number of heads to focus on. H =5; It is the first h Each attention point has an importance score.

2. The traffic signal strategy evaluation and evolution method according to claim 1, characterized in that, The steps of modeling the traffic light control strategy problem of a traffic road network into a Markov decision process model using multi-agent reinforcement learning include: Assume the system has N Each traffic intersection is considered an intelligent agent, and each agent takes a portion of the observed system state as its observation value; the state of an agent is defined as the agent's state. i In time t Observations It consists of the current signal phase represented by a one-hot vector and the number of vehicles in each entering lane connected to the intersection; the agent i The action is defined as a set of candidate actions. One of the four signal phases This serves as the basis for decision-making in the next time period; among which, the four signal phases are east-west straight, north-south straight, east-west left turn, and north-south left turn; Given the system in time t status and the combined action of multiple corresponding intelligent agents The system is based on state transition probabilities Reaching the next state ; intelligent agent i The reward is defined as ;in, It is a punishment factor. For time t In the time of intelligent agents i Entering lane l The number of vehicles queuing on the road; Use each agent i In the n Action value function at the next iteration The deep neural network approximates the total reward by minimizing the loss. The expression is, ； in, It is the next observation value; It is the value of the next action; This represents the relevant parameters.

3. The traffic signal strategy evaluation and evolution method according to claim 1, characterized in that, In the traffic light policy training network, the steps of extracting traffic conditions and neighbor information for each intersection using a multilayer perceptron include: The network inputs information about the road network and the origin and destination information of each vehicle in the traffic flow. It then extracts the signal phase and vehicle quantity information of each traffic intersection's entry lane as the input layer. A multilayer perceptron consisting of two fully connected layers is used to extract features, as shown in the formula. ;in, and These are the weight matrix and bias vector to be learned.

4. The traffic signal strategy evaluation and evolution method according to claim 1, characterized in that, The traffic signal strategy evaluation evolution algorithm adopts a duel network architecture, consisting of an evaluation network and a target network. The evaluation network and the target network are Q-networks with different parameters but the same structure, each consisting of an input layer, a feature extraction layer, a spatiotemporal fusion graph neural network layer, a value function layer, an advantage function layer, and an output layer. At regular intervals, the parameters of the evaluation network are copied to the target network. The target network generates the predicted Q value and calculates the loss by comparing it with the Q value of the evaluation network, so that the evaluation network can update its gradient. The Q network employs an experience replay mechanism, using a deep neural network to approximate the value function. It stores the trained data in a replay memory pool for subsequent random sampling and training. exist t In time, the duel network generates a predicted Q-value, using the formula: ； in, These are the parameters of the spatial module of the spatiotemporal fusion graph neural network layer, which are shared by the value function layer and the advantage function layer; and These are the parameters of the advantage function layer and the value function layer, respectively. It is a collection of actions; Advantage function layer generation A one-dimensional vector, the value function layer produces a one-dimensional scalar; Features obtained from policy-trained networks The formulas for the value function layer and the advantage function layer of the duel network are as follows: ； ； in, , , , These are the parameters to be learned; The loss function formula for the evolution of the duel network optimization strategy is as follows: ； in, T It is the total number of time steps that affect network updates; N It refers to the number of intersections in the road network. This represents all trainable parameters.

5. A traffic signal strategy evaluation and evolution system, characterized in that, include: The modeling unit is used to model the traffic light control strategy problem of traffic road network as a Markov decision process model using multi-agent reinforcement learning. The strategy training network construction unit is used to construct a traffic light strategy training network based on a Markov decision process model. In the traffic light strategy training network, a multilayer perceptron is used to extract the traffic conditions and neighbor information of each intersection, and a spatiotemporal fusion graph attention network is combined to dynamically capture the influence weights of neighboring intersections on the target intersection. The strategy evaluation and evolution unit is used to construct a traffic signal strategy evaluation and evolution algorithm based on a deep Q-network based on a traffic signal strategy training network, thereby realizing the evaluation and evolution of traffic signal strategies. In the traffic signal strategy evaluation and evolution algorithm, the value function and advantage function of the duel network are used to evaluate the advantage of the strategy, predict the best action, and gradually optimize the strategy through parameter gradient updates to carry out the evolution of the advantageous strategy. The step of dynamically capturing the influence weights of neighboring intersections on the target intersection in the traffic light policy training network, which combines spatiotemporal fusion with a graph attention network, includes: Traffic condition information is transmitted to a spatiotemporal fusion graph neural network; in this spatiotemporal fusion graph neural network, the temporal module uses a long short-term memory network, and the spatial module uses a graph attention network; wherein... In Long Short-Term Memory (LSTM) networks, the formula for capturing temporal features is: , Characterizing traffic intersections i Traffic conditions; In a graph attention network, the traffic intersections are first identified. i The nearby intersection j After identifying the target traffic intersection i The importance of the strategy is shown in the formula. ;in, and These are the multilayer perceptron parameters for the adjacent intersections and the target intersection; normalization calculation of adjacent intersections j For the target traffic intersection i Importance score, formula is: ； in, It is the influence coefficient. It is the set of intersections adjacent to the target intersection; Each target intersection has five neighboring intersections: the target intersection itself and four intersections geographically closest to it. The characteristics of these neighboring intersections are combined with their respective importance scores using the following formula: ； in, It is near the intersection j The parameters of the multilayer perceptron; and These are the parameters to be trained; A multi-head attention mechanism is employed, executing multiple sets of attention functions with trainable parameters in parallel, and finally averaging them. The formula is as follows: ； in, H It's about the number of heads to focus on. H =5; It is the first h Each attention point has an importance score.

6. A traffic signal strategy evaluation and evolution system according to claim 5, characterized in that, In the modeling unit, the steps of modeling the traffic light control strategy problem of the traffic road network into a Markov decision process model using multi-agent reinforcement learning include: Assume the system has N Each traffic intersection is considered an intelligent agent, and each agent takes a portion of the observed system state as its observation value; the state of an agent is defined as the agent's state. i In time t Observations It consists of the current signal phase represented by a one-hot vector and the number of vehicles in each entering lane connected to the intersection; the agent i The action is defined as a set of candidate actions. One of the four signal phases This serves as the basis for decision-making in the next time period; among which, the four signal phases are east-west straight, north-south straight, east-west left turn, and north-south left turn; Given the system in time t status and the combined action of multiple corresponding intelligent agents The system is based on state transition probabilities Reaching the next state ; intelligent agent i The reward is defined as ;in, It is a punishment factor. For time t In the time of intelligent agents i Entering lane l The number of vehicles queuing on the road; Use each agent i In the n Action value function at the next iteration The deep neural network approximates the total reward by minimizing the loss. The expression is, ； in, It is the next observation value; It is the value of the next action; This represents the relevant parameters.

7. A traffic signal strategy evaluation and evolution system according to claim 5, characterized in that, In the traffic light policy training network, the steps of extracting traffic conditions and neighbor information for each intersection using a multilayer perceptron include: The network inputs information about the road network and the origin and destination information of each vehicle in the traffic flow. It then extracts the signal phase and vehicle quantity information of each traffic intersection's entry lane as the input layer. A multilayer perceptron consisting of two fully connected layers is used to extract features, as shown in the formula. ;in, and These are the weight matrix and bias vector to be learned.

8. A traffic signal strategy evaluation and evolution system according to claim 5, characterized in that, In the traffic signal strategy evaluation evolution algorithm described above The architecture of the duel network is adopted, which consists of an evaluation network and a target network. The evaluation network and the target network are Q networks with different parameters but the same structure. Both consist of an input layer, a feature extraction layer, a spatiotemporal fusion graph neural network layer, a value function layer, an advantage function layer and an output layer. At regular intervals, the parameters of the evaluation network are copied to the target network. The target network generates the predicted Q value and calculates the loss by comparing it with the Q value of the evaluation network, so that the evaluation network can update its gradient. The Q network employs an experience replay mechanism, using a deep neural network to approximate the value function. It stores the trained data in a replay memory pool for subsequent random sampling and training. exist t In time, the duel network generates a predicted Q-value, using the formula: ； in, These are the parameters of the spatial module of the spatiotemporal fusion graph neural network layer, which are shared by the value function layer and the advantage function layer; and These are the parameters of the advantage function layer and the value function layer, respectively. It is a collection of actions; Advantage function layer generation A one-dimensional vector, the value function layer produces a one-dimensional scalar; Features obtained from policy-trained networks The formulas for the value function layer and the advantage function layer of the duel network are as follows: ； ； in, , , , These are the parameters to be learned; The loss function formula for the evolution of the duel network optimization strategy is as follows: ； in, T It is the total number of time steps that affect network updates; N It refers to the number of intersections in the road network. This represents all trainable parameters.

Images