Multi-intersection traffic signal control method based on multi-agent adaptive coordination

Abstract

The invention provides a multi-intersection traffic signal control method based on multi-agent adaptive coordination, and the method comprises the steps: obtaining all intersections of a road network, dividing the intersections, judging the intersections needing to deploy agents, constructing a regional traffic signal control frame, designing agent elements in the regional traffic signal control frame, and carrying out the design of the intelligent elements in the regional traffic signal control frame. And designing a multi-agent reinforcement learning algorithm based on value function decomposition, and realizing multi-intersection traffic signal coordination control through a regional traffic signal control framework. According to the method, efficient management of complex dynamic traffic flow is realized, the traffic efficiency of urban traffic can be greatly improved, congestion is reduced, and the method is an innovative solution suitable for a future intelligent traffic system.

Description

Technical Field

[0001] The present invention relates to the technical field of signal control methods for urban traffic management, and in particular to a multi-intersection traffic signal control method based on multi-agent adaptive coordination. Background Art

[0002] With the acceleration of urbanization, traffic congestion has become more serious, especially at intersections, where traffic flows frequently converge. The rationality of traffic signal control has a significant impact on traffic efficiency. Traditional traffic signal control methods mostly rely on fixed durations or preset control logic, which is difficult to adapt to real-time traffic changes. At the same time, the signal control method of a single intersection often appears isolated in the overall traffic flow optimization, and it is difficult to achieve systematic global optimization. In recent years, with the rapid development of vehicle-to-everything (V2X) technology and artificial intelligence, multi-agent technology has been introduced into the field of signal control to optimize the overall traffic flow through coordination between agents. However, the existing technology still has deficiencies in multi-agent coordination, real-time adaptability, etc., and cannot fully utilize the dynamic information in the networked environment, and has failed to completely solve the problem of traffic congestion. Therefore, it is very necessary to design a multi-intersection traffic signal control method based on multi-agent adaptive coordination. Summary of the invention

[0003] In order to overcome the deficiencies of the prior art, an object of the present invention is to provide a multi-intersection traffic signal control method based on multi-agent adaptive coordination.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A multi-intersection traffic signal control method based on multi-agent adaptive coordination, comprising:

[0006] Get all the intersections in the road network, divide them, and determine the intersections where agents need to be deployed;

[0007] Construct regional traffic signal control framework;

[0008] Design the intelligent agent elements in the regional traffic signal control framework;

[0009] A multi-agent reinforcement learning algorithm based on value function decomposition is designed, and coordinated control of traffic signals at multiple intersections is realized through the regional traffic signal control framework.

[0010] Preferably, all intersections of the road network are obtained, divided, and the intersections where the intelligent agents need to be deployed are determined, specifically:

[0011] Obtain the road network area, divide it into sub-areas, and determine the number of sub-areas;

[0012] Determine the optimal number of sub-regions based on evaluating clustering quality;

[0013] Divide the road network area into sub-areas based on the clustering algorithm according to the optimal number of sub-areas;

[0014] Determine the centroid intersection of all sub-regions and use it as the intersection where the agent is deployed.

[0015] Preferably, after determining that an intersection needs to deploy an intelligent agent, a sensing device and an intelligent agent are deployed at the intersection, and the sensing device is connected to the intelligent agent.

[0016] Preferably, a regional traffic signal control framework is constructed, specifically:

[0017] The bottom layer of the regional traffic signal control framework is the perception layer, i.e., the intelligent agent, which is used to obtain local traffic information, make corresponding decision actions, and generate Q function values. The middle layer of the regional traffic signal control framework is the Q network layer, which is used to merge the Q values ​​generated by each intelligent agent. The top layer of the regional traffic signal control framework is the coordination layer, which is used to generate the global optimal Q value for all Q values ​​through a hybrid network, and then feed it back to the local intelligent agent to coordinate and control all intelligent agents in the bottom layer.

[0018] Preferably, the intelligent agent elements in the regional traffic signal control framework are designed, specifically including:

[0019] Design the traffic state space;

[0020] Design the actions of the agent;

[0021] Design the reward and punishment function.

[0022] Preferably, the traffic state space is designed as follows:

[0023] Establish a traffic state space model from three perspectives: vehicle position, vehicle speed, and queue length of each lane;

[0024] The design method of the vehicle position is to assume that there are four roads at an intersection, and use the position matrix P to represent the specific position of the vehicle at the intersection, that is:

[0025]

[0026] Among them, the matrix element P 1 represents the position matrix on the i-th road, which is 1 when the vehicle is in the lane cell and 0 when it is not;

[0027] The vehicle velocity is designed in the same way as the vehicle position, where the velocity matrix is ​​represented by V:

[0028]

[0029] Among them, the matrix element V represents the speed matrix on the i-th road. When the vehicle is in a moving state in the lane cell, the corresponding matrix element value is the speed at this moment; when the vehicle is stationary, it is 0. The specific expression is:

[0030]

[0031] Among them, v car is the speed of the vehicle, v max Set the speed limit for vehicles in the lane;

[0032] Since the queue lengths of different lanes are different, and the directions allowed to pass through each lane are also different, they are distinguished, and the vehicle queue length matrix is:

[0033]

[0034] Among them, the matrix element D ij represents the length of the vehicle queue in the jth lane of the i-th road;

[0035] In summary, the traffic state space is defined as S = [P, V, D].

[0036] Preferably, the agent action is designed as follows:

[0037] The action of the agent is designed for the four-phase intersection. The agent selects the appropriate action from the action set according to the traffic flow state information of each phase. The designed action is to maintain the current signal phase or switch the signal phase according to the traffic flow. The action set can be expressed as:

[0038] in It is the phase of turning right when going straight in the east-west direction. It is the phase of turning left in the east-west direction. It is the phase of turning right when going straight in the north-south direction. is the phase of turning left in the north-south direction, and the time for each execution action is a fixed minimum unit time interval τ.

[0039] Preferably, the reward and punishment function is designed as follows:

[0040] Considering that vehicle delay time has a direct impact on intersection congestion, the model is established using it:

[0041]

[0042] Among them, d j Express the delay time of vehicle j, v j Indicates the speed of the vehicle, N tis the number of vehicles;

[0043] Use the queue length of vehicles to build a model:

[0044] r l =L t -L t+1

[0045] Among them, r l is the change in vehicle queue length between two time steps, L t is the length of the vehicle queue at the current time step t, L t+1 The length of the vehicle queue at the next time step;

[0046] In summary, the final reward and punishment function expression is:

[0047] r=ω 1 r d +ω 2 r l +ω 3 r w

[0048] In the formula, ω 1 ,ω 2 ,ω 3 is the weight coefficient, set according to specific needs, r w is the change in the cumulative waiting time between two adjacent cycles, which is:

[0049]

[0050] In the formula, For vehicle i t During the waiting time of a cycle, N t Represents the total number of vehicles in the period.

[0051] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0052] The present invention provides a multi-intersection traffic signal control method based on multi-agent adaptive coordination, the method comprising: obtaining all intersections of the road network, dividing them, determining the intersections where agents need to be deployed, constructing a regional traffic signal control framework, designing agent elements in the regional traffic signal control framework, designing a multi-agent reinforcement learning algorithm based on value function decomposition, and realizing coordinated control of traffic signals at multiple intersections through the regional traffic signal control framework. The present invention realizes efficient management of complex dynamic traffic flows, can greatly improve the travel efficiency of urban traffic, and reduce congestion, and is an innovative solution to adapt to future intelligent transportation systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0055] Figure 2 It is a schematic diagram of the QMIX algorithm flow;

[0056] Figure 3 It is a schematic diagram of the intelligent agent network structure;

[0057] Figure 4 This is a schematic diagram of the LSTM neural network structure;

[0058] Figure 5 This is a schematic diagram of the DRQN ​​network structure;

[0059] Figure 6 It is a schematic diagram of the hybrid network layer structure;

[0060] Figure 7 This is a schematic diagram of the regional traffic signal control framework structure. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] The purpose of the present invention is to provide a multi-intersection traffic signal control method based on multi-agent adaptive coordination, which realizes efficient management of complex dynamic traffic flows, can greatly improve the traffic efficiency of urban traffic, and reduce congestion. It is an innovative solution to adapt to future intelligent transportation systems.

[0063] Figure 1 A flow chart of a method provided by an embodiment of the present invention, such as Figure 1 As shown, the present invention provides a multi-intersection traffic signal control method based on multi-agent adaptive coordination, comprising:

[0064] Step 100: Obtain all intersections of the road network, divide them, and determine the intersections where intelligent agents need to be deployed;

[0065] Step 200: constructing a regional traffic signal control framework;

[0066] Step 300: Designing intelligent agent elements in the regional traffic signal control framework;

[0067] Step 400: Design a multi-agent reinforcement learning algorithm based on value function decomposition, and realize coordinated control of traffic signals at multiple intersections through a regional traffic signal control framework.

[0068] In step 100, all intersections of the road network are obtained, divided, and the intersections where the intelligent agents need to be deployed are determined, specifically:

[0069] Step 101: Obtain a road network area, divide it, and determine the number interval of sub-areas;

[0070] Step 102: Determine the optimal number of sub-regions based on the evaluation of clustering quality;

[0071] Step 103: Divide the road network area into sub-areas based on the clustering algorithm according to the optimal number of sub-areas;

[0072] Step 104: Determine the centroid intersections of all sub-areas and use them as the intersections for deploying the agents.

[0073] In step 101, a road network area is obtained and divided to determine the number of sub-areas, specifically:

[0074] All intersections in the road network constitute the data set, and each intersection is a sample. The total number of intersections is n. If n is large, it is obviously too much work to traverse from 1 to n to evaluate the optimal number of sub-areas. In addition, since the sub-areas divided by the road network are the units for subsequent signal control, if the number of sub-areas is too small, the number of intersections in the sub-areas will be too large, resulting in poor signal coordination control in the sub-areas; if the number of intersections in the sub-areas is too small, the partitioning scheme cannot fully reflect the similarities and differences between intersections.

[0075] Therefore, by considering the optimal effect and applicable scope of the coordinated control of traffic signals at multiple intersections, the selection interval of the number of sub-regions is determined, and the inappropriate number of clusters is preliminarily excluded to reduce the evaluation workload. The traversal interval r of the number of sub-regions is defined as follows:

[0076]

[0077] Where n represents the total number of intersections in the road network, m represents the optimal number of intersections for multi-intersection traffic signal cooperative control, and MIN m Indicates the minimum number of coordinated intersections, MAX m Represents the maximum number of coordinated intersections, and d is defined as follows:

[0078]

[0079] In addition, S represents the total area of ​​the road network, MAX area Indicates the maximum coordinated area of ​​traffic signal control at multiple intersections. Because the shape of the urban road network is irregular, for areas with sparse roads, there may be a small number of intersections but a large road network. Therefore, it is necessary to consider not only the number of intersections in the sub-area, but also the area of ​​the sub-area to prevent exceeding the maximum coordinated road network area.

[0080] m、MIN m , MAX m , MAX area These thresholds are closely related to the road network structure, traffic flow status and the traffic signal control algorithm used. They can be judged based on the historical traffic flow characteristics of the road network, or determined by relevant processing of the traffic signal control algorithm used.

[0081] In step 102, the optimal number of sub-regions is determined based on the evaluation of clustering quality, specifically:

[0082] After determining the selection interval of the number of sub-regions, the interval is traversed in sequence, and the optimal number of sub-regions is determined by evaluating the clustering quality.

[0083] For the road network sub-division scenario, it is impossible to determine the appropriate traffic flow parameter threshold to evaluate the quality of clustering. In addition, the artificially selected threshold is an ideal benchmark with a certain degree of subjectivity. Therefore, an unsupervised method is used to evaluate the clustering quality by examining the separation between sub-regions and the compactness within the sub-regions. These two situations can be comprehensively considered by introducing the silhouette coefficient.

[0084] In a road network dataset D containing n intersection objects, that is, D = {X 1 ,X 2 ,…,X n}, assume that D is divided into k sub-regions in the current clustering scheme, that is, {C 1 ,…,C k}, intersection X i Belongs to sub-area C i , then X i The silhouette coefficient is defined as follows:

[0085]

[0086] Among them, a i Yes X i The average Euclidean distance to other intersections in the same sub-region, i.e., cohesion, indicates that X i The internal compactness of the sub-region, the smaller the value, the better; b i Yes X iIts closest subregion SC i The average distance of all intersections in the grid, i.e., separation, reflects the degree of separation between the sub-region and other sub-regions. The larger the value, the better. It can be seen that this calculation combines the two factors of cohesion and separation very well. i The nearest subregion SC i Defined as:

[0087]

[0088] Where j≠i, p is a sub-region C i That is, use X i The average distance to all intersections in a sub-area is used as the measure X i After reaching the distance to the sub-area, select the distance from X i The sub-area with the shortest distance is the intersection X i The nearest sub-region of . Find all the intersections {X 1 ,…,X n}, and then take the average value to get the overall silhouette coefficient of the current clustering scheme, which is used to measure the tightness of the current clustering scheme of the road network dataset. The overall silhouette coefficient is calculated as follows:

[0089]

[0090] The value range of the average silhouette coefficient is [-1, 1]. The larger the average silhouette coefficient, the more similar the intersections in the sub-regions are. The greater the difference between the intersections in the sub-regions, the better the clustering effect.

[0091] Therefore, we traverse the sub-region number selection interval r and calculate the overall silhouette coefficient for each scenario.

[0092] The number of clusters with the largest overall silhouette coefficient is the optimal number of clusters, that is, the optimal number of sub-regions.

[0093] In step 103, the road network area is divided into sub-areas based on the clustering algorithm according to the optimal number of sub-areas, specifically:

[0094] For a traffic network with N intersections, the data set is D = {x 1 ,x 2 ,…,x N}, where x 1 x 2 , ..., x N It is the parameter that affects the division of road network sub-areas at each intersection. The similarity measurement of intersections is carried out by comprehensively considering the traffic flow characteristics and spatial characteristics. The intersection influencing parameters are defined as the following three-dimensional data:

[0095] xi ={longitude, latitude, average speed}

[0096] The Euclidean distance is used as the standard to measure the similarity between data objects, which is defined as follows:

[0097]

[0098] K-means algorithm is a common clustering algorithm with simple algorithm and fast convergence speed. However, for the urban traffic road network sub-division scenario, the K-means algorithm has problems such as large influence by isolated points and large differences in the numerical range of longitude, latitude and speed attributes. Therefore, the present invention makes corresponding improvements to some problems of the K-means algorithm to cluster intersections in the road network, thereby realizing a road network partitioning algorithm that considers traffic flow characteristics and spatial location.

[0099] Since the latitude and longitude data of the road network vary widely, while the average speed data varies relatively little, and the K-means algorithm measures their similarity by calculating the distance between data objects, attributes with large values ​​often have a greater impact on the distance between data objects. Therefore, when the algorithm processes two attributes with inconsistent data ranges, it will affect the data similarity.

[0100] In addition, since minimizing the sum of the dissimilarity between the reference point and all objects is the partitioning principle of the K-means algorithm, it is very sensitive to isolated points and noisy data. Objects with particularly large values ​​or outliers may significantly affect the distribution of the data.

[0101] Therefore, in order to reduce the weight influence of the larger value range of the attribute on the similarity of the sample points and avoid the influence of isolated points on the distribution, the present invention adopts "Z-score" to achieve data normalization. Normalize the attribute A, define the normalization of the attribute value v as v', and calculate as follows:

[0102]

[0103] in, is the average value of attribute A, that is:

[0104]

[0105] σ A is the standard deviation of attribute A:

[0106]

[0107] Compared with the "minimum-maximum" normalization, there is no need to determine the maximum value of the road network traffic flow attribute, and the influence of isolated points on the minimum and maximum values ​​is also avoided.

[0108] In the K-means algorithm, the centroid of each cluster is determined based on the average value of the objects in the cluster. However, for intersections in actual traffic networks, it is not appropriate to update the centroid using the average value of the intersection object data in the cluster, because the calculated average value may not have practical significance. Therefore, this paper uses the intersection object located in the center of the cluster as the centroid. Since the center point is selected from the existing objects, it is guaranteed that the selected object must be an intersection in the road network, which is more practical.

[0109] Therefore, based on the determined optimal number of sub-area divisions k, k intersections are randomly selected from the road network intersection data set as the initial centroids. Then the Euclidean distance from each intersection to the k centroids is calculated, and the remaining intersections are assigned to the class represented by the centroid closest to it based on the division principle closest to the centroid. Then the remaining intersections in the data set are repeatedly used to determine the new centroid to ensure that the sum of the Euclidean distances from all intersections in the current sub-area to the centroid is minimized. Subsequently, the new centroid determined by calculation is used to replace the centroid of the previous iteration for clustering to obtain the optimized clustering result until no centroids are replaced;

[0110] Take the centroid as the intersection where the agents are deployed.

[0111] After determining the intersection where the intelligent agent needs to be deployed, a sensor device and an intelligent agent are deployed at the intersection, and the sensor device is connected to the intelligent agent, wherein the data collected by the sensor device is determined according to specific needs, for example, including:

[0112] Vehicle speed: The instantaneous speed of vehicles is obtained through on-board equipment, roadside units (RSUs) and cameras, so as to understand the fluidity of the overall traffic flow;

[0113] Lane occupancy rate: Collects vehicle density information on different lanes to determine the congestion situation of each lane;

[0114] Traffic flow: Count the number of vehicles passing through the intersection per unit time to help determine traffic pressure;

[0115] Vehicle type and direction of travel: Identify the type of vehicle (such as cars, trucks, buses, etc.) passing through the intersection and its direction of travel to help optimize the signal timing for a specific direction or lane;

[0116] The agent mainly includes the following modules:

[0117] Information collection module: receives sensor data for use by the signal optimization module;

[0118] Signal optimization module: A control core based on reinforcement learning algorithms or deep learning models to generate optimal signal control strategies;

[0119] Collaborative communication module: communicates with intelligent entities at other intersections through wireless networks, shares and receives real-time traffic information, and forms perception and response of the overall traffic network;

[0120] Each intelligent agent operates independently at a local intersection, but keeps information synchronized with each other through the communication module, making the overall traffic control system coordinated and consistent.

[0121] The present invention is based on multi-agent reinforcement learning. First, the basic principles of multi-agent reinforcement learning are introduced:

[0122] With the development of computational science and deep learning, multi-agent reinforcement learning in recent years is generally divided into four categories: behavior analysis, communication learning, agent modeling, and collaborative learning. Among them:

[0123] (1) Behavior analysis: The intelligent agents in this type of method learn independently and then apply them to a multi-agent environment.

[0124] (2) Communication learning: In this type of method, the agents communicate explicitly with each other and need to rely on the information transmitted by other agents to make decisions.

[0125] (3) Agent modeling: The agents in this method collaborate or compete by modeling the strategies, goals, and categories of other agents.

[0126] (4) Collaborative learning: The agents in this type of method do not communicate explicitly with each other, and mainly follow the idea of ​​full cooperation and mixed tasks in multi-agent reinforcement learning.

[0127] For a traffic area, there must be multiple intersections with different traffic characteristics, and each intersection is related to each other and is a cooperative whole. Therefore, the present invention will use the collaborative learning method to establish a regional traffic signal coordination control model. The collaborative learning will be specifically described below:

[0128] A collaborative multi-agent task can be described as an octet<S,U,P,r,Z,O,n,γ> Among them, s∈S represents the environmental state, u∈U=U n represents the joint action set of the agents, n represents the number of agents, P represents a set of conditional transition probabilities between environmental states, P(s'|s,u):S×U×S→[0,1], r represents the same reward function shared by all agents, r(s,u):S×U→R, γ represents the reward discount factor, in partially observable scenarios, each agent describes a single observation state z∈Z according to the observation function O(s,a):S×A→Z, and each agent has an action-observation history τ∈T≡(Z×U) *, in this behavior observation history, the condition is a random policy π(u|τ):T×U→[0,1], and the joint policy π has a joint action value function:

[0129]

[0130] Value Decomposition Networks (VDNs) is a method belonging to collaborative learning, because independent learning of agents is easier to expand to multiple agents, but there will be non-stationary problems. VDNs can solve such problems well by focusing on learning and then decomposing the Q value function. Its basic idea is:

[0131] Centrally train a joint Q network. This not only can handle the problems caused by non-stationary environment through centralized training, but also can decouple the complex relationships between agents because it is actually learning the local model of each agent. Finally, after training, each agent is only based on its own local observed Q network, which can achieve decentralized execution. That is, the VDNs algorithm follows the Centralized Training and Distributed Execution (CTDE) framework. Specific description:

[0132] The joint action value function of the distributed agent system can be additively decomposed into the value function of each agent, as shown in the following formula:

[0133]

[0134] in, Depends only on each agent’s local observations. Learned from the Q-learning rule by summing using the joint reward Back propagation gradient, that is is learned implicitly, rather than from any reward specific to agent i, and imposes no constraints The action value is used for any specific reward. A property of this approach is that, although learning is centralized, the learned agents can be deployed independently, since each agent takes actions greedily relative to its local value. Equivalent to using Maximizes the central decision-making agent that selects joint actions.

[0135] For the sake of convenience, take two agents as an example, and the reward value will be added or decomposed between the results of the agent observing the environment state, that is, the global reward function is the sum of the local reward functions and can be expressed as:

[0136] r(s,a)=r 1 (o1 ,a 1 )+r 2 (o 2 ,a 2 )

[0137] Among them (o 1 , a 1 ) and (o 2 , a 2 ) are the observations and actions of agents 1 and 2, respectively. This may be the case, for example, in team games, where agents focus on their own goals but do not necessarily observe the goals of their neighbors. In this case, the state-value function can be expressed as:

[0138]

[0139] That is, the original formula is:

[0140]

[0141] Will Written The above formula can be simplified to

[0142]

[0143] represents the expected future return of Agent 1, whose value changes compared to Agent 2 Influence, more dependent on the environmental state and actions observed by oneself (o 1 , a 1 ), even this cannot be accurately estimated However, by allowing Agent 1 to store its own historical observation information and obtain information from Agent 2 in the communication channel, the error can be reduced in this case. The entire expression can be expressed as:

[0144]

[0145] As can be seen from the above analysis, VDNs adopts a completely decomposed method. The overall value function it outputs is the sum of the value functions of each local agent. The decentralized strategy only depends on the observation information and behavior of each agent itself. It is unnecessary to extract a decentralized strategy that is completely consistent with the centralized strategy. Therefore, the QMIX algorithm improves it on this basis. It only needs to ensure that the global output Q iot Perform a global value function on each agent Q to maximize the action a A set of separate value function maximization actions performed on has the same result, which is expressed as follows:

[0146]

[0147] In this way, each agent a chooses its Q through a greedy strategy a Maximize actions to achieve decentralized decision making.

[0148] In order to apply to more monotone function families, we can tot And each Q a The relationship between is constrained to satisfy monotonicity:

[0149]

[0150] The entire algorithm structure consists of three parts: agent network, hybrid network and super network. Figure 2 shown.

[0151] Among them, the intelligent agent network structure is as follows Figure 3 shown.

[0152] like Figure 3 As shown, each agent will observe the current information independently and the last action information Input, through the LSTM network, where h t is the input returned by the network at the previous time step, i.e. h t =LSTM(h t-1 ,a t )). Each agent has an independent value function Q a As output. In the network setting, the agents use LSTM networks to use all their action-observation history information to represent the current state. The entire agent network uses the DRON network structure and applies weight sharing between agent networks. In order to make different agents perform different actions, the agent IDs are also added to the observations.

[0153] Among them, the entire intelligent agent network structure involves two key points: one is the LSTM network structure, and the other is the DRON structure.

[0154] (1) LSTM network structure

[0155] LSTM network is a variant of RNN network, which is mainly used to solve the gradient disappearance problem. LSTM network stores information in memory cells. Its typical LSTM neural network structure is as follows: Figure 4 shown.

[0156] The LSTM neural network consists of three structures: forget gate, input gate, and output gate. Among them: the forget gate is responsible for deciding which information should not be retained in the cell state; the input gate is responsible for deciding which information should be stored in the memory; the output gate is responsible for deciding which information to display from the cell state at time t.

[0157] (2) DRON network structure

[0158] The DRON network is very similar to the DON network discussed in the previous chapter, except that the first convolutional fully connected layer is replaced by an LSTM network. Figure 5 shown.

[0159] like Figure 5 As shown in the figure, the traffic state is used as the input of the convolutional layer, which performs convolution operations on the traffic state and generates a feature map. The resulting feature map is then passed to the LSTM layer, which retains important information about the previous traffic state and updates its memory according to time. After passing through a fully connected layer, the Q value is output. Therefore, unlike DON, there is no need to directly estimate Q(s t ,a t ), but to estimate Q(h t ,a t ). Since the LSTM network is used, the network is trained through time-based back propagation. Unlike DQN, which avoids experience association, DRON saves the entire scene in the experience cache and finds the sampled n time steps from the random batch scene. In this way, it can adapt to randomness and obtain another experience. The structure diagram of the hybrid network is as follows Figure 6 As shown in the figure, the hybrid network takes the output of the agent network as input and performs monotonous mixing to obtain Q tot The weights of the hybrid network are generated by separate super-networks, each of which takes the global state S as input to obtain additional environmental state information and generates weights for the hybrid network layers. Each super-network consists of a single linear layer with a ReLU activation function to ensure that the weights of the hybrid network are non-negative. The global state is passed through the super-networks instead of directly to the hybrid network. This is because Q tot This allows for additional state information to be captured in a non-monotonic manner, so some state functions are overly constrained when passed through a monotone network along with each agent’s value function. Instead, the use of a hypernetwork allows for the weights of the monotone network on state S to be adjusted in an arbitrary way to integrate the full state S into the joint action value estimate as closely as possible.

[0160] In step 200, a regional traffic signal control framework is constructed, specifically:

[0161] Among them, the regional traffic signal control framework is as follows Figure 7As shown, the bottom layer of the regional traffic signal control framework is the perception layer, i.e., the intelligent agent, which is used to obtain local traffic information, make corresponding decision actions, and generate Q function values. The middle layer of the regional traffic signal control framework is the Q network layer, which is used to merge the Q values ​​generated by each intelligent agent. The top layer of the regional traffic signal control framework is the coordination layer, which is used to generate the global optimal Q value for all Q values ​​through the hybrid network, and then feed it back to the local intelligent agent to coordinate and control all the intelligent agents in the bottom layer.

[0162] In this framework, intelligent agents are deployed at the centroid intersection of each sub-area. The centroid intersection intelligent agents can perceive the surrounding traffic status information, and each centroid intersection intelligent agent makes decisions within discrete time intervals based on the perceived traffic status information. This framework is a distributed structure, and each intelligent agent can communicate with the centroid intersection of its adjacent sub-area to publish its current status information, ensuring that the intelligent agents can coordinate with each other, thereby achieving the goal of stabilizing the entire system.

[0163] In step 300, the intelligent agent elements in the regional traffic signal control framework are designed, including:

[0164] Step 301: Designing the traffic state space;

[0165] Step 302: Designing the actions of the intelligent agent;

[0166] Step 303: Design the reward and punishment function.

[0167] In step 301, the traffic state space is designed, specifically:

[0168] Establish a traffic state space model from three perspectives: vehicle position, vehicle speed, and queue length of each lane;

[0169] The design method of the vehicle position is to assume that there are four roads at an intersection, and use the position matrix P to represent the specific position of the vehicle at the intersection, that is:

[0170]

[0171] Among them, the matrix element P 1 represents the position matrix on the i-th road, which is 1 when the vehicle is in the lane cell and 0 when it is not;

[0172] The vehicle velocity is designed in the same way as the vehicle position, where the velocity matrix is ​​represented by V:

[0173]

[0174] Among them, the matrix element V represents the speed matrix on the i-th road. When the vehicle is in a moving state in the lane cell, the corresponding matrix element value is the speed at this moment; when the vehicle is stationary, it is 0. The specific expression is:

[0175]

[0176] Among them, v car is the speed of the vehicle, v max Set the speed limit for vehicles in the lane;

[0177] Since the queue lengths of different lanes are different, and the directions allowed to pass through each lane are also different, they are distinguished, and the vehicle queue length matrix is:

[0178]

[0179] Among them, the matrix element D ij represents the length of the vehicle queue in the jth lane of the i-th road;

[0180] In summary, the traffic state space is defined as S = [P, V, D].

[0181] In step 302, the agent action is designed, specifically:

[0182] The action of the agent is designed for the four-phase intersection. The agent selects the appropriate action from the action set according to the traffic flow state information of each phase. The designed action is to maintain the current signal phase or switch the signal phase according to the traffic flow. The action set can be expressed as:

[0183] in It is the phase of turning right when going straight in the east-west direction. It is the phase of turning left in the east-west direction. It is the phase of turning right when going straight in the north-south direction. is the phase of turning left in the north-south direction, and the time for each execution action is a fixed minimum unit time interval τ.

[0184] In step 303, the reward and punishment function is designed, specifically:

[0185] Considering that vehicle delay time has a direct impact on intersection congestion, the model is established using it:

[0186]

[0187] Among them, d j Express the delay time of vehicle j, v j Indicates the speed of the vehicle, N t is the number of vehicles;

[0188] Use the queue length of vehicles to build a model:

[0189] r l =L t -L t+1

[0190] Among them, r l is the change in vehicle queue length between two time steps, L t is the length of the vehicle queue at the current time step t, L t+1 The length of the vehicle queue at the next time step;

[0191] In summary, the final reward and punishment function expression is:

[0192] r=ω 1 r d +ω 2 r l +ω 3 r w

[0193] In the formula, ω 1 ,ω 2 ,ω 3 is the weight coefficient, set according to specific needs, r w is the change in the cumulative waiting time between two adjacent cycles, which is:

[0194]

[0195] In the formula, For vehicle i t During the waiting time of a cycle, N t Indicates the total number of vehicles in the cycle

[0196] In step 400, a multi-agent reinforcement learning algorithm based on value function decomposition is designed to realize coordinated control of traffic signals at multiple intersections through a regional traffic signal control framework, specifically:

[0197] The above article introduces the multi-agent reinforcement learning algorithm based on value function decomposition. Based on this algorithm, the coordinated control of traffic signals at multiple intersections is realized through the regional traffic signal control framework, in which no additional information exchange and communication channels are required between each agent. At the same time, in order to prevent the traffic environment from being unstable due to each agent not understanding the strategies of other agents, the global traffic environment information is fed back to the hybrid network, so that each agent can use the global traffic status and actions during the training process, so that each agent can adjust its own strategy according to the strategies of other agents, thereby achieving the global optimal strategy;

[0198] The algorithm is as follows: first, the weights, sample pool and experience replay cache of the current network and the target network are initialized; then, in each simulation scenario, the local agent selects an action according to the update strategy of the current traffic environment state, and each agent receives the current environment state and the new environment state after executing the action. l+1 feedback and update the hidden parameters of the LSTM neural network; then the local agent will use the current learning experience 1 , a 1 , R 1 , O 2 ,,a 2 , R 2 .....} is stored in the experience replay cache, and randomly sampled from it to update the local learning strategy; then the strategy of maximizing the Q value of each local agent is trained end-to-end through the hybrid network to minimize the loss function; finally, the weights of the target network are updated according to the main network.

[0199] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0200] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A multi-intersection traffic signal control method based on multi-agent adaptive coordination, characterized in that: include: Get all the intersections in the road network, divide them, and determine the intersections where agents need to be deployed; Construct regional traffic signal control framework; Design the intelligent agent elements in the regional traffic signal control framework; A multi-agent reinforcement learning algorithm based on value function decomposition is designed, and coordinated control of traffic signals at multiple intersections is realized through the regional traffic signal control framework.

2. The multi-intersection traffic signal control method based on multi-agent adaptive coordination according to claim 1 is characterized in that: Get all the intersections of the road network, divide them, and determine the intersections where the agent needs to be deployed, specifically: Obtain the road network area, divide it into sub-areas, and determine the number of sub-areas; Determine the optimal number of sub-regions based on evaluating clustering quality; Divide the road network area into sub-areas based on the clustering algorithm according to the optimal number of sub-areas; Determine the centroid intersection of all sub-regions and use it as the intersection where the agent is deployed.

3. The multi-intersection traffic signal control method based on multi-agent adaptive coordination according to claim 1 is characterized in that: After determining the intersection where the intelligent agent needs to be deployed, a sensor device and an intelligent agent are deployed at the intersection, and the sensor device is connected to the intelligent agent.

4. The multi-intersection traffic signal control method based on multi-agent adaptive coordination according to claim 1 is characterized in that: Construct a regional traffic signal control framework, specifically: The bottom layer of the regional traffic signal control framework is the perception layer, i.e., the intelligent agent, which is used to obtain local traffic information, make corresponding decision actions, and generate Q function values. The middle layer of the regional traffic signal control framework is the Q network layer, which is used to merge the Q values ​​generated by each intelligent agent. The top layer of the regional traffic signal control framework is the coordination layer, which is used to generate the global optimal Q value for all Q values ​​through a hybrid network, and then feed it back to the local intelligent agent to coordinate and control all intelligent agents in the bottom layer.

5. The multi-intersection traffic signal control method based on multi-agent adaptive coordination according to claim 1 is characterized in that: Design the intelligent agent elements in the regional traffic signal control framework, including: Design the traffic state space; Design the actions of the agent; Design the reward and punishment function.

6. The multi-intersection traffic signal control method based on multi-agent adaptive coordination according to claim 5 is characterized in that: The traffic state space is designed as follows: Establish a traffic state space model from three perspectives: vehicle position, vehicle speed, and queue length of each lane; The design method of the vehicle position is to assume that there are four roads at an intersection, and use the position matrix P to represent the specific position of the vehicle at the intersection, that is: Among them, the matrix element P1 represents the position matrix on the i-th road, which is 1 when the vehicle is in the lane cell and 0 when it is not; The vehicle velocity is designed in the same way as the vehicle position, where the velocity matrix is ​​represented by V: Among them, the matrix element V represents the speed matrix on the i-th road. When the vehicle is in a moving state in the lane cell, the corresponding matrix element value is the speed at that moment; when the vehicle is stationary, it is 0. The specific expression is: Among them, v car is the speed of the vehicle, v max Set the speed limit for vehicles in the lane; Since the queue lengths of different lanes are different, and the directions allowed to pass through each lane are also different, they are distinguished, and the vehicle queue length matrix is: Among them, the matrix element D ij represents the length of the vehicle queue in the jth lane of the i-th road; In summary, the traffic state space is defined as S = [P, V, D].

7. The multi-intersection traffic signal control method based on multi-agent adaptive coordination according to claim 5 is characterized in that: Design the agent's actions, specifically: The action of the agent is designed for the four-phase intersection. The agent selects the appropriate action from the action set according to the traffic flow state information of each phase. The designed action is to maintain the current signal phase or switch the signal phase according to the traffic flow. The action set can be expressed as: in It is the phase of turning right when going straight in the east-west direction. It is the phase of turning left in the east-west direction. It is the phase of turning right when going straight in the north-south direction. is the phase of turning left in the north-south direction, and the time for each execution action is a fixed minimum unit time interval τ.

8. The multi-intersection traffic signal control method based on multi-agent adaptive coordination according to claim 5 is characterized in that: The reward and punishment function is designed as follows: Considering that vehicle delay time has a direct impact on intersection congestion, the model is established using it: Among them, d j Express the delay time of vehicle j, b j Indicates the speed of vehicles passing, N t is the number of vehicles; Use the queue length of vehicles to build a model: r l =L t -L t+1 Among them, r l is the change in vehicle queue length between two time steps, L t is the length of the vehicle queue at the current time step t, L t+1 The length of the vehicle queue at the next time step; In summary, the final reward and punishment function expression is: r=ω1r d +ω2r l +ω3r w In the formula, ω1, ω2, and ω3 are weight coefficients, which are set according to specific needs. w is the change in the cumulative waiting time between two adjacent cycles, which is: In the formula, For vehicle i t During the waiting time of a cycle, N t Represents the total number of vehicles in the period.

Images