Multi-intersection traffic signal control method based on multi-agent adaptive coordination

Abstract

The application provides a kind of multi-intersection traffic signal control method based on multi-agent adaptive coordination, which comprises: obtaining all intersections of road network, dividing, judging the intersection needing to deploy agent, constructing regional traffic signal control framework, designing agent element in regional traffic signal control framework, designing multi-agent reinforcement learning algorithm based on value function decomposition, realizing multi-intersection traffic signal coordination control through regional traffic signal control framework.The application realizes efficient management of complex dynamic traffic flow, can greatly improve the traffic efficiency of urban traffic, reduce congestion, and is an innovative solution for future intelligent transportation system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of signal control methods for urban traffic management, in particular to a multi-intersection traffic signal control method based on multi-agent adaptive coordination. BACKGROUND

[0002] With the acceleration of urbanization, traffic congestion problems are becoming more and more serious, especially at intersections, where traffic converges frequently, and the rationality of traffic signal control has a significant impact on traffic efficiency. Traditional traffic signal control methods mostly rely on fixed time or preset control logic, which is difficult to adapt to real-time traffic changes. At the same time, the signal control method of a single intersection is often isolated in the overall traffic flow optimization, and it is difficult to achieve systematic global optimization. In recent years, with the rapid development of vehicle-to-everything (V2X) and artificial intelligence, multi-agent technology has been introduced into the field of signal control, optimizing overall traffic flow through coordination between agents. However, existing technologies still have deficiencies in multi-agent coordination, real-time adaptability, etc., and cannot fully utilize dynamic information in a networked environment, failing to completely solve the problem of traffic congestion. Therefore, it is necessary to design a multi-intersection traffic signal control method based on multi-agent adaptive coordination. SUMMARY

[0003] In order to overcome the deficiencies of the prior art, the purpose of the present application is to provide a multi-intersection traffic signal control method based on multi-agent adaptive coordination.

[0004] To achieve the above purpose, the present application provides the following solutions:

[0005] A multi-intersection traffic signal control method based on multi-agent adaptive coordination, comprising:

[0006] Obtaining all intersections of a road network, dividing them, and determining the intersections that need to deploy agents;

[0007] Constructing a regional traffic signal control framework;

[0008] Designing agent elements in the regional traffic signal control framework;

[0009] Designing a multi-agent reinforcement learning algorithm based on value function decomposition to realize multi-intersection traffic signal coordination control through the regional traffic signal control framework.

[0010] Preferably, obtaining all intersections of a road network, dividing them, and determining the intersections that need to deploy agents, specifically:

[0011] Obtaining a road network region, dividing it, and determining the number of sub-regions;

[0012] Determine the optimal number of sub-regions based on the evaluation of clustering quality;

[0013] Divide the road network region into sub-regions based on the clustering algorithm according to the optimal number of sub-regions;

[0014] Determine the centroid intersection of all sub-regions as the intersection for deploying intelligent agents.

[0015] Preferably, after determining the intersection that needs to deploy intelligent agents, the sensing device and intelligent agent are deployed at the intersection, and the sensing device is connected to the intelligent agent.

[0016] Preferably, a regional traffic signal control framework is constructed, specifically:

[0017] The lowest layer of the regional traffic signal control framework is the perception layer, i.e., the intelligent agent, which is used to obtain local traffic information and make corresponding decision actions to generate Q function values. The middle layer of the regional traffic signal control framework is the Q network layer, which is used to combine the Q values generated by each intelligent agent. The highest layer of the regional traffic signal control framework is the coordination layer, which is used to generate a globally optimal Q value through a hybrid network for all Q values, and then feedback it to the local intelligent agent to coordinate and control all intelligent agents in the lowest layer.

[0018] Preferably, the intelligent agent element in the regional traffic signal control framework is designed, specifically including:

[0019] Design the traffic state space;

[0020] Design the intelligent agent action;

[0021] Design the reward and punishment function.

[0022] Preferably, the traffic state space is designed, specifically:

[0023] A traffic state space model is established from three aspects: the position of the vehicle, the speed of the vehicle, and the queue length of each lane;

[0024] The design method of the vehicle position is to set a crossroads with four roads, and use the position matrix P to represent the specific position of the vehicle at the crossroads, i.e.:

[0025]

[0026] Wherein, the matrix element P1 represents the position matrix on the i-th road, and is 1 when the vehicle is in the lane cell, and is 0 otherwise;

[0027] The design method of the vehicle speed is consistent with that of the vehicle position, wherein the speed matrix is represented by V:

[0028]

[0029] wherein the matrix element V represents the speed matrix on the ith road, when the vehicle is in the driving state in the lane cell, the corresponding matrix element value is the speed at this moment; when the vehicle is in the static state, it is 0, and the specific expression is:

[0030]

[0031] wherein v car is the speed of the vehicle, v max is the speed limit of the lane for the vehicle;

[0032] Since the queue lengths of different lanes are different, and the directions allowed to pass through each lane are also different, they are distinguished, and the vehicle queue length matrix is:

[0033]

[0034] wherein the matrix element D ij represents the vehicle queue length of the jth lane of the ith road;

[0035] In summary, the traffic state space is defined as S = [P, V, D].

[0036] Preferably, the agent action is designed, specifically:

[0037] The action of the agent is designed for the four-phase intersection, and the agent selects the appropriate action from the behavior set according to the traffic flow state information of each phase, wherein the designed behavior is to keep the current signal phase or switch the signal phase according to the traffic flow, and the behavior set can be expressed as:

[0038] wherein is the east-west direction straight right turn phase, is the east-west direction left turn phase, is the south-north direction straight right turn phase, is the south-north direction left turn phase, and the time of each execution action is a fixed minimum unit time interval τ.

[0039] Preferably, the reward and punishment function is designed, specifically:

[0040] Considering that the vehicle delay time has a direct impact on the congestion of the intersection, a model is established by using the same:

[0041]

[0042] wherein d j represents the delay time of vehicle j, v j represents the speed of the vehicle, and N t is the number of vehicles;

[0043] A model is established using the queue length of the vehicle:

[0044] r l = L t - L t+1

[0045] Wherein, r l is the change of vehicle queue length between two time steps, L t is the vehicle queue length at the current time step t, L t+1 is the vehicle queue length at the next time step;

[0046] Therefore, the final reward and punishment function expression is obtained:

[0047] r = ω1r d + ω2r l + ω3r w

[0048] In the formula, ω1, ω2, ω3 are weight coefficients, which are set according to specific requirements, r w is the change of accumulated waiting time between two adjacent periods, which is:

[0049]

[0050] In the formula, is the waiting time of vehicle i t in a period, N t represents the total number of vehicles in the period.

[0051] According to the specific embodiments provided by the application, the following technical effects are disclosed:

[0052] The application provides a multi-intersection traffic signal control method based on multi-agent adaptive coordination, which comprises the following steps: acquiring all intersections of a road network, dividing the intersections, judging intersections needing to deploy agents, constructing a regional traffic signal control framework, designing agent elements in the regional traffic signal control framework, designing a multi-agent reinforcement learning algorithm based on value function decomposition, and realizing multi-intersection traffic signal coordination control through the regional traffic signal control framework. The application realizes efficient management of complex dynamic traffic flow, can greatly improve the passing efficiency of urban traffic, reduces congestion, and is an innovative solution for adapting to future intelligent transportation systems. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0054] Figure 1 The method flowchart provided for the embodiments of the present application is as follows

[0055] Figure 2 The QMIX algorithm flowchart is as follows

[0056] Figure 3 The intelligent agent network structure diagram is as follows

[0057] Figure 4 The LSTM neural network structure diagram is as follows

[0058] Figure 5 The DRQN network structure diagram is as follows

[0059] Figure 6 The hybrid network layer structure diagram is as follows

[0060] Figure 7 The regional traffic signal control framework structure diagram is as follows. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0062] The purpose of the present application is to provide a multi-intersection traffic signal control method based on multi-agent adaptive coordination, which realizes efficient management of complex dynamic traffic flow, greatly improves the passing efficiency of urban traffic, reduces congestion, and is an innovative solution for future intelligent transportation systems.

[0063] Figure 1 The method flowchart provided for the embodiments of the present application is as follows Figure 1 As shown in the figure, the present application provides a multi-intersection traffic signal control method based on multi-agent adaptive coordination, which comprises:

[0064] Step 100: Obtain all intersections of the road network, divide them, and determine the intersections that need to deploy intelligent agents.

[0065] Step 200: Construct a regional traffic signal control framework;

[0066] Step 300: Design an agent element in the regional traffic signal control framework;

[0067] Step 400: Design a multi-agent reinforcement learning algorithm based on value function decomposition to realize multi-intersection traffic signal coordination control through the regional traffic signal control framework.

[0068] In step 100, all intersections of the road network are obtained, and they are divided to determine the intersections where agents need to be deployed, specifically:

[0069] Step 101: Obtain the road network region and divide it to determine the number interval of sub-regions;

[0070] Step 102: Determine the optimal number of sub-regions based on the evaluation of clustering quality;

[0071] Step 103: Divide the road network region into sub-regions based on the clustering algorithm according to the optimal number of sub-regions;

[0072] Step 104: Determine the centroid intersection of all sub-regions as the intersection where the agent is deployed.

[0073] In step 101, the road network region is obtained and divided to determine the number interval of sub-regions, specifically:

[0074] All intersections in the road network form a data set, and each intersection is a sample. If the total number of intersections n is large, it is obviously too much work to evaluate the optimal number of sub-regions from 1 to n. In addition, since the sub-regions divided by the road network are the units for subsequent signal control, if the number of sub-regions is too small, the number of intersections in the sub-region will be too large, resulting in poor signal coordination control effect in the sub-region. The number of intersections in the sub-region is too small, and the partitioning scheme cannot fully reflect the similarity and difference between intersections.

[0075] Therefore, by considering the optimal effect and applicable range of multi-intersection traffic signal coordination control, the selection interval of the number of sub-regions is determined to preliminarily exclude inappropriate clustering numbers and reduce the evaluation workload. The traversal interval r of the number of sub-regions is defined as follows:

[0076]

[0077] where n represents the total number of intersections in the road network, m represents the optimal number of intersections for multi-intersection traffic signal coordination control, MIN m represents the minimum number of coordinated intersections, and MAX m represents the maximum number of coordinated intersections, and d is defined as follows:

[0078]

[0079] In addition, S represents the total area of the road network, MAX area represents the maximum coordination area of the intersection traffic signal control. Because the shape of the urban road network is irregular, for the area with sparse roads, there can be a case that the number of intersections is small but the range of the road network is large. Therefore, not only the limit of the number of intersections in the sub-area needs to be considered, but also the area of the sub-area needs to be limited to prevent exceeding the maximum coordination road network area.

[0080] m, MIN m , MAX m , MAX area These thresholds are closely related to the road network structure, traffic flow state and the traffic signal control algorithm used, which can be determined according to the historical traffic flow characteristics of the road network, or can be determined by related processing of the traffic signal control algorithm used.

[0081] In step 102, the number of optimal sub-areas is determined based on the evaluation of the clustering quality, specifically:

[0082] After determining the selection interval of the number of sub-areas, the interval is traversed in sequence, and the optimal number of sub-areas is determined by evaluating the clustering quality.

[0083] For the road network sub-area division scenario, it is not possible to determine the appropriate traffic flow parameter threshold to evaluate the clustering quality, in addition, the threshold selected by human is an ideal benchmark, which is subjective. Therefore, an unsupervised method is adopted, and the clustering quality is evaluated by investigating the separation between sub-areas and the compactness within sub-areas, and the two cases can be considered comprehensively by introducing the silhouette coefficient.

[0084] In a road network data set D containing n intersection objects, that is, D = {X1, X2, …, X n}, it is assumed that D is divided into k sub-areas in the current clustering scheme, that is, {C1, …, C k}, the intersection X i belongs to the sub-area C i , then the silhouette coefficient of X i is defined as follows:

[0085]

[0086] Where a i is the average Euclidean distance of X i and other intersections in the same sub-area, that is, the cohesion degree, which indicates the internal compactness of the sub-area to which X i belongs, the smaller the value is, the better; b i is the Euclidean distance between X i and its nearest sub-area SC iThe average distance of all intersections in a sub-region, i.e. the separation degree, reflects the separation degree of the sub-region from other sub-regions, and the greater the value is, the better. It can be seen that the calculation well combines the cohesion degree and the separation degree. The nearest sub-region SC i of an intersection X i is defined as:

[0087]

[0088] where j≠i, and p is an intersection in a certain sub-region C i . That is, the average distance of all intersections of a certain sub-region to an intersection X i is used as a measure of the distance of the intersection X i to the sub-region, and the sub-region with the shortest distance to the intersection X i is selected as the nearest sub-region of the intersection X i . After the contour coefficients of all intersections {X1,…,X n} in the road network are calculated, the overall contour coefficient of the current clustering scheme is obtained by averaging the contour coefficients, which is used to measure the tightness of the current clustering scheme of the road network data set. The overall contour coefficient is calculated as follows:

[0089]

[0090] The average contour coefficient ranges from -1 to 1, and the greater the average contour coefficient is, the more similar the intersections in the sub-region are, and the greater the difference between the intersections in different sub-regions is, and the better the clustering effect is.

[0091] Therefore, the number of sub-regions is selected in the interval r, and the overall contour coefficient in each scenario is calculated,

[0092] The clustering number with the maximum overall contour coefficient is the optimal clustering number, i.e. the optimal number of sub-regions.

[0093] In step 103, the road network region is divided into sub-regions based on the optimal number of sub-regions and the clustering algorithm, specifically:

[0094] For a traffic road network containing N intersections, the data set is D={x1,x2,…,x N}, where x1x2,..., x N is a parameter that affects the division of the road network sub-region for each intersection, which comprehensively considers the traffic flow characteristics and spatial characteristics to measure the similarity of the intersections. The intersection influence parameter is defined as the following three-dimensional data:

[0095] x i ={longitude, latitude, average speed}

[0096] The Euclidean distance is used as a measure of the similarity between data objects, which is defined as follows:

[0097]

[0098] The K-means algorithm is a common clustering algorithm, and the algorithm is simple and has fast convergence speed. However, for the scene of dividing the sub-area of the urban traffic network, the K-means algorithm has problems such as being greatly affected by isolated points, and great differences in latitude and longitude and speed attribute value range. Therefore, the application improves the K-means algorithm in view of some problems, so as to cluster the intersections in the road network, thereby realizing the road network partition algorithm considering the traffic flow characteristics and spatial position.

[0099] Since the latitude and longitude data of the road network has a large change range, and the average speed data has a relatively small change range, and the K-means algorithm measures the similarity of data objects by calculating the distance between them, therefore, the attribute with a large value has a great influence on the distance between data objects. Therefore, when the algorithm processes two attributes with inconsistent data ranges, it will affect the similarity of data.

[0100] In addition, since the sum of the dissimilarity between the reference point and all objects is the division principle of the K-means algorithm, it is sensitive to isolated points and noise data, and objects with particularly large values or outliers may significantly affect the distribution of data.

[0101] Therefore, in order to reduce the weight influence of the large value range of the attribute on the similarity of the sample points, and avoid the influence of the isolated points on the distribution, the application realizes data normalization by using "Z-score". The normalization of attribute A is defined as v', and the calculation is as follows:

[0102]

[0103] Among them, is the average value of attribute A, that is:

[0104]

[0105] σ A is the standard deviation of attribute A:

[0106]

[0107] Compared with the "min-max" normalization, it is not necessary to determine the extreme value of the traffic flow attribute of the road network, and the influence of the isolated point on the minimum and maximum value is also avoided.

[0108] In the K-means algorithm, the cluster centroid of each cluster is determined based on the average of the objects in the cluster. However, for intersections in an actual traffic network, it is not appropriate to update the centroid using the average of the intersection object data in the cluster, as the calculated average may not have practical significance. Therefore, this thesis uses the most central intersection object in the cluster as the centroid. Since the central point is selected from the existing objects, it is guaranteed that the selected object is an intersection in the road network, making it more practical.

[0109] Therefore, based on the determined optimal number of sub-regions k, k intersections are randomly selected from the intersection data set of the road network as initial centroids. Then, the Euclidean distance of each intersection to the k centroids is calculated, and the remaining intersections are assigned to the class represented by the nearest centroid based on the nearest partition principle. Then, new centroids are determined using the remaining intersections in the data set to ensure that the sum of the Euclidean distances of all intersections in the current sub-region to the centroid is minimized. Subsequently, the newly determined centroids are used to replace the centroids of the previous iteration for clustering, resulting in an optimized clustering result, until no centroid is replaced;

[0110] The centroid is used as the intersection where the intelligent agent is deployed.

[0111] After determining the intersection where the intelligent agent needs to be deployed, a sensor device and an intelligent agent are deployed at the intersection, and the sensor device is connected to the intelligent agent. The data collected by the sensor device is determined according to specific needs, such as:

[0112] Vehicle speed: The instantaneous speed of vehicles is obtained through on-board devices, roadside units (RSUs), and cameras, etc., to facilitate understanding of the flow of overall traffic flow;

[0113] Lane occupancy rate: The density information of vehicles on different lanes is collected to determine the congestion of each lane;

[0114] Vehicle flow: The number of vehicles passing through the intersection per unit time is counted to help determine traffic pressure;

[0115] Vehicle type and driving direction: Identify the type of vehicles (such as cars, trucks, buses, etc.) and their driving direction passing through the intersection to help optimize signal timing for specific directions or lanes;

[0116] The intelligent agent mainly includes the following modules:

[0117] Information collection module: receives sensor data for use by the signal optimization module;

[0118] Signal optimization module: control core based on reinforcement learning algorithm or deep learning model, generates the optimal signal control strategy;

[0119] Cooperative communication module: communicate with other intersection agents through wireless network, share and receive real-time traffic information, form the perception and response of the overall traffic network;

[0120] Each agent independently runs on the local intersection, but keeps information synchronization with each other through the communication module, so that the overall traffic control system has coordination and consistency.

[0121] The application is based on multi-agent reinforcement learning implementation, first, the basic principles of multi-agent reinforcement learning are introduced:

[0122] With the development of computer science and deep learning, multi-agent reinforcement learning in recent years is generally divided into four categories: behavior analysis, communication learning, agent modeling and collaborative learning.

[0123] (1) Behavior analysis, the agent of this method is independently learned, and then applied to the multi-agent environment.

[0124] (2) Communication learning, the agents of this method communicate explicitly, and need to rely on the information transmitted by other agents to make decisions.

[0125] (3) Agent modeling, the agent of this method models the strategy, goal, category and the like of other agents, and cooperates or competes

[0126] (4) Collaborative learning, the agents of this method do not communicate explicitly, and mainly follow the complete cooperation and mixed task idea in multi-agent reinforcement learning.

[0127] For a traffic area, there are multiple intersections with different traffic characteristics, and each intersection is associated with each other, which is a cooperative whole. Therefore, the cooperative learning method is used to establish a regional traffic signal coordination control model. The cooperative learning will be described below:

[0128] A cooperative multi-agent task can be described as an eight-tuple <S, U, P, r, Z, O, n, γ>, wherein s∈S represents the environment state, u∈U=U n represents the joint action set of the agent, n represents the number of agents, P represents a set of conditional transition probabilities between environment states, P(s'|s,u):S×U×S→[0,1], r represents the same reward function shared by all agents, r(s,u):S×U→R, γ represents the reward discount factor, in the partially observable scene, each agent describes the individual observation state z∈Z according to the observation function O(s,a):S×A→Z, each agent has an action-observation history τ∈T≡(Z×U) *In this behavioral observation history, the condition is a random strategy π(u|τ): T x U→ [0,1], and the joint strategy π has a joint action value function:

[0129]

[0130] Value Decomposition Networks (VDNs) is a method belonging to cooperative learning, because it is easier to extend to multi-agent for independent learning, but there is a non-stationary problem, and VDNs can solve this problem by centralized learning and decomposing Q value function. The basic idea is:

[0131] A joint Q network is trained centrally, which can not only deal with the problem of non-stationary environment by centralized training, but also decouple the complex relationship between agents because it is actually learning the local model of each agent. Finally, since each agent only bases on its own local observation of the Q network after training, it can be executed in a decentralized manner. That is, the VDN algorithm follows the Centralized Training and Distributed Execution (CTDE) framework. Specifically:

[0132] The joint action value function of the distributed agent system can be additively decomposed into the value function of each agent, as shown in the following formula:

[0133]

[0134] Wherein, Only depends on the local observation of each agent. Learn Backpropagation gradient, that is, Is learned implicitly, not from any reward specific to agent i, and no constraints Action value for any specific reward. One feature of this method is that although learning needs to be centralized, the learned agents can be deployed independently because each agent takes a greedy strategy to perform actions relative to its local value. Equivalent to selecting a central decision-making agent for joint action by maximizing .

[0135] For convenience of illustration, take two agents as an example, and the reward value is additive or decomposed between the results of the agent observing the environment state, that is, the global return function is the sum of the local return function, which can be expressed as:

[0136] r(s,a)=r1(o 1a 1 )+r2(o 2 ,a 2 )

[0137] where (o 1 ,a 1 ) and (o 2 ,a 2 ) are the observations and actions of agent 1 and 2 respectively. For example, this can be the case in team games where agents are focused on their own objectives but do not necessarily observe the objectives of their neighbors. In this case the state value function can be expressed as:

[0138]

[0139] i.e. the original expression is:

[0140]

[0141] Write as then the above expression can be simplified to

[0142]

[0143] which represents the future expected return of agent 1, whose value changes more than the future expected return of agent 2, depending on the observed state of the environment and action (o 1 ,a 1 ) of agent 1 itself. Even so, it is not possible to accurately estimate However, by allowing agent 1 to store its own historical observation information, while also having access to the information of agent 2 in the communication channel, it is possible to reduce the error in this case. The entire expression can be expressed as:

[0144]

[0145] As can be seen from the above analysis, VDNs use a completely decentralized approach, whose output of the overall value function is the sum of each local agent value function, and the decentralized policy is only dependent on the observation information and behavior of each agent itself, and it is not necessary to extract a decentralized policy that is completely consistent with the centralized policy. Therefore, the QMIX algorithm is improved on this basis, only to ensure that the global output Q iot performs a global value function maximization action and a set of individual value function maximization actions performed on each agent Q a have the same result, which can be expressed as follows:

[0146]

[0147] Thus, each agent a chooses its Q a maximizing action to achieve decentralized decision making.

[0148] To make it applicable to more monotone function families, we constrain the relationship between Q tot and each Q a to satisfy monotonicity:

[0149]

[0150] The whole algorithm structure consists of three parts: agent network, hybrid network and super network, as shown in Figure 2 .

[0151] The structure of the agent network is shown in Figure 3 .

[0152] As shown in Figure 3 , each agent inputs the current independent observation information and the last action information into the LSTM network, where h t is the input returned by the network at the last time step, i.e., h t = LSTM(h t-1 , a t )). Each agent has an independent value function Q a as output. Under the network setting, agents can use LSTM networks to represent the current state using all their action-observation history information. The whole agent network uses the DRON network structure, and weight sharing is applied between agent networks. In order to make different agents perform different actions, the IDs of the agents are also added to the observations.

[0153] The whole agent network structure involves two key points: one is the LSTM network structure, and the other is the DRON structure.

[0154] (1) LSTM network structure

[0155] The LSTM network is a variant of the RNN network, mainly used to solve the gradient disappearance problem, and the LSTM network will save information in the memory unit. The typical LSTM neural network structure is shown in Figure 4 .

[0156] The LSTM neural network consists of three structural components: forget gate, input gate and output gate. Among them: the forget gate is responsible for deciding which information should not be left in the unit state; the input gate is responsible for deciding which information should be saved in memory; the output gate is responsible for deciding which information from the unit state should be displayed at time t.

[0157] (2) DRON network structure

[0158] DRON network is similar to DON network discussed in the last chapter, using LSTM network instead of the first convolutional and fully connected layer, as shown in Figure 5 .

[0159] As shown in Figure 5 , the traffic state is input to the convolutional layer, the convolutional layer performs convolution operation on the traffic state and generates feature maps, and then the obtained feature maps are transmitted to the LSTM layer, which retains important information about the previous traffic state and updates its memory according to time, and outputs the Q value after a fully connected layer. Therefore, unlike DON, there is no need to directly estimate Q(s t ,a t ), but to estimate Q(h t ,a t ). Since LSTM network is used, the network is trained by time-based backpropagation. Unlike the experience association method of DQN, DRON saves the entire scene in the experience buffer and samples n time steps from a random batch of scenes. In this way, both randomness and another experience can be adapted. The structure diagram of the hybrid network is shown in Figure 6 , which is a feedforward neural network that takes the output of the agent network as input, performs monotonic mixing, and obtains the Q tot value. The weights are limited to positive values to ensure that the monotonicity constraint is met, so that the hybrid network can arbitrarily approach any monotonic function. The weights of the hybrid network are generated by separate hypernetworks, each of which takes the global state S as input to obtain additional environmental state information and generates the weights of the hybrid network layer. Each hypernetwork is composed of a single linear layer with a Relu activation function to ensure that the weights of the hybrid network are non-negative. The global state is passed through the hypernetwork rather than directly to the hybrid network. This is because Q tot allows additional state information to be obtained in a non-monotonic manner, so some state functions are passed through the monotonic network with the value function of each agent, which is subject to excessive constraints. In contrast, the use of hypernetworks allows the weights of the monotonic network on state S to be adjusted in an arbitrary manner, thereby integrating as much of the complete state S as possible into the joint action value estimate.

[0160] In step 200, a regional traffic signal control framework is constructed, specifically:

[0161] Among them, the regional traffic signal control framework is as shown in Figure 7As shown, the lowest layer of the regional traffic signal control framework is the perception layer, i.e., the agent, which is used to obtain local traffic information and make corresponding decision actions, generate Q function values, the middle layer of the regional traffic signal control framework is the Q network layer, which is used to combine the Q values generated by each agent, and the highest layer of the regional traffic signal control framework is the coordination layer, which is used to generate a global optimal Q value through a mixing network for all Q values, and then feed it back to the local agent to coordinate and control all agents in the lowest layer;

[0162] In this framework, an agent is deployed at the centroid intersection of each sub-region, and the centroid intersection agent can perceive the surrounding traffic state information, and each centroid intersection agent makes decisions at discrete time intervals according to the perceived traffic state information. This framework is a distributed structure, and each agent can communicate with the centroid intersection of its adjacent sub-region to publish its current state information, ensuring that the agents can coordinate with each other to achieve the goal of stabilizing the entire system.

[0163] In step 300, the agent elements in the regional traffic signal control framework are designed, specifically including:

[0164] Step 301: Design the traffic state space;

[0165] Step 302: Design the agent action;

[0166] Step 303: Design the reward and punishment function.

[0167] In step 301, the traffic state space is designed, specifically:

[0168] A traffic state space model is established from three aspects of the position of the vehicle, the speed of the vehicle, and the queue length of each lane;

[0169] Wherein, the design method of vehicle position is to set a crossroads with four roads, and use the position matrix P to represent the specific position of the vehicle at the crossroads, that is:

[0170]

[0171] Wherein, the matrix element P1 represents the position matrix on the i-th road, and is 1 when the vehicle is in the lane cell, and is 0 otherwise;

[0172] The design of vehicle speed is consistent with the design of vehicle position, wherein the speed matrix is represented by V:

[0173]

[0174] Wherein, the matrix element V represents the speed matrix on the i-th road, when the vehicle is in the driving state in the lane cell, the corresponding matrix element value is the speed at this moment; when the vehicle is in the static state, it is 0, and the specific expression is:

[0175]

[0176] Wherein, v car is the speed of the vehicle, v max is the speed limit of the lane for the vehicle;

[0177] Since the queue lengths of different lanes are different, and the directions allowed to pass through each lane are also different, they are distinguished, and the vehicle queue length matrix is:

[0178]

[0179] Wherein, the matrix element D ij represents the vehicle queue length of the j-th lane of the i-th road;

[0180] In summary, the traffic state space is defined as S=[P,V,D].

[0181] In step 302, the agent action is designed, specifically:

[0182] The actions of the agent are designed for the four-phase intersection, and the agent selects the appropriate action from the behavior set according to the traffic flow state information of each phase, wherein the designed behavior is to keep the current signal phase or switch the signal phase according to the traffic flow, and the behavior set can be represented as:

[0183] Wherein is the east-west direction straight right turn phase, is the east-west direction left turn phase, is the south-north direction straight right turn phase, is the south-north direction left turn phase, and the execution time of each action is a fixed minimum unit time interval τ.

[0184] In step 303, the reward and punishment function is designed, specifically:

[0185] Considering that the vehicle delay time has a direct impact on the congestion of the intersection, a model is established by using it:

[0186]

[0187] Wherein, d j represents the delay time of vehicle j, v j represents the speed of the vehicle, and N t is the number of vehicles;

[0188] A model is established using the queue length of the vehicle:

[0189] r l = L t - L t+1

[0190] wherein r l is the change of the vehicle queue length between two time steps, L t is the vehicle queue length at the current time step t, L t+1 is the vehicle queue length at the next time step;

[0191] Therefore, the final reward and punishment function expression is obtained as:

[0192] r = ω1r d + ω2r l + ω3r w

[0193] In the formula, ω1, ω2, ω3 are weight coefficients, which are set according to specific requirements, r w is the change of the accumulated waiting time between two adjacent periods, which is:

[0194]

[0195] In the formula, is the waiting time of vehicle i t in a period, N t represents the total number of vehicles in the period

[0196] In step 400, a multi-agent reinforcement learning algorithm based on value function decomposition is designed to realize multi-intersection traffic signal coordinated control through a regional traffic signal control framework, specifically:

[0197] The above describes the multi-agent reinforcement learning algorithm based on value function decomposition, which realizes multi-intersection traffic signal coordinated control through a regional traffic signal control framework, wherein no additional information exchange and communication channels are required between each agent. At the same time, in order to prevent the traffic environment from being unstable due to each agent not knowing the strategies of other agents, the global traffic environment information is fed back to the hybrid network, so that each agent can utilize the global traffic state and action in the training process, so that each agent can adjust its own strategy according to the strategies of other agents, thereby realizing the global optimal strategy;

[0198] The algorithm is specifically: first, initialize the weight values of the current network and the target network, the sample pool and the experience replay buffer; then in each simulation scenario, the local agent will select an action according to the updated strategy of the current traffic environment state, and each agent will receive the current environment state and the new environment state Ol+1 feedback and update the hidden layer parameters of the LSTM neural network; then the local agent stores the current learning experience {O1, a1, R1, O2, a2, R2,...} into the experience replay buffer and randomly samples from it to update the local learning strategy; then the strategy with the maximum Q value of each local agent is trained end-to-end by the hybrid network to minimize the loss function; finally, the weights of the target network are updated according to the master network.

[0199] The various embodiments are described in a progressive manner in the specification, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be mutually referred to.

[0200] The principles and implementation manners of the present application are described by using specific examples in the specification. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation manners and application ranges can be changed according to the idea of the present application. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A multi-intersection traffic signal control method based on multi-agent adaptive coordination, characterized in that, The application relates to a method for constructing a regional traffic signal control framework based on multi-agent reinforcement learning. The method comprises the following steps: acquiring all intersections of a road network, dividing the intersections, and judging intersections that need to be deployed with agents; constructing a regional traffic signal control framework; designing agent elements in the regional traffic signal control framework; designing a multi-agent reinforcement learning algorithm based on value function decomposition to realize multi-intersection traffic signal coordinated control through the regional traffic signal control framework; acquiring all intersections of a road network, dividing the intersections, and judging intersections that need to be deployed with agents, specifically as follows: acquiring a road network region and dividing the region to determine the number interval of sub-regions; determining the optimal number of sub-regions based on the evaluation of clustering quality; dividing the road network region into sub-regions based on the optimal number of sub-regions and a clustering algorithm; determining the centroid intersection of all sub-regions as the intersection that needs to be deployed with agents; 2. The multi-intersection traffic signal control method based on multi-agent adaptive coordination according to claim 1, characterized in that, judging the intersection that needs to be deployed with agents, deploying sensing devices and agents at the intersection, and connecting the sensing devices to the agents. constructing a regional traffic signal control framework, specifically as follows: 3.The multi-intersection traffic signal control method based on multi-agent adaptive coordination according to claim 1, wherein, the lowest layer of the regional traffic signal control framework is a perception layer, i.e. an agent, which is used to acquire local traffic information, make corresponding decision actions, and generate a Q function value; the middle layer of the regional traffic signal control framework is a Q network layer, which is used to combine the Q values generated by each agent; and the highest layer of the regional traffic signal control framework is a coordination layer, which is used to generate a global optimal Q value through a mixed network based on all Q values, and then feed back the global optimal Q value to the local agent to coordinate and control all agents in the lowest layer. designing agent elements in the regional traffic signal control framework, specifically including: designing a traffic state space; designing agent actions; 4. The multi-intersection traffic signal control method based on multi-agent adaptive coordination according to claim 3, characterized in that, designing a reward and punishment function. designing a traffic state space, specifically as follows: establishing a traffic state space model from the positions of vehicles, the speeds of vehicles and the queuing lengths of each lane; wherein the design method of the positions of vehicles is to set a position matrix P to represent the specific positions of vehicles at an intersection, i.e. wherein a matrix element P1 represents a position matrix on the i-th road, and is 1 when a vehicle is in a lane cell, and is 0 when the vehicle is not in the lane cell; the design method of the speeds of vehicles is consistent with the design method of the positions of vehicles, wherein a speed matrix is represented by V: where v car is the speed of the vehicle, v max is the vehicle speed limit for the lane; wherein a matrix element V represents a speed matrix on the i-th road, and is the speed when a vehicle is in a lane cell, and is 0 when the vehicle is in a static state, and the specific expression is: wherein the matrix elements D ij denotes the vehicle queue length of the jth lane of the ith road; since the queuing lengths of different lanes are different, and the directions allowed to pass through each lane are also different, the queuing lengths of vehicles are distinguished, and a vehicle queuing length matrix is:

5. The multi-intersection traffic signal control method based on multi-agent adaptive coordination according to claim 3, characterized in that, in summary, the traffic state space is defined as S=[P,V,D]. designing agent actions, specifically as follows: wherein is a phase for straight right turn in the east-west direction, is a phase for left turn in the east-west direction, is a phase for straight right turn in the south-north direction, is a phase for left turn in the south-north direction, and each execution action time is a fixed minimum unit time interval τ.

6. The multi-intersection traffic signal control method based on multi-agent adaptive coordination according to claim 3, wherein, designing agent actions for a four-phase intersection, and the agent selects appropriate actions from a behavior set according to the traffic flow state information of each phase, wherein the behavior set is designed according to the traffic flow to keep the current signal phase or switch the signal phase, and the behavior set can be represented as designing a reward and punishment function, specifically as follows: Considering the vehicle delay time has a direct impact on the congestion of intersection, the model is established as follows: where d j expressing the delay time of the vehicle j, v j denotes the speed of the vehicle passage, N t is the number of vehicles; The model is established by using the queue length of vehicles as follows: r l = L t - L t+1 where r l is the change in vehicle queue length between two time steps, L t is the vehicle queue length at the current time step t, L t+1 is the vehicle queue length at the next time step; Finally, the expression of the final reward-punishment function is obtained as follows: r = ω1r d + ω2r l + ω3r w In the formula, ω1, ω2, ω3 are weight coefficients, which are set according to specific requirements, r w is the change of accumulated waiting time between two adjacent periods, and is In the formula, is the vehicle i t The waiting time in a cycle, N t denotes the total number of vehicles in the cycle.

Images